<p>AI is moving fast from research to real-world deployment, and when things go wrong, the consequences are no longer hypothetical. In this episode, Sean McGregor, co-founder of the AI Verification &amp; Evaluation Research Institute and also the founder of the AI Incident Database, joins Chris and Dan to discuss AI safety, verification, evaluation, and auditing. They explore why benchmarks often fall short, what red-teaming at DEF CON reveals about machine learning risks, and how organizations can better assess and manage AI systems in practice.</p><p>Featuring:</p><ul><li>Sean McGregor– <a href="https://www.linkedin.com/in/seanbmcgregor/">LinkedIn</a></li><li>Chris Benson – <a href="https://chrisbenson.com/">Website</a>, <a href="https://www.linkedin.com/in/chrisbenson">LinkedIn</a>, <a href="https://bsky.app/profile/chrisbenson.bsky.social">Bluesky</a>, <a href="https://github.com/chrisbenson">GitHub</a>, <a href="https://x.com/chrisbenson">X</a></li><li>Daniel Whitenack – <a href="https://www.datadan.io/">Website</a>, <a href="https://github.com/dwhitena">GitHub</a>, <a href="https://x.com/dwhitena">X</a></li></ul><p>Links:</p><ul><li><a href="https://www.averi.org/">AI Verification &amp; Evaluation Research Institute</a></li><li><a href="https://incidentdatabase.ai/">AI Incident Database</a></li><li><a href="https://aaai.org/conference/aaai/aaai-26/iaai-26-call/">38th convening of IAAI</a></li><li><a href="https://www.youtube.com/watch?v=rE17hShOF_Q">BenchRisk</a></li><li><a href="https://www.youtube.com/watch?v=BOjTPSimNJk">State of Global AI Incident Reporting</a></li></ul><p>Upcoming Events: </p><ul><li>Register for <a href="https://practicalai.fm/webinars">upcoming webinars here</a>!</li></ul>

Practical AI

Practical AI LLC

AI incidents, audits, and the limits of benchmarks

FEB 13, 202642 MIN
Practical AI

AI incidents, audits, and the limits of benchmarks

FEB 13, 202642 MIN

Description

AI is moving fast from research to real-world deployment, and when things go wrong, the consequences are no longer hypothetical. In this episode, Sean McGregor, co-founder of the AI Verification & Evaluation Research Institute and also the founder of the AI Incident Database, joins Chris and Dan to discuss AI safety, verification, evaluation, and auditing. They explore why benchmarks often fall short, what red-teaming at DEF CON reveals about machine learning risks, and how organizations can better assess and manage AI systems in practice.Featuring:Sean McGregor– LinkedInChris Benson – Website, LinkedIn, Bluesky, GitHub, XDaniel Whitenack – Website, GitHub, XLinks:AI Verification & Evaluation Research InstituteAI Incident Database38th convening of IAAIBenchRiskState of Global AI Incident ReportingUpcoming Events: Register for upcoming webinars here!