The AI Demo Looked Great. Then Real Users Showed Up.

Someone on your team is already using an AI tool nobody approved. And the next vendor demo you sit through will look great whether or not the product holds up on your data. Stella Liu, Head of AI Applied Science and Lead Data Scientist in ASU's AI Acceleration group, joins Mike Allton to explain why traditional software testing leaves gaps with generative AI. The same prompt can give a different answer every time, so a product that passed a builder's quick check can miss the edge cases real users hit on day one. In this episode: The three things Stella evaluates: performance, quality and safety Why a polished demo tells you little about real use The vendor who never asks about your data, and other red flags The first question to put to any AI vendor What to add to your procurement checklist Personal vs enterprise AI accounts, and where your team's data can end up Connect with Stella LinkedIn: https://www.linkedin.com/in/wenxingl/ Data Science x AI newsletter: https://datasciencexai.substack.com AI Evals and Analytics course on Maven: https://maven.com/ai-evals-and-analytics/ai-evals-analytics-playbook Mentioned: Shadow AI - https://theaihat.com/shadow-ai Take the AI Opportunity Score: https://theaihat.com/ai-opportunity-score/ Watch on YouTube: https://youtu.be/0cKSJOtBASs Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.