<p>In the second episode of Gradient Descent, Vishnu Vettrivel (CTO of Wisecube) and Alex Thomas (Principal Data Scientist) explore the innovative yet controversial idea of using LLMs to judge and evaluate other AI systems. They discuss the hidden human role in AI training, limitations of traditional benchmarks, automated evaluation strengths and weaknesses, and best practices for building reliable AI judgment systems.</p><p>Timestamps:</p><p><a href="https://www.youtube.com/watch?v=fYB_RCxQ_YI&list=PLEBj677AOh-TTcqBwLQtc8G274BeGnruX&index=1" target="_blank" rel="ugc noopener noreferrer">00:00</a> – Introduction &amp; Context </p><p><a href="https://www.youtube.com/watch?v=fYB_RCxQ_YI&list=PLEBj677AOh-TTcqBwLQtc8G274BeGnruX&index=1&t=60s" target="_blank" rel="ugc noopener noreferrer">01:00</a> – The Role of Humans in AI </p><p><a href="https://www.youtube.com/watch?v=fYB_RCxQ_YI&list=PLEBj677AOh-TTcqBwLQtc8G274BeGnruX&index=1&t=238s" target="_blank" rel="ugc noopener noreferrer">03:58</a> – Why Is Evaluating LLMs So Difficult? </p><p><a href="https://www.youtube.com/watch?v=fYB_RCxQ_YI&list=PLEBj677AOh-TTcqBwLQtc8G274BeGnruX&index=1&t=540s" target="_blank" rel="ugc noopener noreferrer">09:00</a> – Pros and Cons of LLM-as-a-Judge </p><p><a href="https://www.youtube.com/watch?v=fYB_RCxQ_YI&list=PLEBj677AOh-TTcqBwLQtc8G274BeGnruX&index=1&t=870s" target="_blank" rel="ugc noopener noreferrer">14:30</a> – How to Make LLM-as-a-Judge More Reliable? </p><p><a href="https://www.youtube.com/watch?v=fYB_RCxQ_YI&list=PLEBj677AOh-TTcqBwLQtc8G274BeGnruX&index=1&t=1170s" target="_blank" rel="ugc noopener noreferrer">19:30</a> – Trust and Reliability Issues </p><p><a href="https://www.youtube.com/watch?v=fYB_RCxQ_YI&list=PLEBj677AOh-TTcqBwLQtc8G274BeGnruX&index=1&t=1500s" target="_blank" rel="ugc noopener noreferrer">25:00</a> – The Future of LLM-as-a-Judge </p><p><a href="https://www.youtube.com/watch?v=fYB_RCxQ_YI&list=PLEBj677AOh-TTcqBwLQtc8G274BeGnruX&index=1&t=1800s" target="_blank" rel="ugc noopener noreferrer">30:00</a> – Final Thoughts and Takeaways </p><p><br></p><p>Listen on:</p><p>• <a href="https://youtube.com/@WisecubeAI/podcasts" target="_blank" rel="ugc noopener noreferrer">⁠YouTube⁠</a>: https://youtube.com/@WisecubeAI/podcasts</p><p>• <a href="https://podcasts.apple.com/us/podcast/gradient-descent-podcast-about-ai-and-data/id1801323847" target="_blank" rel="ugc noopener noreferrer">⁠Apple Podcast⁠</a>: https://apple.co/4kPMxZf</p><p>• <a href="https://open.spotify.com/show/1nG58pwg2Dv6oAhCTzab55" target="_blank" rel="ugc noopener noreferrer">⁠Spotify⁠</a>: https://open.spotify.com/show/1nG58pwg2Dv6oAhCTzab55</p><p>• <a href="https://music.amazon.com/podcasts/79f6ed45-ef49-4919-bebc-e746e0afe94c/gradient-descent---podcast-about-ai-and-data" target="_blank" rel="ugc noopener noreferrer">⁠Amazon Music⁠</a>: https://bit.ly/4izpdO2 </p><p><br></p><p>Our solutions: • https://askpythia.ai/ - <a href="https://askpythia.ai/ ">⁠⁠LLM Hallucination Detection Tool⁠⁠</a> </p><p>• https://www.wisecube.ai - <a href="https://www.wisecube.ai">⁠⁠Wisecube AI⁠⁠</a> platform for large-scale biomedical knowledge analysis</p><p><br></p><p>Follow us: </p><p>• <a href="https://askpythia.ai/" target="_blank" rel="ugc noopener noreferrer">⁠Pythia Website⁠</a>: www.askpythia.ai</p><p>• <a href="www.wisecube.ai" target="_blank" rel="ugc noopener noreferrer">⁠Wisecube Website⁠</a>: www.wisecube.ai</p><p>• <a href="www.linkedin.com/company/wisecube" target="_blank" rel="ugc noopener noreferrer">⁠Linkedin⁠</a>: www.linkedin.com/company/wisecube</p><p>• <a href="www.facebook.com/wisecubeai" target="_blank" rel="ugc noopener noreferrer">⁠Facebook⁠</a>: www.facebook.com/wisecubeai</p><p>• <a href="www.reddit.com/r/pythia/" target="_blank" rel="ugc noopener noreferrer">⁠Reddit⁠</a>: www.reddit.com/r/pythia/</p><p><br></p><p>Mentioned Materials:</p><p>- <a href="https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG" target="_blank" rel="ugc noopener noreferrer">Best Practices for LLM-as-a-Judge</a>: https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG </p><p>- <a href="https://arxiv.org/pdf/2412.05579v2" target="_blank" rel="ugc noopener noreferrer">LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods</a>: https://arxiv.org/pdf/2412.05579v2</p><p>- <a href="https://arxiv.org/abs/2306.05685" target="_blank" rel="ugc noopener noreferrer">Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena</a>: https://arxiv.org/abs/2306.05685</p><p>- <a href="https://www.evidentlyai.com/llm-guide/llm-as-a-judge" target="_blank" rel="ugc noopener noreferrer">Guide to LLM-as-a-Judge</a>: https://www.evidentlyai.com/llm-guide/llm-as-a-judge </p><p>- <a href="https://arxiv.org/pdf/2502.01534" target="_blank" rel="ugc noopener noreferrer">Preference Leakage: A Contamination Problem in LLM-as-a-Judge</a>: https://arxiv.org/pdf/2502.01534</p><p>- <a href="https://arxiv.org/pdf/2305.17926" target="_blank" rel="ugc noopener noreferrer">Large Language Models Are Not Fair Evaluators</a>: https://arxiv.org/pdf/2305.17926</p><p>- <a href="https://arxiv.org/pdf/2402.14016v2" target="_blank" rel="ugc noopener noreferrer">Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment</a>: https://arxiv.org/pdf/2402.14016v2</p><p>- <a href="https://arxiv.org/pdf/2403.17710v4" target="_blank" rel="ugc noopener noreferrer">Optimization-based Prompt Injection Attack to LLM-as-a-Judge</a>: https://arxiv.org/pdf/2403.17710v4</p><p>- <a href="https://aws.amazon.com/blogs/machine-learning/llm-as-a-judge-on-amazon-bedrock-model-evaluation/ " target="_blank" rel="ugc noopener noreferrer">AWS Bedrock: Model Evaluation</a>: https://aws.amazon.com/blogs/machine-learning/llm-as-a-judge-on-amazon-bedrock-model-evaluation/ </p><p>- <a href="https://huggingface.co/learn/cookbook/en/llm_judge" target="_blank" rel="ugc noopener noreferrer">Hugging Face: LLM Judge Cookbook</a>: https://huggingface.co/learn/cookbook/en/llm_judge</p>