<p>In this episode of Gradient Descent, we explore two competing approaches to fine-tuning LLMs: Reinforcement Learning with Human Feedback (RLHF) and Direct Preference Optimization (DPO). Dive into the mechanics of RLHF, its computational challenges, and how DPO simplifies the process by eliminating the need for a separate reward model. We also discuss supervised fine-tuning, emerging methods like Identity Preference Optimization (IPO) and Kahneman-Tversky Optimization (KTO), and their real-world applications in models like Llama 3 and Mistral. Learn practical LLM optimization strategies, including task modularization to boost performance without extensive fine-tuning. </p><p><br></p><p>Timestamps:</p><p>Intro - 0:00</p><p>Overview of LLM Fine-Tuning - 00:48</p><p>Deep Dive into RLHF - 02:46</p><p>Supervised Fine-Tuning vs. RLHF - 10:38</p><p>DPO and Other RLHF Alternatives - 14:43</p><p>Real-World Applications in Frontier Models - 22:23</p><p>Practical Tips for LLM Optimization - 25:18</p><p>Closing Thoughts - 36:05</p><p><br></p><p>References:</p><p>[1] Training language models to follow instructions with human feedback https://arxiv.org/abs/2203.02155</p><p>[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model https://arxiv.org/abs/2305.18290 </p><p>[3] Hugging Face Blog on DPO: Simplifying Alignment: From RLHF to Direct Preference Optimization (DPO) https://huggingface.co/blog/ariG23498/rlhf-to-dpo</p><p>[4] Comparative Analysis: RLHF and DPO Compared https://crowdworks.blog/en/rlhf-and-dpo-compared/</p><p>[5] YouTube Explanation: How to fine-tune LLMs directly without reinforcement learning https://www.youtube.com/watch?v=k2pD3k1485A</p><p><br></p><p>Listen on:</p><p>• <a href="https://podcasts.apple.com/us/podcast/gradient-descent-podcast-about-ai-and-data/id1801323847" target="_blank" rel="noopener noreferer">Apple Podcasts</a>: </p><p>https://podcasts.apple.com/us/podcast/gradient-descent-podcast-about-ai-and-data/id1801323847</p><p>• <a href="https://open.spotify.com/show/1nG58pwg2Dv6oAhCTzab55" target="_blank" rel="noopener noreferer">Spotify</a>: </p><p>https://open.spotify.com/show/1nG58pwg2Dv6oAhCTzab55 </p><p>• <a href="https://music.amazon.com/podcasts/79f6ed45-ef49-4919-bebc-e746e0afe94c/gradient-descent---podcast-about-ai-and-data" target="_blank" rel="noopener noreferer">Amazon Music</a>: </p><p>https://music.amazon.com/podcasts/79f6ed45-ef49-4919-bebc-e746e0afe94c/gradient-descent---podcast-about-ai-and-data </p><p>• <a href="https://youtube.com/@WisecubeAI/podcasts" target="_blank" rel="noopener noreferer">YouTube</a>: https://youtube.com/@WisecubeAI/podcasts</p><p><br></p><p>Our solutions:</p><p>- https://askpythia.ai/ - <a href="https://askpythia.ai/" target="_blank" rel="noopener noreferer">LLM Hallucination Detection Tool</a></p><p>- https://www.wisecube.ai - <a href="https://www.wisecube.ai" target="_blank" rel="noopener noreferer">Wisecube AI platform for large-scale biomedical knowledge analysis</a></p><p><br></p><p>Follow us: </p><p>- <a href="https://askpythia.ai/" target="_blank" rel="noopener noreferer">Pythia Website</a>: https://askpythia.ai/</p><p>- <a href="https://www.wisecube.ai" target="_blank" rel="noopener noreferer">Wisecube Website</a>: https://www.wisecube.ai</p><p>- <a href="https://www.linkedin.com/company/wisecube/" target="_blank" rel="noopener noreferer">LinkedIn</a>: https://www.linkedin.com/company/wisecube/ </p><p>- <a href="https://www.facebook.com/wisecubeai" target="_blank" rel="noopener noreferer">Facebook</a>: https://www.facebook.com/wisecubeai</p><p>- <a href="https://x.com/wisecubeai" target="_blank" rel="noopener noreferer">Twitter</a>: https://x.com/wisecubeai</p><p>- <a href="https://www.reddit.com/r/pythia/" target="_blank" rel="noopener noreferer">Reddit</a>: https://www.reddit.com/r/pythia/</p><p>- <a href="https://github.com/wisecubeai" target="_blank" rel="noopener noreferer">GitHub</a>: https://github.com/wisecubeai</p><p><br></p><p>#FineTuning #LLM #RLHF #AI #MachineLearning #AIDevelopment</p>