Adventures in DevOps

Share Episode From your feedback, we know that on-call issues, especially those that arise at the database layer are always the most challenging to solve. This episode builds on our Grafana Observability episode as we go deeper into the challenge of data ownership in data lakes. Apache Druid creator, previous Yahoo Distinguished Engineer, Fellow at Splunk, and currently the Chief Architect as Imply, Eric Tschetter joins us to confirm that there are unsolved problems in data platforms, and it doesn't get better when we throw agents into the mix. In the true DevOps mindset aligment, Eric reminds us that whoever is operating the system, knows what is going on, they know their service boundaries, and so it's their choice to make. And the sad reality is that at large enough companies, you will always have independent teams making independent decisions, it's never possible to fully align them, and realistically we get down to why it is even a mistake to try. 💡 Notable Links: ✨ Episode: Grafana — Observability ✨ Episode: AST can run anything Book: Lean Manufacturing — The Toyota Way Monkeys typing the Complete Works of Shakespeare Paper: Ironies of Automation (& AI) Jeff Bezos API Rant — 2011 Book: The Formula, Science Success Show: Burn Notice 🎯 Picks: Warren - The Agency Eric - 継続は力なり — keizoku wa chikara nari

Share Episode Sauce Labs CTO Anoop Tripathi addresses the root causes of broken test suites and why teams frequently become numb to intermittent test failures. Anoop shares concrete lessons from his career, detailing how bad practices like embedding fixed sleep routines instead of signal-based waiting ruin test reliability. We discuss the necessity of designing products for testability from day one—such as using pair programming where one developer writes the implementation while the other writes the test suite, or building dedicated test interfaces directly into the application architecture. We also analyze the hidden business costs of poor quality, where engineering debt forces sales teams to spend their time managing upset clients instead of acquiring new revenue. We explore organizational anti-patterns—such as teams claiming to be 'Agile' with one-year sprint cycles, or shipping debug builds to production because release builds continuously crash, highlighting a pair-programming model where one developer writes the code while another writes the corresponding tests Finally, we tackle the dangers of chasing vanity metrics, from engineers writing automated assert pass statements to claim fake coverage, to organizations bragging about more code. Anoop distinguishes between output gain and outcome gain, warning that LLMs are creating a massive testing and review crisis by generating 20-page documentation files for minor features. 💡 Notable Links: Quality Iron Triange of Software Book: Sapiens ✨ Episode: DORA Report ✨ Episode: Grafana — Managing Open Source 🎯 Picks: Warren - Proof of Quantum Consciousness - Paper Annop - Book: Genesis: Artificial Intelligence, Hope, and the Human Spirit

Share Episode In this episode, we sit down with Nikhil Mungel, Head of AI R&D at Cribl, to discuss how LLMs have dramatically transformed the tech hiring landscape. The conversation opens with an honest look at the flood of AI-generated, low-quality resumes and remote interview cheating tactics, which have pushed many organizations toward in-person whiteboard interviews or closed-network referrals. We explore how software development has evolved from character-by-character coding into "judgment work," where an engineer's value relies on holistic decision-making, taste, and business alignment rather than raw syntax memorization. Nikhil emphasizes the importance of an "ownership mindset," urging developers to move beyond acting as mere specification-translators and instead evaluate technical trade-offs through a business-focused lens. We also pull back the curtain on modern technical interviews, highlighting why expressing strong convictions, asking intentional career questions, and proving human reasoning without LLM assistance remain essential for standing out. 💡 Notable Links: Show: Pluribus 🎯 Picks: Warren - Sony made the walkman worse Nikhil - Book: Leadership Strategy and Tactics: Field Manual

Share Episode Pushkar Gopalakrishna, Senior Staff Software Engineer at Snap, previously Cruise and AWS, joins to explore why engineering organizations pivot away from Terraform and HCL toward Kubernetes-native tools even when they might not be better. We unpack how developer friction, copy-pasted control structures, and misaligned organizational incentives create massive tech debt—often forcing SREs to manually update infrastructure repositories for compliance, resulting in broken pipelines and severe operational friction. We debate over the mechanics of Crossplane, detailing how its continuous reconciliation loop and Custom Resource Definitions (CRDs) allow teams to express cloud infrastructure as YAML alongside their application manifests at potentially the cost of async validation. Pushkar pulls back the curtain on how Cruise managed infrastructure at scale using a custom internal platform called 'Juno' to bootstrap GCP projects, repositories, and permissions, while leveraging Crossplane for application-level resources. We also dive into the dangers of using CI tools for continuous deployment, detailing a terrifying incident where a pipeline bug accidentally marked three production Kubernetes namespaces for deletion, and how moving to ArgoCD and Argo Rollouts helped prevent future outages for autonomous vehicles. Finally, we touch on the realities of non-production environment isolation, testing against live APIs, and why platform teams must balance providing a seamless developer experience without stripping away developer accountability. 💡 Notable Links: Crossplane Podcast Guest Request for Principal Engineer — What work are you doing? Amazon Multi-level fullyment center for drones ✨ Episode: Terraform vs OpenTofu 🎯 Picks: Warren - Books: The Murderbot Diaries Pushkar - DJI mini drone

Share Episode Co-Founder of Grafana Labs, Anthony Woods, joins to share his perspective on how open source solutions are thriving despite the deluge of pull requests being thrown at them through their open source repositories. Most importantly, he outlines how observability is no longer being done by users looking directly at dashboards. The data rarely makes to LLMs or automation, it rarely makes sense even to humans looking at them, without the context. The context is critical component, and having a model that was built on the semantic concepts relevant to your use cases. And of course we can't stay away from asking on the record the current state of security of open source repositories from a vendor side. Given how their was an inevitable incident with some of the Grafana open source repos, we dig in to figure out how they are dealing with the real world impacts of malware being spread throughout the ecosystem. 💡 Notable Links: Coinbase's ridiculous spend on observability Book: Crossing the Chasm Silicon Valley Show: Hot Dog or Not Hot Dog Shai Hulud — Grafana open source compromise Podcast Guest Request for Package Manager Security Expert ✨ Episode: Productivity 🎯 Picks: Warren - Best Starfleet Captain: Pike Anthony - The Bitter Lesson