Orchestrating cloud data migrations at HCL Tech

Large-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and strict operational controls. In this episode, Kenten Danas is joined by Bhanu Prakash, Technical Lead at HCL Tech, to talk through how his team uses Airflow to orchestrate end-to-end data and platform migrations to the cloud, generate Dags dynamically for micro-batch workloads, and automate incident response through a ServiceNow integration. Key Takeaways: 00:00 Introduction. 01:19 Bhanu's role as an Airflow and ETL SME at HCL Tech, leading architecture and delivery across data engineering engagements. 02:30 Six years of Airflow use cases: data migration, scheduled analytics, customer segmentation refreshes, and web-application-adjacent data ops. 03:19 When to recommend Airflow to clients: everything except real-time, with cloud-agnostic portability as the deciding factor over ADF and Glue. 06:19 Where Airflow fits in the current cloud migration project: watermarking, cross-reference tables, on-prem to cloud movement, and downstream Databricks and Snowflake processing. 08:01 Running through Airflow versions 2.3, 2.11, and now 3.2, and how backwards compatibility kept 99% of the code unchanged. 11:10 Building a Dag generation utility driven by JSON config, plus runtime Dag creation for micro-batching (illustrated with an e-commerce example). 14:45 Making Dags configurable for clients with pull and push triggering, and the blue-green pattern for promoting Dags through dev, stage, pre-prod, and prod. 16:28 Using Airflow's ServiceNow integration to build an auto-resolution engine that uses LLMs to build context from past incidents and apply gated fixes. 18:12 A wish-list feature: making Airflow an AI-first platform where cross-tool debugging happens inside the Airflow UI. Resources Mentioned: HCL Tech Apache Airflow ServiceNow Databricks Snowflake Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Orchestration #Airflow