<description>Summary&lt;br /&gt;In this episode of the Data Engineering Podcast, the creators of Feldera talk about their incremental compute engine designed for continuous computation of data, machine learning, and AI workloads. The discussion covers the concept of incremental computation, the origins of Feldera, and its unique ability to handle both streaming and batch data seamlessly. The guests explore Feldera's architecture, applications in real-time machine learning and AI, and challenges in educating users about incremental computation. They also discuss the balance between open-source and enterprise offerings, and the broader implications of incremental computation for the future of data management, predicting a shift towards unified systems that handle both batch and streaming data efficiently.&lt;br /&gt;&lt;br /&gt;Announcements&lt;br /&gt;&lt;ul&gt;&lt;li&gt;Hello and welcome to the Data Engineering Podcast, the show about modern data management&lt;/li&gt;&lt;li&gt;Imagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at &lt;a href="https://www.dataengineeringpodcast.com/datafold" target="_blank"&gt;dataengineeringpodcast.com/datafold&lt;/a&gt; today!&lt;/li&gt;&lt;li&gt;As a listener of the Data Engineering Podcast you clearly care about data and how it affects your organization and the world. For even more perspective on the ways that data impacts everything around us you should listen to Data Citizens® Dialogues, the forward-thinking podcast from the folks at Collibra. You'll get further insights from industry leaders, innovators, and executives in the world's largest companies on the topics that are top of mind for everyone. They address questions around AI governance, data sharing, and working at global scale. In particular I appreciate the ability to hear about the challenges that enterprise scale businesses are tackling in this fast-moving field. While data is shaping our world, Data Citizens Dialogues is shaping the conversation. Subscribe to &lt;a href="https://www.collibra.com/podcasts" target="_blank"&gt;Data Citizens Dialogues&lt;/a&gt; on Apple, Spotify, Youtube, or wherever you get your podcasts.&lt;/li&gt;&lt;li&gt;Your host is Tobias Macey and today I'm interviewing Leonid Ryzhyk, Lalith Suresh, and Mihai Budiu about Feldera, an incremental compute engine for continous computation of data, ML, and AI workloads&lt;/li&gt;&lt;/ul&gt;Interview&lt;br /&gt;&lt;ul&gt;&lt;li&gt;Introduction&lt;/li&gt;&lt;li&gt;Can you describe what Feldera is and the story behind it?&lt;/li&gt;&lt;li&gt;DBSP (the theory behind Feldera) has won multiple awards from the database research community. Can you explain what it is and how it solves the incremental computation problem?&lt;/li&gt;&lt;li&gt;Depending on which angle you look at it, Feldera has attributes of data warehouses, federated query engines, and stream processors. What are the unique use cases that Feldera is designed to address?&lt;ul&gt;&lt;li&gt;In what situations would you replace another technology with Feldera?&lt;/li&gt;&lt;li&gt;When is it an additive technology?&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;Can you describe the architecture of Feldera?&lt;ul&gt;&lt;li&gt;How have the design and scope evolved since you first started working on it?&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;What are the state storage interfaces available in Feldera?&lt;ul&gt;&lt;li&gt;What are the opportunities for integrating with or building on top of open table formats like Iceberg, Lance, Hudi, etc.?&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;Can you describe a typical workflow for an engineer building with Feldera?&lt;/li&gt;&lt;li&gt;You advertise Feldera's utility in ML and AI use cases in addition to data management. What are the features that make it conducive to those applications?&lt;/li&gt;&lt;li&gt;What is your philosophy toward the community growth and engagement with the open source aspects of Feldera and how you're balancing that with sustainability of the project and business?&lt;/li&gt;&lt;li&gt;What are the most interesting, innovative, or unexpected ways that you have seen Feldera used?&lt;/li&gt;&lt;li&gt;What are the most interesting, unexpected, or challenging lessons that you have learned while working on Feldera?&lt;/li&gt;&lt;li&gt;When is Feldera the wrong choice?&lt;/li&gt;&lt;li&gt;What do you have planned for the future of Feldera?&lt;/li&gt;&lt;/ul&gt;Contact Info&lt;br /&gt;&lt;ul&gt;&lt;li&gt;Leonid&lt;ul&gt;&lt;li&gt;&lt;a href="https://ryzhyk.net/" target="_blank"&gt;Website&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://github.com/ryzhyk" target="_blank"&gt;GitHub&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://www.linkedin.com/in/leonid-ryzhyk-0ba031b9/" target="_blank"&gt;LinkedIn&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;Lalith&lt;ul&gt;&lt;li&gt;&lt;a href="https://www.linkedin.com/in/lalith-suresh-34bb8911/" target="_blank"&gt;LinkedIn&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://lalith.in/research/" target="_blank"&gt;Website&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;Mihai&lt;ul&gt;&lt;li&gt;&lt;a href="https://mihaibudiu.github.io/work/index.html" target="_blank"&gt;Website&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://github.com/mihaibudiu" target="_blank"&gt;GitHub&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;/ul&gt;Parting Question&lt;br /&gt;&lt;ul&gt;&lt;li&gt;From your perspective, what is the biggest gap in the tooling or technology for data management today?&lt;/li&gt;&lt;/ul&gt;Closing Announcements&lt;br /&gt;&lt;ul&gt;&lt;li&gt;Thank you for listening! Don't forget to check out our other shows. &lt;a href="https://www.pythonpodcast.com" target="_blank"&gt;Podcast.__init__&lt;/a&gt; covers the Python language, its community, and the innovative ways it is being used. The &lt;a href="https://www.aiengineeringpodcast.com" target="_blank"&gt;AI Engineering Podcast&lt;/a&gt; is your guide to the fast-moving world of building AI systems.&lt;/li&gt;&lt;li&gt;Visit the &lt;a href="https://www.dataengineeringpodcast.com" target="_blank"&gt;site&lt;/a&gt; to subscribe to the show, sign up for the mailing list, and read the show notes.&lt;/li&gt;&lt;li&gt;If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.&lt;/li&gt;&lt;/ul&gt;Links&lt;br /&gt;&lt;ul&gt;&lt;li&gt;&lt;a href="https://www.feldera.com/" target="_blank"&gt;Feldera&lt;/a&gt;&lt;ul&gt;&lt;li&gt;&lt;a href="https://github.com/feldera/feldera" target="_blank"&gt;GitHub&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2203.16684" target="_blank"&gt;DBSP&lt;/a&gt; paper&lt;ul&gt;&lt;li&gt;&lt;a href="https://docs.rs/dbsp/latest/dbsp/" target="_blank"&gt;Rust Crate&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://timelydataflow.github.io/differential-dataflow/" target="_blank"&gt;Differential Dataflow&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://trino.io/" target="_blank"&gt;Trino&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://flink.apache.org/" target="_blank"&gt;Flink&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://spark.apache.org/" target="_blank"&gt;Spark&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://materialize.com/" target="_blank"&gt;Materialize&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://clickhouse.com/" target="_blank"&gt;Clickhouse&lt;/a&gt;&lt;ul&gt;&lt;li&gt;&lt;a href="https://www.dataengineeringpodcast.com/clickhouse-data-warehouse-episode-88/" target="_blank"&gt;Podcast Episode&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://duckdb.org/" target="_blank"&gt;DuckDB&lt;/a&gt;&lt;ul&gt;&lt;li&gt;&lt;a href="https://www.dataengineeringpodcast.com/duckdb-in-process-olap-database-episode-270/" target="_blank"&gt;Podcast Episode&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://www.snowflake.com" target="_blank"&gt;Snowflake&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://arrow.apache.org/" target="_blank"&gt;Arrow&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://substrait.io/" target="_blank"&gt;Substrait&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://datafusion.apache.org/" target="_blank"&gt;DataFusion&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://en.wikipedia.org/wiki/Digital_signal_processing" target="_blank"&gt;DSP == Digital Signal Processing&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://en.wikipedia.org/wiki/Change_data_capture" target="_blank"&gt;CDC == Change Data Capture&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://prql-lang.org/" target="_blank"&gt;PRQL&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://en.wikipedia.org/wiki/Log-structured_merge-tree" target="_blank"&gt;LSM (Log-Structured Merge) Tree&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://iceberg.apache.org/" target="_blank"&gt;Iceberg&lt;/a&gt;&lt;ul&gt;&lt;li&gt;&lt;a href="https://www.dataengineeringpodcast.com/iceberg-with-ryan-blue-episode-52/" target="_blank"&gt;Podcast Episode&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://delta.io/" target="_blank"&gt;Delta Lake&lt;/a&gt;&lt;ul&gt;&lt;li&gt;&lt;a href="https://www.dataengineeringpodcast.com/delta-lake-data-lake-episode-85/" target="_blank"&gt;Podcast Episode&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://www.openvswitch.org/" target="_blank"&gt;Open VSwitch&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://en.wikipedia.org/wiki/Feature_engineering" target="_blank"&gt;Feature Engineering&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="https://calcite.apache.org/" target="_blank"&gt;Calcite&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;The intro and outro music is from &lt;a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" target="_blank"&gt;The Hug&lt;/a&gt; by &lt;a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" target="_blank"&gt;The Freak Fandango Orchestra&lt;/a&gt; / &lt;a href="http://creativecommons.org/licenses/by-sa/3.0/" target="_blank"&gt;CC BY-SA&lt;/a&gt;</description>

Data Engineering Podcast

Tobias Macey

Feldera: Bridging Batch and Streaming with Incremental Computation

NOV 4, 202447 MIN
Data Engineering Podcast

Feldera: Bridging Batch and Streaming with Incremental Computation

NOV 4, 202447 MIN

Description

Summary<br />In this episode of the Data Engineering Podcast, the creators of Feldera talk about their incremental compute engine designed for continuous computation of data, machine learning, and AI workloads. The discussion covers the concept of incremental computation, the origins of Feldera, and its unique ability to handle both streaming and batch data seamlessly. The guests explore Feldera's architecture, applications in real-time machine learning and AI, and challenges in educating users about incremental computation. They also discuss the balance between open-source and enterprise offerings, and the broader implications of incremental computation for the future of data management, predicting a shift towards unified systems that handle both batch and streaming data efficiently.<br /><br />Announcements<br /><ul><li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li><li>Imagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at <a href="https://www.dataengineeringpodcast.com/datafold" target="_blank">dataengineeringpodcast.com/datafold</a> today!</li><li>As a listener of the Data Engineering Podcast you clearly care about data and how it affects your organization and the world. For even more perspective on the ways that data impacts everything around us you should listen to Data Citizens® Dialogues, the forward-thinking podcast from the folks at Collibra. You'll get further insights from industry leaders, innovators, and executives in the world's largest companies on the topics that are top of mind for everyone. They address questions around AI governance, data sharing, and working at global scale. In particular I appreciate the ability to hear about the challenges that enterprise scale businesses are tackling in this fast-moving field. While data is shaping our world, Data Citizens Dialogues is shaping the conversation. Subscribe to <a href="https://www.collibra.com/podcasts" target="_blank">Data Citizens Dialogues</a> on Apple, Spotify, Youtube, or wherever you get your podcasts.</li><li>Your host is Tobias Macey and today I'm interviewing Leonid Ryzhyk, Lalith Suresh, and Mihai Budiu about Feldera, an incremental compute engine for continous computation of data, ML, and AI workloads</li></ul>Interview<br /><ul><li>Introduction</li><li>Can you describe what Feldera is and the story behind it?</li><li>DBSP (the theory behind Feldera) has won multiple awards from the database research community. Can you explain what it is and how it solves the incremental computation problem?</li><li>Depending on which angle you look at it, Feldera has attributes of data warehouses, federated query engines, and stream processors. What are the unique use cases that Feldera is designed to address?<ul><li>In what situations would you replace another technology with Feldera?</li><li>When is it an additive technology?</li></ul></li><li>Can you describe the architecture of Feldera?<ul><li>How have the design and scope evolved since you first started working on it?</li></ul></li><li>What are the state storage interfaces available in Feldera?<ul><li>What are the opportunities for integrating with or building on top of open table formats like Iceberg, Lance, Hudi, etc.?</li></ul></li><li>Can you describe a typical workflow for an engineer building with Feldera?</li><li>You advertise Feldera's utility in ML and AI use cases in addition to data management. What are the features that make it conducive to those applications?</li><li>What is your philosophy toward the community growth and engagement with the open source aspects of Feldera and how you're balancing that with sustainability of the project and business?</li><li>What are the most interesting, innovative, or unexpected ways that you have seen Feldera used?</li><li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on Feldera?</li><li>When is Feldera the wrong choice?</li><li>What do you have planned for the future of Feldera?</li></ul>Contact Info<br /><ul><li>Leonid<ul><li><a href="https://ryzhyk.net/" target="_blank">Website</a></li><li><a href="https://github.com/ryzhyk" target="_blank">GitHub</a></li><li><a href="https://www.linkedin.com/in/leonid-ryzhyk-0ba031b9/" target="_blank">LinkedIn</a></li></ul></li><li>Lalith<ul><li><a href="https://www.linkedin.com/in/lalith-suresh-34bb8911/" target="_blank">LinkedIn</a></li><li><a href="https://lalith.in/research/" target="_blank">Website</a></li></ul></li><li>Mihai<ul><li><a href="https://mihaibudiu.github.io/work/index.html" target="_blank">Website</a></li><li><a href="https://github.com/mihaibudiu" target="_blank">GitHub</a></li></ul></li></ul>Parting Question<br /><ul><li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li></ul>Closing Announcements<br /><ul><li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" target="_blank">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.aiengineeringpodcast.com" target="_blank">AI Engineering Podcast</a> is your guide to the fast-moving world of building AI systems.</li><li>Visit the <a href="https://www.dataengineeringpodcast.com" target="_blank">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li><li>If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.</li></ul>Links<br /><ul><li><a href="https://www.feldera.com/" target="_blank">Feldera</a><ul><li><a href="https://github.com/feldera/feldera" target="_blank">GitHub</a></li></ul></li><li><a href="https://arxiv.org/abs/2203.16684" target="_blank">DBSP</a> paper<ul><li><a href="https://docs.rs/dbsp/latest/dbsp/" target="_blank">Rust Crate</a></li></ul></li><li><a href="https://timelydataflow.github.io/differential-dataflow/" target="_blank">Differential Dataflow</a></li><li><a href="https://trino.io/" target="_blank">Trino</a></li><li><a href="https://flink.apache.org/" target="_blank">Flink</a></li><li><a href="https://spark.apache.org/" target="_blank">Spark</a></li><li><a href="https://materialize.com/" target="_blank">Materialize</a></li><li><a href="https://clickhouse.com/" target="_blank">Clickhouse</a><ul><li><a href="https://www.dataengineeringpodcast.com/clickhouse-data-warehouse-episode-88/" target="_blank">Podcast Episode</a></li></ul></li><li><a href="https://duckdb.org/" target="_blank">DuckDB</a><ul><li><a href="https://www.dataengineeringpodcast.com/duckdb-in-process-olap-database-episode-270/" target="_blank">Podcast Episode</a></li></ul></li><li><a href="https://www.snowflake.com" target="_blank">Snowflake</a></li><li><a href="https://arrow.apache.org/" target="_blank">Arrow</a></li><li><a href="https://substrait.io/" target="_blank">Substrait</a></li><li><a href="https://datafusion.apache.org/" target="_blank">DataFusion</a></li><li><a href="https://en.wikipedia.org/wiki/Digital_signal_processing" target="_blank">DSP == Digital Signal Processing</a></li><li><a href="https://en.wikipedia.org/wiki/Change_data_capture" target="_blank">CDC == Change Data Capture</a></li><li><a href="https://prql-lang.org/" target="_blank">PRQL</a></li><li><a href="https://en.wikipedia.org/wiki/Log-structured_merge-tree" target="_blank">LSM (Log-Structured Merge) Tree</a></li><li><a href="https://iceberg.apache.org/" target="_blank">Iceberg</a><ul><li><a href="https://www.dataengineeringpodcast.com/iceberg-with-ryan-blue-episode-52/" target="_blank">Podcast Episode</a></li></ul></li><li><a href="https://delta.io/" target="_blank">Delta Lake</a><ul><li><a href="https://www.dataengineeringpodcast.com/delta-lake-data-lake-episode-85/" target="_blank">Podcast Episode</a></li></ul></li><li><a href="https://www.openvswitch.org/" target="_blank">Open VSwitch</a></li><li><a href="https://en.wikipedia.org/wiki/Feature_engineering" target="_blank">Feature Engineering</a></li><li><a href="https://calcite.apache.org/" target="_blank">Calcite</a></li></ul>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" target="_blank">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" target="_blank">CC BY-SA</a>