Reflecting On The Past 6 Years Of Data Engineering

32m ·

Reflecting On The Past 6 Years Of Data Engineering

Airflow and Luigi -> Dagster, Prefect, Lyft, etc. Orchestration is now a part of most vertical tools Cloud data warehouses Data lakes DataOps and MLOps Data quality to data observability Metadata for everything Data catalog -> active metadata Business intelligence Read only reports to metric/semantic layers Embedded analytics and data APIs Rise of ELT dbt Corresponding introduction of reverse ETL What are the most interesting, unexpected, or challenging lessons that you have learned while working on running the podcast? What do you have planned for the future of the podcast? Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ ( covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast ( helps you go from idea to production with machine learning. Visit the site ( to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email ( with your story. To help other people find the show please leave a review on Apple Podcasts ( and tell your friends and co-workers The intro and outro music is from The Hug ( by The Freak Fandango Orchestra ( / CC BY-SA (
Read more

Login to see and leave a comments

  • Davyhunt

    It is a component of the vast majority of vertical tools. Data warehouses hosted on the cloud


Podcast hosts