Overcoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov
The shift to a unified data platform is reshaping how pharmaceutical companies manage and orchestrate data. Establishing standards across regions and teams ensures scalability and efficiency in handling large-scale analytics.In this episode, Evgenii Prusov, Senior Data Platform Engineer of Daiichi Sankyo Europe GmbH, joins us to discuss building and scaling a centralized data platform with Airflow and Astronomer.Key Takeaways:00:00 Introduction.02:49 Building a centralized data platform for 15 European countries.05:19 Adopting SaaS to manage Airflow from day one.07:01 Leveraging Airflow for data orchestration across products.08:16 Teaching non-Python users how to work with Airflow is challenging.12:25 Creating a global data community across Europe, the US and Japan.14:04 Monthly calls help share knowledge and align regional teams.15:47 Contributing to the open-source Airflow project as a way to deepen expertise.16:32 Desire for more guidelines, debugging tutorials and testing best practices in Airflow.Resources Mentioned: Evgenii Prusovhttps://www.linkedin.com/in/prusov/Daiichi Sankyo Europe GmbH | LinkedInhttps://www.linkedin.com/company/daiichi-sankyo-europe-gmbh/Daiichi Sankyo Europe GmbH | Websitehttps://www.daiichi-sankyo.euApache Airflowhttps://airflow.apache.org/Astronomerhttps://www.astronomer.io/Snowflakehttps://www.snowflake.com/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
--------
19:26
--------
19:26
Building a Data-Driven Beauty and Wellness Marketplace at StyleSeat with Paschal Onuorah
StyleSeat is revolutionizing how beauty and wellness professionals grow their businesses through data-driven tools. From streamlining scheduling to optimizing marketing, their platform empowers professionals to focus on their craft while expanding their client base.In this episode, Paschal Onuorah, Senior Data Engineer at StyleSeat, shares how the company leverages Airflow, dbt, and Cosmos to drive marketplace intelligence, improve client connections and deliver measurable growth for professionals.Key Takeaways:00:00 Introduction.05:44 The role of the data engineering team in driving business success.08:52 Leveraging technology for real-time business intelligence.10:52 Data-driven strategies for improving marketing outcomes.13:05 How adopting the right tools can increase revenue growth.14:25 Advantages of simplifying and integrating technical workflows.18:45 Benefits of multi-environment configurations for development and production.20:17 Foundational skills and best practices for learning Airflow effectively.22:33 Opportunities for deeper tool integration and improved data visualization.Resources Mentioned:Paschal Onuorahhttps://www.linkedin.com/in/onuorah-paschal/StyleSeat | LinkedInhttps://www.linkedin.com/company/styleseat/StyleSeat | Websitehttps://www.styleseat.comApache Airflowhttps://airflow.apache.org/dbthttps://www.getdbt.com/Astronomer Cosmoshttps://www.astronomer.io/cosmos/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
--------
23:05
--------
23:05
Building the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak
The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines.In this episode, we’re joined by Ian Buss, Principal Software Engineer at Astronomer, and Piotr Chomiak, Principal Product Manager at Astronomer, who share insights into the Astro Executor and remote execution.Key Takeaways:00:00 Introduction.04:13 How product leadership drives scalability for enterprise needs.08:23 Architectural changes that improve reliability and remove bottlenecks.10:15 Metrics that enhance visibility into system performance.12:54 The role of remote execution in addressing security requirements.15:56 Differences between open-source solutions and managed offerings.19:04 Broad industry adoption and applicability of remote execution.20:39 Future advancements in language support and multi-tenancy.Resources Mentioned:Ian Busshttps://www.linkedin.com/in/ian-buss/Piotr Chomiakhttps://www.linkedin.com/in/piotr-chomiak-b1955624/Astronomer | Websitehttps://www.astronomer.ioApache Airflowhttps://airflow.apache.org/Airflow Slack Communityhttps://airflow.apache.org/community/Beyond Analytics conferencehttps://astronomer.io/beyond/dataflowcastThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
--------
22:25
--------
22:25
Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille
Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable.In this episode, Sébastien Crocquevieille, Data Engineer at Numberly, unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features.Key Takeaways:00:00 Introduction.02:13 Overview of the company’s operations and global presence.04:00 The tech stack and structure of the data engineering team.04:24 Running nearly 2,000 DAGs in production using Airflow.05:42 How Airflow’s UI empowers stakeholders to self-serve and troubleshoot.07:05 Details on the Kubernetes-based Airflow setup using Helm charts.09:31 Transition from GitSync to NFS for DAG syncing due to performance issues.14:11 Making every team member Airflow-literate through local installation.17:56 Using custom libraries and plugins to extend Airflow functionality.Resources Mentioned:Sébastien Crocquevieillehttps://www.linkedin.com/in/scroc/Numberly | LinkedInhttps://www.linkedin.com/company/numberly/Numberly | Websitehttps://numberly.com/Apache Airflowhttps://airflow.apache.org/Grafanahttps://grafana.com/Apache Kafkahttps://kafka.apache.org/Helm Chart for Apache Airflowhttps://airflow.apache.org/docs/helm-chart/stable/index.htmlKuberneteshttps://kubernetes.io/GitLabhttps://about.gitlab.com/KubernetesPodOperator – Airflowhttps://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.htmlBeyond Analytics Conferencehttps://astronomer.io/beyond/dataflowcastThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
--------
24:17
--------
24:17
How Moniepoint Group Uses Airflow for Exposure Monitoring with Adeolu Adegboye
Managing financial data at scale requires precise orchestration and proactive monitoring to maintain operational efficiency.In this episode, we are joined by Adeolu Adegboye, Data Engineer at Moniepoint Group, who shares how his team uses data pipelines and workflow automation to manage high volumes of transactions, ensure timely alerts and support diverse stakeholders across the business.Key Takeaways:(00:00) Introduction. (02:48) The role of data engineering in supporting all business operations.(04:17) Leveraging workflow orchestration to manage daily processes.(05:20) Proactively monitoring for anomalies to prevent potential issues.(08:12) Simplifying complex insights for non-technical teams.(13:01) Improving efficiency through dynamic and parallel workflows.(14:19) Optimizing system performance to handle large-scale operations.(17:19) Exploring creative and innovative uses for workflow automation.Resources Mentioned:Adeolu Adegboyehttps://www.linkedin.com/in/adeolu-adegboye/Moniepoint Group | LinkedInhttps://www.linkedin.com/company/moniepoint-inc/Moniepoint Group | Websitehttps://www.moniepoint.comApache Airflowhttps://airflow.apache.org/ClickHousehttps://clickhouse.com/Grafanahttps://grafana.com/Beyond Analytics Conferencehttps://astronomer.io/beyond/dataflowcastThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning
Om The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.
Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems.
Podcast Webpage: https://www.astronomer.io/podcast/
Lyssna på The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI, Hard Fork och många andra poddar från världens alla hörn med radio.se-appen
Sverige