Coming Soon

Building Data Pipelines at Scale

Pipeline fundamentals and Airbyte ingestion, Airflow DAGs and schedules, production dbt, data quality with Great Expectations, Spark and PySpark, Kafka streaming, plus a two-part pipeline capstone.

Duration: 8 WeeksModules: 8Projects: 6 modules + 2-part capstoneLevel: Intermediate

What You Will Learn

Design ingestion and batch pipelines with Airbyte and Airflow operators and schedules.
Ship production-grade dbt models with testing and maintainability in mind.
Apply Great Expectations for profiling, tests, and documentation.
Scale transforms with Apache Spark and PySpark; integrate real-time flows with Kafka.
Build an end-to-end pipeline and harden it for production monitoring and review.

Full Curriculum

01Pipeline Fundamentals & Data Ingestion with Airbyte
02Orchestration with Apache Airflow: DAGs, Operators & Schedules
03Transforming at Scale with dbt: Production-Grade Models
04Data Quality with Great Expectations: Tests, Profiling & Docs
05Distributed Processing with Apache Spark & PySpark
06Real-Time Streaming with Apache Kafka
07Capstone Project Part 1: End-to-End Pipeline BuildProject
08Capstone Project Part 2: Production Readiness, Monitoring & ReviewProject

Instructor

Isuru Alagiyawanna

Lead Instructor & Head of AI Engineering

Isuru leads Zuu Crew's technical direction and curriculum design. He has trained students and professionals across practical machine learning, MLOps, and AI engineering systems with a strong emphasis on shipping production-grade work.