What you'll build5 outcomes
- 01Design ingestion and batch pipelines with Airbyte and Airflow operators.
- 02Ship production-grade dbt models with testing and maintainability in mind.
- 03Apply Great Expectations for profiling, tests, and documentation.
- 04Scale transforms with Apache Spark and PySpark; integrate real-time flows with Kafka.
- 05Build an end-to-end pipeline and harden it for production monitoring.
Full curriculum8 items · 2 projects
01Pipeline Fundamentals & Data Ingestion with Airbyte
02Orchestration with Apache Airflow : DAGs, Operators & Schedules
03Transforming at Scale with dbt : Production-Grade Models
04Data Quality with Great Expectations : Tests, Profiling & Docs
05Distributed Processing with Apache Spark & PySpark
06Real-Time Streaming with Apache Kafka
07Capstone Project Part 1 : End-to-End Pipeline Build
Project
08Capstone Project Part 2 : Production Readiness, Monitoring & Review
Project
Instructor
Dulanga Jayarathna
Instructor · Associate Tech Lead
Dulanga Jayarathna is an Associate Tech Lead at PickMe (Digital Mobility Solutions Lanka) and an associate instructor at Zuu Crew AI. He specialises in data engineering and analytics, with deep hands-on experience building scalable data systems.
MSc Big Data Analytics (Merit) : Robert Gordon University, UKBSc (Hons) Information Technology : University of MoratuwaAssociate Tech Lead : PickMe | Digital Mobility Solutions Lanka