ShipStack
← All courses
Data Self-paced Intermediate

Data Engineering on the Cloud

Batch and streaming pipelines with Spark, Kafka, Airflow and a modern warehouse.

SHIPSTACK · COURSE DATA Data Engineering onthe Cloud 6 MODULES · SELF-PACED · INTERMEDIATE ~55h

About this course

Learn to move, transform and serve data at scale. Distributed processing with Spark, streaming with Kafka, orchestration with Airflow, warehouse modelling with dbt, and building it all on cloud services.

What you will be able to do

  • Process large datasets with Spark
  • Build streaming pipelines with Kafka
  • Orchestrate workflows with Airflow
  • Model warehouses with dbt
  • Add data quality checks everywhere

Curriculum

01Data engineering foundations
  • Batch vs. streaming
  • Storage formats
  • Lakehouse
02Spark
  • DataFrames
  • Partitioning
  • Performance
03Streaming
  • Kafka
  • Consumers and producers
  • Exactly-once
04Orchestration
  • Airflow DAGs
  • Scheduling
  • Retries
05Warehousing
  • dbt
  • Star schemas
  • Testing
06Cloud
  • Managed services
  • Costs
  • Security

Tools you will use

SparkKafkaAirflowdbtSnowflakeGCP / Azure

Prerequisites

  • Python
  • SQL

Projects you will build

SHIPSTACK · PROJECT SkyStream Real-time flight data pipelinewith Airflow and Snowflake Intermediate Data Engineering 1 Flight API 2 Object storage 3 Airflow + dbt 4 Dashboard 5 milestones
Data Engineering Intermediate

Real-time flight data pipeline with Airflow and Snowflake

Stream live flight positions, model them into a warehouse and power a delay-analytics dashboard that refreshes every few minutes.

Kafka · Airflow · Snowflake Coming soon