ShipStack
← All projects
Data Engineering Intermediate

SkyStream

Real-time flight data pipeline with Airflow and Snowflake

Stream live flight positions, model them into a warehouse and power a delay-analytics dashboard that refreshes every few minutes.

SHIPSTACK · PROJECT SkyStream Real-time flight data pipelinewith Airflow and Snowflake Intermediate Data Engineering 1 Flight API 2 Object storage 3 Airflow + dbt 4 Dashboard 5 milestones

The scenario

SkyStream is an end-to-end data engineering project: ingest public flight-tracking data, land it in cloud storage, orchestrate transformations with Airflow, model it in Snowflake with dbt, and serve a dashboard on top. Data quality checks run at every hop.

System architecture

01 Flight API Live positions
02 Kafka Stream
03 Object storage Raw zone
04 Airflow + dbt Transform
05 Snowflake Warehouse
06 Dashboard Delays + routes

High-level flow of SkyStream, from input to output.

What you will learn

  • Streaming ingestion
  • Orchestration with Airflow
  • Warehouse modelling with dbt
  • Data quality testing

What you will build

  • Ingestion service
  • Airflow DAGs
  • dbt models
  • Delay dashboard

Build plan

  1. Ingestion
  2. Raw landing
  3. Airflow DAGs
  4. dbt models
  5. Dashboard

Tech stack

KafkaAirflowSnowflakedbtGreat ExpectationsSuperset