Data Engineer Roadmap

Data engineering has a deceptively large surface area - SQL, Python, cloud warehouses, orchestration, streaming, governance - but the skills have a clear dependency order. Learn them in sequence, build pipelines that run in production, and you move fast from junior to a well-rounded data engineer.

The ordered learning path

Data engineering rewards engineers who understand each layer before jumping to the next. The sequence below is ordered by dependency: modeling skills assume solid SQL, warehouse skills assume modeling, and streaming skills assume batch pipelines first. Follow the order and you avoid the "I know Spark but can't explain a star schema" trap that shows up in interviews.

1. 1

   Strong SQL

   Step 1

   Window functions, CTEs, aggregations, subqueries, EXPLAIN plans, and index awareness. SQL is the lingua franca of data - every other tool either generates it or competes with it.
2. 2

   Python for data engineering

   Step 2

   File I/O, pandas for small transforms, requests for API ingestion, scripting CLI tools. You do not need data science; you need solid Python that can read, transform and write data reliably.
3. 3

   Data modeling - star schema and normalisation

   Step 3

   Understand 3NF for OLTP and the dimensional model (facts, dimensions, star vs snowflake schema) for analytics. Bad modeling compounds technical debt across every downstream query.
4. 4

   Cloud data warehouses

   Step 4

   Snowflake, BigQuery or Redshift - pick one and learn it well: loading patterns, clustering keys, partitioning, query cost, and how the warehouse separates compute from storage.
5. 5

   Batch ETL/ELT with dbt

   Step 5

   dbt is the dominant transformation layer - SQL models, incremental strategies, tests, documentation, and the ref() dependency graph. Learn it; it is on almost every data engineering job description.
6. 6

   Orchestration with Airflow

   Step 6

   Apache Airflow (or Prefect / Dagster) for scheduling, dependency management, retries and alerting. Understand DAGs, task dependencies, and how to build pipelines that recover from failures.
7. 7

   Streaming with Kafka and Spark

   Step 7

   Real-time ingestion via Kafka topics, stream processing with Spark Structured Streaming or Flink. Understand at-least-once vs exactly-once delivery and event-time vs processing-time windows.
8. 8

   The lakehouse concept

   Step 8

   Delta Lake, Apache Iceberg or Apache Hudi on object storage (S3/GCS). Open table formats that give warehouse-like ACID semantics over cheap blob storage - the architecture most modern stacks are moving to.
9. 9

   Data quality and governance

   Step 9

   dbt tests, Great Expectations, or Monte Carlo for anomaly detection. Data catalogues (Datahub, Amundsen) for discoverability. GDPR/CCPA compliance for PII. This is what separates mature data platforms from pipelines that break silently.

SQL and Python foundations

The most common data engineering interview failure is not Kafka or Spark - it is SQL. Candidates who cannot write a window function or explain a query plan are screened out early at every serious company.

Data modeling

Data modeling is the design work of data engineering. A well-modeled warehouse is fast, readable, and cheap to query. A poorly-modeled one accumulates query debt that compounds with every new analyst and every new dashboard.

The star schema is the default for analytics work: a central fact table (events, transactions, orders) surrounds by dimension tables (users, products, dates). Queries are fast because joins are simple and the cardinality is understood. Start here before reaching for more complex patterns.

Warehouses and ETL/ELT

The shift from ETL (transform before loading) to ELT (load raw data, transform inside the warehouse) is one of the most important architectural shifts in the field. Modern warehouses are cheap enough on compute that it is almost always better to load everything raw and transform in SQL.

Orchestration and streaming

Pipelines need to run on schedules, retry on failure, alert when something goes wrong, and handle dependencies between tasks. That is orchestration. Separately, some use cases need data in seconds, not hours - that is streaming.

• Apache Airflow - define DAGs in Python, set up task dependencies, schedule runs, and monitor in a UI. Understand the executor model (Local, Celery, Kubernetes) and how to write idempotent tasks.
• Prefect / Dagster - modern alternatives to Airflow with better local development experience and native data-awareness. Worth knowing as the ecosystem is shifting.
• Apache Kafka - distributed log for real-time event streaming. Topics, partitions, consumer groups, offsets. The backbone of most streaming architectures.
• Spark Structured Streaming / Flink - stateful stream processing at scale: windowed aggregations, late data handling, checkpointing for fault tolerance.
• When to use streaming - fraud detection, real-time dashboards, event-driven pipelines. Most pipelines are batch; do not over-engineer with streaming until latency requirements demand it.

Lakehouse and governance

The lakehouse pattern puts open table formats (Delta Lake, Apache Iceberg, Apache Hudi) on top of cheap object storage (S3, GCS, ADLS) and provides warehouse-like capabilities: ACID transactions, time travel, schema enforcement, and query engine choice. It is where most enterprise data platforms are heading.

Governance becomes critical as the platform grows. Know the basics: data cataloguing (what data exists and where), lineage (where did this number come from), PII classification (what is sensitive and how is it masked), and access control (who can see what).

Turning skills into a career

Data engineering skills are worth little without a portfolio of real pipelines. Build end-to-end projects that ingest from a real source, model with dbt, orchestrate with Airflow, and surface in a dashboard. Open-source data (government datasets, public APIs) is sufficient to build a portfolio that gets interviews.

• Share code on GitHub with a clear README: what the pipeline does, the data source, the schema, how to run it locally.
• Write about what you learned - a short blog post or LinkedIn post about a problem you solved reaches hiring managers.
• Contribute to open-source data projects (dbt packages, Airflow providers, Airbyte connectors) - it demonstrates seniority and gives you collaborators.
• For the transition from backend engineering, the Backend Roadmap skills (SQL, Python, containers) overlap heavily - you are closer than you think.