Python for Data Engineers: What UK Employers Actually Expect in 2026

A data engineer uses Python to move, check and schedule data: scripts and pipelines that other people rely on the next morning. In the UK, as of 16 October 2026, Python is named on about a third of permanent Data Engineer ads, just behind SQL. It is not the same Python a data analyst is hired for.

Courses often sell one Python: pandas, a notebook, a chart. That is a useful analyst skill. It is a weak description of this job. This guide uses one public source, says what the word "Python" means on a data engineer advert, and sets an order of study that matches how people actually get a first role. If you are still choosing the role, start with how to become a data engineer in the UK and how the three data roles differ.

What a UK advert means by Python

IT Jobs Watch tracks skills that appear alongside a job title in UK permanent vacancies. For the six months to 16 October 2026 it lists 2,121 ads with Data Engineer in the title. The median salary on the ads that quote one is £70,000, from 1,494 quoted salaries. The skill counts below are co-occurrences: an ad can name several tools, so the shares do not add up to 100%.

Read 33.90% the same way you should read any job-board share. It is not "every data engineer job requires Python", and it is not "Python is optional, ignore it". About one in three of these ads bother to name it. The ads that do not name it can still hand you a Python repo on week one, because the pipeline is already written in it. The ads that do name it rarely spell out a library.

Two limits on this source, so you do not over-read it. First, it only covers ads whose title contains Data Engineer. Analytics engineer, data platform engineer and machine learning engineer sit outside the count. Second, it does not split salary by skill, so this article will not invent a Python pay premium. For the bands by seniority, see Data Engineer salary in the UK.

The page also ranks Data Engineer 106th among UK permanent titles in this window, up from 228th a year earlier, with the same £70,000 median. Treat the jump in raw ad counts cautiously: IT Jobs Watch's overall sample of permanent jobs grew over the same period. The useful signal is the skill mix, not a claim that demand tripled.

Analyst Python and engineer Python

The same word covers two jobs. If you already read Python for data analysts, keep that version for analyst applications. Do not reuse it as a data engineer portfolio.

On the analyst ads we reviewed for that earlier guide, pandas is the library hiding inside the word Python. On these engineer ads, pandas is named 16 times. That gap is the point. A recruiter who wants Spark-shaped Python writes Python, PySpark or Databricks. They almost never write pandas.

SQL, Python or Spark

You will meet all three in a real data engineering job. They are not a ranking of prestige. They answer different shapes of work.

dbt is named on 129 of these ads (6.08%). Apache Airflow is named on 147 (6.93%). dbt is where a lot of the SQL lives. Airflow, or a cloud scheduler, is where a lot of the Python is called from. The comparison of what each tool is for sits in dbt vs Airflow vs Dagster. How the warehouse, Spark and the orchestrator fit together is in the modern data stack explained.

Cloud does not change the language. Azure is named on 551 ads (25.98%), AWS on 397 (18.72%), GCP on 187 (8.82%). Learn the Python once. Learn the cloud that the adverts you are applying to already pay for.

Libraries to learn, and libraries to skip

Job ads say "Python". The work is a short list. Learn it in this order, and stop when the list runs out.

1. Enough core Python to not get stuck

Functions, dictionaries, reading and writing a file, and reading an error without pasting it somewhere first. A little typing helps you notice when a column arrived as text. You do not need classes, decorators or a packaged library for a first role. If the syntax itself is the blocked part, the free Python Foundations crash course is seven short daily lessons on that layer, before you touch a pipeline. It is a start. It is not a portfolio.

2. A small amount of pandas, then stop

Use pandas to open a messy extract, check the row count, and see the nulls. Then leave it. A data engineer who can only operate inside a notebook will struggle the first time the job is "this has to run at 6am without you". The pandas getting started guide is enough documentation for this slice. Do not take a second course that re-teaches the same ten methods.

3. A check that can fail the run

Data quality is named on 451 of the 2,121 ads (21.26%). That line is vague on purpose. In practice it means the pipeline should stop, loudly, when the input is wrong. The shape of that habit is small:

def require_rows(label: str, before: int, after: int, minimum_ratio: float = 0.95) -> None:
    if before == 0:
        raise ValueError(f"{label}: source had no rows")
    if after < before * minimum_ratio:
        raise ValueError(f"{label}: row count fell from {before} to {after}")

You can write the same idea as a dbt test, a PySpark filter, or a pytest function. The tool is secondary. The skill is noticing the drop before a dashboard does.

4. PySpark, when the ads you want name it

PySpark is the Python API for Apache Spark. IT Jobs Watch lists them separately: PySpark on 209 ads (9.85%), Apache Spark on 144 (6.79%). Do not add those numbers. An advert can name either, or both. Databricks, named on 303 ads (14.29%), is the platform a large share of that work actually runs on. If the advert says Databricks, they mean Spark-shaped Python on that platform, not a new language.

Learn PySpark after you can write the SQL the Spark job is replacing, and after you can explain why a join multiplied the rows. The PySpark getting started guide is the right document. A cluster tutorial that never shows a bad join is the wrong one.

Leave these until the role changes

• Scala, as a default. 62 ads (2.92%). Some Spark codebases are still Scala. Learn it for that team, not before you have the offer.
• Django, Flask, FastAPI. Web applications. Data engineer interviews rarely open with them. A small script that exposes a health check is enough exposure.
• scikit-learn, TensorFlow, model training. That is a data scientist or AI engineering path. The data scientist roadmap is the right article if that is the job you actually want.
• LeetCode-style algorithms, as the main plan. Some software screens test this. Data engineer screens in the UK more often test SQL, a pipeline you built, and whether you can see a broken join. The questions we see in practice are in data engineer interview questions.

What job-ready Python looks like

For a first UK Data Engineer role, job-ready means you can do the following on a dataset you have not seen before, without a tutorial open beside you:

• Read a file or a query result and say how many rows arrived, and how many survived each step.
• Name the grain: one row is one order, one customer, or one event, and prove it.
• Write the transformation as a function you could run again tomorrow, not as cells in a notebook.
• Fail the run when the row count, the null rate, or the key uniqueness is worse than you allow.
• Put it in git, with a README that says how to rerun it and what it refuses to claim.

Git itself is named on 161 of these ads (7.59%). That understates it. Teams assume version control the way they assume you can read an error. A folder of notebooks with no history does not read as engineering.

You do not need a custom framework. You do need to predict the row count before you trust the output. That single habit separates people who have finished a Python playlist from people who can be left with a pipeline.

How long it takes

Assume you already write intermediate SQL: joins, aggregates, a window function, and a suspicion of nulls. From there, eight to twelve weeks at six to eight hours a week is enough to reach the list above, if those hours are spent on a real table and not on watching someone else type.

A workable split is two weeks on core syntax if you need it, four to six weeks on one pipeline (ingest, transform, check, rerun), and two weeks making that pipeline explainable in a README. Add PySpark only after that, and only if the roles you are applying to name Spark or Databricks. Another six to eight weeks is a fair block for a first Spark project. Starting there, with shaky SQL, produces a cluster bill and a join you cannot debug.

AI is named on 428 of the same 2,121 ads (20.18%). Use a coding assistant to draft a transformation, then predict the row count yourself before you trust it. The failure modes that matter for this job are covered in using Copilot, Cursor and AI coding agents as a data engineer.

Mistakes that waste months

Treating engineer Python as analyst Python

A tidy pandas notebook with three charts answers an analyst brief. It does not show that you can leave a run unattended. If the README never says how the job fails, it is the wrong project for this role.

Opening Spark before SQL

Spark is a way to run a transformation at volume. If you cannot write the transformation in SQL on a small table, Spark hides the bug inside a slower log. SQL is on more ads. Interviews still open with it.

Collecting cloud certificates instead of one pipeline

Azure, AWS and GCP are all named on these ads. A certificate without a pipeline you can draw on a whiteboard does not tell a hiring manager whether you notice a broken load. One cloud, one project, is the stronger application.

Trusting generated code you cannot check

A model will happily filter the wrong column and hand you a file that looks finished. If you cannot say what the row count should be before you run the script, you are not ready to use the assistant on someone else's data.

Showing Python in a portfolio

One pipeline is enough. Give it these pieces, in this order, in the README:

1. The source, and what was wrong with it: late files, duplicate keys, a column that changed type.
2. The transformation, and the grain of the table you produce.
3. The check that would fail the run, and an example of it failing.
4. How someone else reruns it tomorrow, including the command.
5. What you would not claim, because the data cannot carry it.

Put the script in a public GitHub repository. Keep the failed check in the history. The failed check is the evidence that you did the engineer's job rather than the tutorial's job. The rest of the structure UK recruiters look for is in how to build a data engineer portfolio. A realistic week in the role, once you are in it, is in what a data engineer actually does all day.

Read next

• How to Become a Data Engineer in the UK: A Realistic 2026 Roadmap
• The Modern Data Stack Explained: dbt, Airflow, Snowflake and Spark
• How to Build a Data Engineer Portfolio That Actually Gets UK Interviews