PySpark Contracts

PySpark - Spark Python API
UK

The following table provides summary statistics for contract job vacancies with a requirement for PySpark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited PySpark over the 6 months to 2 June 2024 with a comparison to the same period in the previous 2 years.

6 months to
2 Jun 2024
Same period 2023 Same period 2022
Rank 346 436 480
Rank change year-on-year +90 +44 -21
Contract jobs citing PySpark 245 254 414
As % of all contract jobs advertised in the UK 0.57% 0.45% 0.47%
As % of the Libraries, Frameworks & Software Standards category 2.70% 1.96% 1.75%
Number of daily rates quoted 195 175 272
10th Percentile £413 £453 £463
25th Percentile £475 £513 £522
Median daily rate (50th Percentile) £530 £600 £600
Median % change year-on-year -11.67% - +11.11%
75th Percentile £625 £713 £717
90th Percentile £763 £880 £800
UK excluding London median daily rate £500 £600 £558
% change year-on-year -16.67% +7.62% +3.72%
Number of hourly rates quoted 6 0 1
10th Percentile £15.51 - -
25th Percentile £16.98 - -
Median hourly rate £20.74 - £78.75
75th Percentile £21.88 - -
90th Percentile £22.88 - -
UK excluding London median hourly rate £18.14 - -

All Software Libraries and Frameworks
UK

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 9,060 12,931 23,608
As % of all contract IT jobs advertised in the UK 21.02% 22.94% 26.89%
Number of daily rates quoted 6,282 9,442 17,368
10th Percentile £350 £400 £400
25th Percentile £425 £463 £463
Median daily rate (50th Percentile) £538 £550 £550
Median % change year-on-year -2.27% - +10.00%
75th Percentile £650 £675 £669
90th Percentile £775 £775 £750
UK excluding London median daily rate £500 £500 £500
% change year-on-year - - +11.11%
Number of hourly rates quoted 257 148 222
10th Percentile £21.89 £30.53 £22.75
25th Percentile £43.00 £51.25 £35.00
Median hourly rate £47.50 £65.00 £52.50
Median % change year-on-year -26.92% +23.81% +16.67%
75th Percentile £66.25 £75.00 £67.36
90th Percentile £74.46 £80.48 £76.96
UK excluding London median hourly rate £48.43 £65.00 £55.00
% change year-on-year -25.49% +18.18% +22.22%

PySpark
Job Vacancy Trend

Job postings citing PySpark as a proportion of all IT jobs advertised.

Job vacancy trend for PySpark in the UK

PySpark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing PySpark.

Daily rate trend for PySpark in the UK

PySpark
Daily Rate Histogram

Daily rate distribution for jobs citing PySpark over the 6 months to 2 June 2024.

Daily rate histogram for PySpark in the UK

PySpark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing PySpark.

Hourly rate trend for PySpark in the UK

PySpark
Top 13 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing PySpark within the UK over the 6 months to 2 June 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Jobs
England +77 200 £550 -8.33% 91
London +93 152 £550 -11.29% 51
Work from Home +82 90 £500 -16.67% 49
UK excluding London +42 53 £500 -16.67% 43
Scotland +54 18 £500 -6.98% 7
North of England +11 13 £490 -18.33% 13
North West +24 9 £475 -9.52% 8
South East +14 8 £375 -38.14% 8
South West +9 6 £463 -22.92% 7
Wales - 6 £330 -
Yorkshire -3 4 £490 -24.62% 4
Midlands +22 2 £600 +26.32% 7
West Midlands +15 2 £600 +26.32% 1

PySpark
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 50 (20.41%) Apache Spark
2 41 (16.73%) Apache Airflow
3 29 (11.84%) Apache
4 1 (0.41%) Jupyter
Applications
1 3 (1.22%) Sketch
2 2 (0.82%) Microsoft Excel
Business Applications
1 2 (0.82%) Dynamics CRM
2 1 (0.41%) Workday HCM
Cloud Services
1 138 (56.33%) Azure
2 74 (30.20%) AWS
3 51 (20.82%) Azure Data Factory
4 39 (15.92%) Azure Synapse Analytics
5 35 (14.29%) Serverless
6 32 (13.06%) Power Platform
7 23 (9.39%) Azure DevOps
8 21 (8.57%) Amazon S3
9 19 (7.76%) AWS Lambda
9 19 (7.76%) Snowflake
10 18 (7.35%) GCP
11 17 (6.94%) dbt
12 14 (5.71%) Azure Monitor
13 13 (5.31%) Azure Logic Apps
14 12 (4.90%) Amazon EMR
15 11 (4.49%) Azure Cosmos DB
15 11 (4.49%) GitHub
16 9 (3.67%) AWS Step Functions
16 9 (3.67%) Entra ID
17 8 (3.27%) Azure Key Vault
Database & Business Intelligence
1 66 (26.94%) Big Data
2 65 (26.53%) Data Lake
3 54 (22.04%) Data Warehouse
4 32 (13.06%) Power BI
5 23 (9.39%) Delta Lake
5 23 (9.39%) Spark SQL
6 21 (8.57%) Amazon Redshift
7 17 (6.94%) Azure SQL Database
8 15 (6.12%) Amazon Athena
9 13 (5.31%) Parquet
10 11 (4.49%) Apache Hive
10 11 (4.49%) NoSQL
11 10 (4.08%) Relational Database
12 9 (3.67%) Tableau
13 8 (3.27%) BigQuery
13 8 (3.27%) Hadoop
13 8 (3.27%) Impala
13 8 (3.27%) SQL Server
14 6 (2.45%) Data Mining
15 5 (2.04%) Snowflake Schema
Development Applications
1 27 (11.02%) Git
2 14 (5.71%) JIRA
3 5 (2.04%) GitLab
4 4 (1.63%) Deequ
5 2 (0.82%) Jenkins
5 2 (0.82%) JMeter
5 2 (0.82%) MLflow
5 2 (0.82%) TestCafe
5 2 (0.82%) Zephyr
6 1 (0.41%) Maven
6 1 (0.41%) RStudio
6 1 (0.41%) TeamCity
6 1 (0.41%) Travis CI
General
1 50 (20.41%) Finance
2 41 (16.73%) Social Skills
3 34 (13.88%) Analytical Skills
4 21 (8.57%) Banking
5 15 (6.12%) Public Sector
6 10 (4.08%) Marketing
7 9 (3.67%) Retail
8 6 (2.45%) Front Office
8 6 (2.45%) Investment Banking
8 6 (2.45%) Presentation Skills
9 5 (2.04%) Pharmaceutical
10 4 (1.63%) Financial Institution
11 2 (0.82%) Czech Language
11 2 (0.82%) Health Technology
11 2 (0.82%) Inclusion and Diversity
12 1 (0.41%) Back Office
12 1 (0.41%) Law
12 1 (0.41%) Legal
12 1 (0.41%) Multimedia
12 1 (0.41%) Publishing
Job Titles
1 135 (55.10%) Data Engineer
2 50 (20.41%) Azure Engineer
3 46 (18.78%) Azure Data Engineer
4 37 (15.10%) Developer
5 27 (11.02%) Python Engineer
6 26 (10.61%) Senior
7 24 (9.80%) Python Developer
8 22 (8.98%) Python Data Engineer
9 21 (8.57%) Lead
10 18 (7.35%) Architect
10 18 (7.35%) Senior Data Engineer
11 16 (6.53%) AWS Engineer
12 15 (6.12%) Data Scientist
13 14 (5.71%) AWS Data Engineer
14 10 (4.08%) Analyst
14 10 (4.08%) Data Analyst
15 8 (3.27%) AWS Architect
15 8 (3.27%) Lead Data Engineer
15 8 (3.27%) Solutions Architect
16 7 (2.86%) AWS Solution Architect
Libraries, Frameworks & Software Standards
1 17 (6.94%) Kafka
2 16 (6.53%) NumPy
2 16 (6.53%) Pandas
3 11 (4.49%) Apache Avro
4 8 (3.27%) pytest
4 8 (3.27%) SAML
5 7 (2.86%) ARM Templates
6 6 (2.45%) REST
7 5 (2.04%) JSON
7 5 (2.04%) JSON Schema
7 5 (2.04%) PyTorch
8 4 (1.63%) Great Expectations
9 3 (1.22%) CSS
9 3 (1.22%) Django
9 3 (1.22%) Flask
9 3 (1.22%) HTML
9 3 (1.22%) React
9 3 (1.22%) Spark Streaming
9 3 (1.22%) TensorFlow
10 2 (0.82%) Node.js
Miscellaneous
1 8 (3.27%) Distributed Computing
2 7 (2.86%) Data Structures
3 6 (2.45%) Algorithms
4 4 (1.63%) Biopharmaceutical
4 4 (1.63%) Greenfield Project
4 4 (1.63%) Smart Meter
5 3 (1.22%) Team-Oriented Environment
6 2 (0.82%) Data Protection Act
6 2 (0.82%) NHS
6 2 (0.82%) Onboarding
6 2 (0.82%) Public Cloud
6 2 (0.82%) Self-Motivation
7 1 (0.41%) Cloud Native
7 1 (0.41%) Hedge funds
7 1 (0.41%) Low-Code
7 1 (0.41%) Market Risk
7 1 (0.41%) Marketing Technology
7 1 (0.41%) Podcast
7 1 (0.41%) Product Ownership
7 1 (0.41%) Reference Data
Operating Systems
1 5 (2.04%) Linux
2 4 (1.63%) Unix
Processes & Methodologies
1 86 (35.10%) ETL
2 79 (32.24%) Data Engineering
3 77 (31.43%) Agile
4 76 (31.02%) Analytics
5 74 (30.20%) Data Pipeline
6 68 (27.76%) Data Modelling
7 53 (21.63%) CI/CD
7 53 (21.63%) DevOps
8 39 (15.92%) Data Science
9 36 (14.69%) Business Intelligence
10 35 (14.29%) Software Engineering
11 23 (9.39%) Continuous Integration
11 23 (9.39%) Data Transformation
12 22 (8.98%) Visualisation
13 21 (8.57%) Problem-Solving
14 19 (7.76%) SDLC
15 18 (7.35%) Stakeholder Management
16 17 (6.94%) Data Integration
17 16 (6.53%) Machine Learning
18 15 (6.12%) Data Analytics
Programming Languages
1 215 (87.76%) Python
2 111 (45.31%) SQL
3 46 (18.78%) Scala
4 19 (7.76%) Java
5 12 (4.90%) T-SQL
6 6 (2.45%) Bash
6 6 (2.45%) R
7 5 (2.04%) JavaScript
8 4 (1.63%) PowerShell
9 2 (0.82%) C#
9 2 (0.82%) Groovy
9 2 (0.82%) TypeScript
10 1 (0.41%) C++
10 1 (0.41%) Gremlin
Qualifications
1 23 (9.39%) Security Cleared
2 19 (7.76%) SC Cleared
3 9 (3.67%) DV Cleared
4 7 (2.86%) Degree
5 5 (2.04%) Computer Science Degree
6 3 (1.22%) Azure Certification
6 3 (1.22%) Microsoft Certification
7 2 (0.82%) IBM Certification
7 2 (0.82%) Master's Degree
7 2 (0.82%) Numerate Degree
8 1 (0.41%) BPSS Clearance
8 1 (0.41%) PhD
Quality Assurance & Compliance
1 39 (15.92%) Data Quality
2 4 (1.63%) QA
System Software
1 8 (3.27%) Docker
Systems Management
1 16 (6.53%) Terraform
2 11 (4.49%) Kubernetes
3 4 (1.63%) Argo
4 3 (1.22%) Grafana
5 1 (0.41%) Ganglia
5 1 (0.41%) Kibana
Vendors
1 112 (45.71%) Databricks
2 28 (11.43%) Microsoft
3 11 (4.49%) Oracle
4 10 (4.08%) Foundry
4 10 (4.08%) Palantir
5 9 (3.67%) Google
6 8 (3.27%) Ab Initio
6 8 (3.27%) Talend
7 6 (2.45%) SAP
8 5 (2.04%) Cloudera
9 3 (1.22%) Informatica
10 2 (0.82%) Atlassian
10 2 (0.82%) IBM
10 2 (0.82%) OpenAI
11 1 (0.41%) Brio
11 1 (0.41%) Salesforce
11 1 (0.41%) Sybase
11 1 (0.41%) Workday