flows & pipelines in a cloud environment using a progressive tech stack e.g. Databricks, Spark, Python, PySpark, Delta Lake, SQL, Logic Apps, Azure Functions, ADLS, Parquet, Neo4J, Flask Ingest and integrate data from a large number of disparate data sources Design and build complex data models for analytical insight Write more »
relational models, and formats to support geo-enabled applications and graph database structures, using knowledge of various data formats including XML, JSON, flat file, Parquet etc. You can also manage metadata across your development teams systems and beyond, with a practitioner knowledge of tracking data provenance and data lineage. more »
Ensure systems meet business requirements and industry practices for data integrity and quality. Manage ETL and ELT pipelines across many data sources (CSV/parquet files, API endpoints, etc) Design and build data models for the business end users. Write complex SQL queries for standard as well as ad more »
/OpenShift including high availability Desired experience: Worked with Python 3.9+ Familiar with Python test automation Experience with SQL and Timeseries databases Familiar with Parquet, Arrow, Airflow, Databricks Experience with cloud AWS services, such as S3, EC2, RDS etc Quality engineering best practice and tooling including TDD, BDD This more »
our team. In this role, you will manage and consolidate data from various sources within the business, including Mongo Atlas , AWS S3 & Glue , CSV & Parquet files . You will report on the success of our most important strategic goals, using data to craft compelling narratives that inform and drive more »
scale data processing and analytics Desired experience: Worked with Python 3.9+ Familiar with Python test automation Experience with SQL and Timeseries databases Familiar with Parquet, Arrow, Airflow, Databricks Experience with cloud AWS services, such as S3, EC2, RDS etc Quality engineering best practice and tooling including TDD, BDD This more »
Manchester, North West, United Kingdom Hybrid / WFH Options
True Worth Consulting Ltd
of experience in a technical environment. Preferred : Proficiency in an additional modular coding language (Python, Java, C++). Experience with semi-structured data (JSON, Parquet, XML). Knowledge of database administration and architecture (Postgres, MS SQL, Oracle, MySQL). Experience with RESTful API frameworks. Development Opportunities: Successful candidates will more »
new technologies and frameworks Nice to have: Knowledge of databases, SQL Familiarity with Boost ASIO Familiarity with data serialization formats such Apache Arrow/Parquet, Google Protocol Buffers, Flatbuffers Experience with gRPC, http/REST and Websocket protocols Experience with Google Cloud/AWS and/or containerization in more »
early stage startups. Experience with Scala and C++ Experience with the internals of query processing and optimization. Understanding of Big Data file formats (e.g. Parquet, ORC, Avro). Understanding of Big Data processing engine internals. Since we are a remote company, we are open to candidates from overseas. No more »
of data modelling (particularly star-schema) and can produce, maintain, and update relevant data models for specific business needs including bespoke serialization techniques (e.g. Parquet) and table formats (Delta etc) This is an incredibly exciting role that rarely becomes available. You will own a well-designed data lake serving more »
and previous experience in early stage startups. Experience with CI and CD tools such as GitHub actions. Understanding of Big Data file formats (e.g. Parquet, ORC, Avro). Understanding of Big Data processing engine internals. Since we are a fully remote company, we would consider applications from candidtates overseas. more »
experience working as a Software Engineer ideally using Rust Outstanding CompSci Fundamentals needed! Knowledge of data engineering principles/data lake architectures - Arrow/Parquet etc. Excellent knowledge of low-level concepts (e.g. compilers, x86, LLVM etc. more »
is redeveloping our teams data platform from a legacy system to one based on AWS EMR, with Amazon RDS and DynamoDB ingestion converted to Parquet files, interrogatable through Spark and MapReduce. This modern platform will support rapid data insight generation, data experiments for new product development, our live Machine … Amazon DynamoDB, AWS Glue, MapReduce, Hive, Spark, YARN, Airflow. Ability to work with a range of structured, semi-structured and unstructured file formats including Parquet, json, csv, pdf, jpg. Accomplished data wrangling and scripting abilities, with advanced knowledge of Python, Linux shell scripting, SQL and NoSQL. Tools and methods more »
ROLE SUMMARY DATA ENGINEER REMOTE/TRAVEL TO CLIENT SITE IN LONDON Valcon UK are looking to recruit a self-motivated, highly logical and intellectually curious Data Engineer’s to join our expanding Data capability in the UK. As a more »
