Senior Data Engineer

Описание

This is us

At Avenga, we believe that human creativity empowers technology that matters. Operating globally, our 6000+ specialists provide a full spectrum of services, including business and tech advisory, enterprise solutions, CX, UX and Ul design, managed services, product development, and software development.

This is the job

We are looking for a talented Senior Data Engineer to join our client’s team. The main objective of the project is to implement clinical solutions for drug development and testing. The process covers all stages up to drug manufacturing, and the solution is designed in a modular fashion.

As a Senior Data Engineer, you will lead the end-to-end design and implementation of the Data Surveillance Platform on Databricks, ensuring alignment with operational, analytical, and regulatory requirements. You will define the data lakehouse architecture, develop ingestion and transformation pipelines using Python and Databricks workflows, and collaborate closely with data engineers, analysts, and stakeholders

This is you

• 4+ years of experience as a Data Engineer

• Proficient in Azure cloud services

• Strong experience in Databricks

• Strong knowledge of SQL and database management

• Proficiency in Python and PySpark

• English — Upper-intermediate or higher

Nice-to-have skills:

• Experience with building AI\ML on top of Databricks

This is your role

• Design and develop highly scalable data ingestion and processing frameworks to transform a variety of datasets, capture metadata, lineage and implement data quality

• Working with the data lakehouse structure, covering the bronze, custom client, silver, and gold layers, and establish standards for data modeling, schema evolution, and data governance

• Design and build ingestion and transformation pipelines using Python (PySpark) and Databricks workflows, implementing automated data harmonization, quality validation, and mapping logic

• Develop and enforce audit trail mechanisms, ensuring all data processing is versioned, traceable

• Assemble large and complex data sets to meet functional and non-functional business requirements

• Optimize the performance of queries and data processing

• Monitor the health and performance of queries

• Collaborate with data engineers, data analysts, and other stakeholders to understand their requirements, provide data solutions

 
At Avenga, everyone matters. We provide equal opportunities in recruitment, career development, and leadership, regardless of race, ethnicity, gender identity, sexual orientation, disability, age, religion, or any other characteristic. We are committed to fostering a work environment where our diverse community of employees, candidates, and business partners actively shapes our growth. By bringing together people from different backgrounds and experiences, we build a workplace where everyone feels free to be themselves while honoring the boundaries of others.