Senior Data Engineer (Data Ingestion)

Описание

WE ARE

SoftServe is a global digital solutions company headquartered in Austin, Texas, founded in 1993. Our associates work on 2,000+ projects with clients across North America, EMEA, APAC, and LATAM. We are about people who create bold things, make a difference, have fun, and love their work.

Big Data & Analytics is the data consulting and engineering arm of our Center of Excellence. Hundreds of data engineers and architects build end-to-end data & analytics solutions: from strategy and architecture through technical design, proof of concept, and full-scale implementation. We work with clients across healthcare, finance, manufacturing, retail, and energy industries.

We hold top-level partnerships with major cloud providers and collaborate with technology leaders such as AWS, GCP, Microsoft, Databricks, Snowflake, Confluent, and others.

IF YOU ARE

An experienced Data Engineer with 5+ years of hands-on practice in building and scaling data solutionsConfident working with Databricks, including developing and maintaining ingestion pipelinesSkilled in Python and SQL for data manipulation, transformation, and integrationComfortable using PySpark to handle distributed data processing at scaleExperienced in web scraping and integrating semi-structured or unstructured data from external sourcesFamiliar with working with large datasets and building data pipelines optimized for performance and reliabilityCommunicative in English (intermediate level or higher), able to collaborate effectively with stakeholders across teams

AND YOU WANT TO

Work on data ingestion, where data from various sources will be ingested into Databricks and then into PostgresAutomate and scale the existing name normalization proof of conceptBuild and support infrastructure around the solutionCreate a data store for reference and integrate it with external systems for validated namesWork on enriching metadata and ensuring sufficient process performanceHandle large datasets (up to 1,000,000 records) and integrate the solution into the onboarding flow for future normalizationCollaborate with stakeholders to iterate on the proof of concept, expand features, and ensure scalability and performanceInfluence the direction of the process and contribute with your technical ideas and solutions

TOGETHER WE WILL

Address different business and technology challenges, engage in impactful projects, use top-notch technologies, and drive multiple initiatives as a part of the Center of ExcellenceSupport your technical and personal growth, as we have a dedicated career plan for all roles in our companyInvestigate new technologies, build internal prototypes, and share knowledge with the SoftServe Data CommunityUpskill with full access to Udemy learning coursesPass professional certifications, encouraged and covered by the companyAdopt best practices from experts while working in a team of top-notch Engineers and ArchitectsCollaborate with world-leading companies and attend professional events