Job Description

Overview:
Senior or Principal Data Engineer needed with over 3 years of experience.
Location can be UK/EU/Kenya/South Africa and US


Technologies
SQL Python Aflow or other scheduling tools Responsibilities:

• Experience building data pipelines for various heterogenous data sources.
• Identifying, designing, and implementing scalable data delivery pipelines and automating manual processes
• Building required infrastructure for optimal data extraction, transformation and loading of data using cloud technologies like AWS, Azure etc.,
• Develop end to end processes on the enterprise level for use by the clinical data configuration specialist to prepare data extraction and transformations of raw data quickly and efficiently from various sources at the study level
• Coordinate with downstream users such as statistical programmers, SDTM programming, analytics, and clinical data programmers to ensure that outputs meet requirements of end users
• Experience creating ELT and ETL to ingest data into data warehouse and data lakes
• Experience creating reusable data pipelines for heterogenous data ingestions
• Manage and maintain pipelines and troubleshoot data in data lake or warehouse
• Provide visualization and analysis of data stored in data lake
• Define and track KPIs and provide continuous improvement
• Develop and maintain, tools, libraries, and reusable templates of data pipelines and standards for study level consumption by data configuration specialist
• Collaborate with various vendors and cross functional teams to build and align on data transfer specification and ensure a streamlined process of data integration
• Provide ad-hoc analysis and visualization as needed
• Ensure accurate delivery of data format and data frequency with quality deliverables per specification
• Participate in the development, maintenance and training rendered by standards and other functions on transfer specs and best practices used by business.
• Collaborate with system architecture team in designing and developing data pipelines as per business needs
• Network with key business stakeholders on refining and enhancing the integration of structured and non-structured data.
• Provide expertise for structured and non-structured data ingestion
• Develop organizational knowledge of key data sources, systems and be a valuable resource to people in the company on how to best integrate data to pursue company objectives.
• Provides technical leadership on various aspects of clinical data flow including assisting with the definition, build, and validation of application program interfaces (APIs), data streams, data staging to various systems for data extraction and integration
• Experience in creating data integrity and data quality checks for data ingestion
• Coordinates with data base builders, clinical data configuration specialists and data management (DM) programmers ensuring accuracy of data integration per SOPs
• Provide technical support / consultancy and end-user support, work with Information Technology (IT) in troubleshooting, reporting, and resolving system issues
• Develop and deliver training programs to internal and external team, ensure timely communication of new and/or revised data transfer specs
• Continuous Improvement/Continuous Development
• Efficiently prepare and process large datasets for various end users for downstream consumption
• Understand end to end requirements for stakeholders and contribute to process and conventions for clinical data ingestion and data transfer agreements
• Adhere to SOPs for computer system validation and all GCP (Good Clinical Practice) regulations
• Ensure compliance with own Learning Curricula, corporate and/or GxP requirements
• Assists with quality review of above activities performed by a vendor, as needed
• Assess and enable clinical data visualization software in the data flows
• Performs other duties as assigned within timelines
• Performs clinical data engineering tasks according to applicable SOPs (standard operating procedures) and processes.

• Software work stream
• Little to no PM experience needed
• Creating end to end pipeline
• Build pipeline from scratch
• Create templates
• Display data
• Extraction and transferring data from multiple sources to build pipelines
• Need to understand how to pull data from API (no specific API mentioned)
• Create pipelines on the cloud to allow individuals to reuse the data
• Not essential to have industry experience although would be beneficial
• Will need to hit the ground running and handle a lot of cases.