Data Pipeline Engineer - Harnham : Job Details

DATA PIPELINE ENGINEERGEN AI STARTUPNEW YORK, NY (HYBRID)$170,000 - $200,000THE COMPANYWe are working with a generative AI startup that has recently raised its next round of funding and is scaling like crazy. This innovative startup within the construction industry is looking to hire a Data Pipeline Engineer to design, develop, and maintain pipeline infrastructure and workflows while working closely with the ML team. THE ROLE - DATA PIPELINE ENGINEER Design, develop and maintain robust file processing pipeline infrastructureOrchestrate the flow of data through various stages of processingEnsure observability and monitoring of the pipeline's healthIntegrate data from various sources including industry storage platforms, project management tools, and external APIsImplement data quality checks and error handling mechanisms to ensure data integrityCollaborate with the machine learning team to enhance pipeline functionality and efficiencySKILLS AND REQUIREMENTS Experience in ETL development and data engineeringExpert coding proficiency in Python and database systems (SQL, noSQL)Strong experience with pipeline orchestration tools (eg. Prefect), infrastructure-as-code (eg. Terraform), and observability and monitoring toolsUnderstanding of serverless architectures (eg. AWS Lambda)Familiarity with ML workflows and requirements (to effectively collaborate with the ML team)Knowledge of data modeling and data warehouse conceptsStartup experienceTHE BENEFITSData Pipeline Engineer role offers a competitive salary range of $170,000- $200,000, depending on experience, along with equity in the company. Comprehensive health, dental, and vision benefits are also included.HOW TO APPLYPlease register your interest by sending your resume to Malia Jalbert via the Apply link on this page.KEYWORDS Data Pipelines, Engineer, ETL, Data Engineering, Artificial Intelligence, Machine Learning, Python, Amazon Web Services, AWS, Data Warehouse, Terraform, Prefect, SQL, NoSQL, Data Warehouse