Sr. Data Engineer Job at Long Finch Technologies, Boston, MA

dDRxUGFNdW52d2l1V2ZmT1A1UU1NbG4rdkE9PQ==
  • Long Finch Technologies
  • Boston, MA

Job Description

Job Description: Sr. Data Engineer – Data Ingestion Framework

Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as

the data Lakehouse storage layer and Snowflake as the target data platform.

 

Engagement:   Data Platform Modernization / Data Ingestion Framework

Position Overview

We are looking for an experienced Lead Data Engineer with strong expertise in Python and DBT to design and develop a scalable, reusable, and technology-agnostic data ingestion framework.

The role involves replacing existing ETL capabilities with an open-source technology stack leveraging Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.

The ideal candidate should have hands-on experience building enterprise-grade data ingestion frameworks, developing reusable connectors, implementing metadata-driven pipelines, and integrating heterogeneous source systems with modern cloud data platforms.

Key Responsibilities

1. Data Ingestion Framework Development

  • Develop a reusable, scalable, and technology-agnostic data ingestion framework using Python.
  • Design modular and extensible components to support multiple source systems.
  • Develop metadata-driven and configuration-based ingestion capabilities.
  • Build reusable connectors and adapters for different source technologies.
  • Implement full and incremental data ingestion patterns, including CDC where applicable.
  • Implement robust error handling, logging, auditing, retry, and reprocessing mechanisms.

2. Data Integration & Lakehouse Architecture

  • Develop ingestion pipelines to integrate data from heterogeneous source systems into AWS S3 and Snowflake.
  • Implement data ingestion from SQL Server, Oracle, PostgreSQL, REST APIs, SFTP, AWS S3, and other source systems.
  • Design and implement data lakehouse ingestion patterns using AWS S3.
  • Work with file formats such as CSV, JSON, Parquet, and other structured/semi-structured formats.
  • Implement schema handling, data validation, partitioning, and data movement strategies.

3. Python & dbt Engineering

  • Develop production-grade Python applications and reusable ingestion libraries.
  • Build and maintain dbt models for data transformation and integration.
  • Implement dbt testing, documentation, incremental models, and reusable macros.
  • Optimize Python-based ingestion processes and dbt transformations for performance and scalability.
  • Follow engineering best practices for modularity, maintainability, and code quality.

4. Workflow Orchestration

  • Implement and manage workflow orchestration using Control-M and Dagester.
  • Develop reusable workflows for scheduling, dependency management, retries, and pipeline monitoring.
  • Integrate orchestration workflows with Python ingestion components and Snowflake.

5. Testing, Deployment & Production Support

  • Develop unit tests and integration tests for ingestion framework components.
  • Support migration and deployment across DEV, SIT, UAT, and PROD environments.
  • Collaborate with QA teams to perform functional, integration, regression, and performance testing.
  • Troubleshoot ingestion failures, data discrepancies, and performance bottlenecks.
  • Support production deployment, stabilization, documentation, and knowledge transfer.

6. Technical Leadership & Collaboration

  • Work closely with the onshore solution architect, customer stakeholders, and offshore engineering teams.
  • Provide technical guidance, code reviews, and engineering best practices.
  • Participate in architecture discussions, technical design reviews, and sprint planning.
  • Mentor developers and ensure adherence to established engineering standards.

Required Technical Skills

  • Python: Expert-level hands-on experience in Python development, including object-oriented programming, reusable libraries, APIs, error handling, and performance optimization.
  • DBT: Strong hands-on expertise in DBT Core, including models, macros, tests, incremental processing, and deployment practices.
  • Data Engineering: Extensive experience building enterprise-scale ETL/ELT pipelines and reusable data ingestion frameworks.
  • Snowflake: Strong experience with Snowflake architecture, data loading, stages, Snowpipe, SQL, and performance optimization.
  • AWS: Hands-on experience with AWS S3, IAM, and cloud-based data integration patterns.
  • Data Integration: Experience integrating relational databases, REST APIs, SFTP, and cloud-based storage systems.
  • SQL: Advanced SQL development and query optimization skills.
  • CI/CD: Experience with Git-based version control, automated builds, testing, and deployment pipelines.

Good-to-Have Skills

  • Hands-on experience with  Dagster for data orchestration and asset-based pipeline development.
  • Experience with utilizing Control-M JOB scheduler.
  • Experience with Apache Iceberg and open table formats.
  • Experience implementing metadata-driven ingestion frameworks.
  • Experience migrating ETL workloads from SnapLogic or similar integration platforms.
  • Experience with Docker, Terraform, and infrastructure automation.
  • Experience implementing data quality, observability, and monitoring solutions.
  • Experience working in an Agile environment utilizing SCRUM

Qualifications

  • 10–12 years of overall experience in data engineering and data integration.
  • Strong architectural and hands-on engineering capabilities.
  • Experience working on enterprise data modernization initiatives.
  • Excellent communication and stakeholder management skills.
  • Ability to work independently in a distributed onshore/offshore delivery model.

Mandatory Skills

  1. Expert-level Python development.
  2. Strong hands-on dbt experience.
  3. Enterprise data ingestion framework development.
  4. Snowflake and AWS S3 experience.
  5. Strong SQL and data integration experience.

Preferred:   Dagster, Control-M, Apache Iceberg, and SnapLogic migration experience.

 

 

Job Tags

Contract work, Local area, Remote work

Similar Jobs

Hartford Healthcare

Nurse Navigator Job at Hartford Healthcare

Hartford Healthcare seeks a Nurse Navigator to guide patients through the healthcare continuum. In this RN role, youll coordinate care, interpret clinical information, educate patients and families, and serve as a consistent point of contact. You will collaborate with... 

Waller Independent School District

Substitute - School Nurse (RN, LVN, or EMT/Paramedic)(26-27) Job at Waller Independent School District

 ...School Nurse The School Nurse must implement comprehensive program of health services for assigned campus(es), provide health services...  ...information resulting from the day's events. Nurse Substitute (LVN or RN) - Per Day - Long-term* LVN or RN Certification - $ 120.00 -... 

Centene Corporation

LTSS Service Care Manager - Behavioral Health Job at Centene Corporation

 ...Position Purpose: Develops, assesses and coordinates holistic care management activities, with primary focus and support towards populations with significant mental/behavioral health needs, to enable quality, cost-effective healthcare outcomes. Evaluates member service... 

Unilever

UFH Program & Operations Manager, US Hub Job at Unilever

Job Title: UFH Program & Operations ManagerLocation: Trumbull R&D, CTWork Level: 2BWant to Change The World?Unilever is one of the worlds leading suppliers of Beauty, Personal and Home Care products selling in over 190 countries and used by 3.7 billion consumers every... 

Domino's Pizza

Pizza Delivery Driver Job at Domino's Pizza

 ...Domino's Pizza is hiring immediately for Delivery Drivers to join their team! Job type: Full time and Part time, Permanent Do you know why Domino's Pizza hires so many drivers? Well, aside from the fact that our delicious pizza is the perfect solution for dinner...