Job Summary
We are seeking an experienced Senior Data Engineer with strong expertise in dbt, Snowflake, Apache Airflow, SQL, and Python. This role will be responsible for leading the transition of existing Denodo-based data transformations to dbt, designing scalable data models, building and managing workflow orchestration solutions, optimizing data platforms, and establishing data engineering best practices. The ideal candidate will possess strong technical leadership skills, extensive experience with large-scale data transformation initiatives, and the ability to mentor engineering teams while driving high-quality technical solutions.
Key Responsibilities
- Design, develop, and architect scalable dbt projects, including staging, intermediate, and mart layers.
- Develop complex dbt models using advanced SQL and implement incremental models, snapshots, Jinja macros, custom tests, seeds, and reusable packages.
- Configure and maintain dbt materializations, model dependencies, documentation, and data lineage.
- Lead the migration of existing Denodo-based transformations to dbt and establish development standards and best practices.
- Develop, deploy, and optimize dbt models on Snowflake.
- Implement and maintain Snowflake configurations, including clustering keys, transient tables, warehouse sizing, access controls, roles, and grants.
- Optimize SQL queries to improve performance, scalability, and cost efficiency.
- Troubleshoot and resolve Snowflake performance and data processing issues.
- Design, develop, and maintain Apache Airflow DAGs and workflow orchestration solutions.
- Integrate dbt workflows with Airflow to support automated data pipeline execution.
- Configure pipeline scheduling, dependencies, retries, alerts, monitoring, failure handling, reruns, and backfills.
- Leverage advanced Airflow capabilities, including TaskFlow API, dynamic task mapping, sensors, task groups, and custom operators.
- Develop and optimize complex SQL queries using joins, CTEs, aggregations, and window functions.
- Work with Oracle databases, including schema design, indexing, partitioning, and query optimization.
- Develop Python-based automation tools, utilities, and data validation processes.
- Troubleshoot data quality issues and resolve complex data processing challenges.
- Lead technical design discussions, code reviews, and engineering best practice initiatives.
- Establish coding standards and CI/CD practices.
- Mentor and support data engineering team members.
- Develop technical documentation and operational runbooks.
- Collaborate with architects, business stakeholders, quality assurance teams, and platform teams to deliver scalable data solutions.
Required Qualifications
- 7-10 years of experience in Data Engineering, Analytics Engineering, ETL, or ELT development.
- Minimum 3-4 years of hands-on production experience with dbt and Apache Airflow.
- Expert-level experience designing and developing dbt projects in production environments.
- Strong knowledge of dbt architecture, incremental models, Jinja macros, snapshots, testing frameworks, and documentation.
- Experience developing and maintaining Apache Airflow DAGs from the ground up.
- Proven experience integrating dbt and Airflow within production environments.
- Advanced hands-on experience with Snowflake, including performance optimization and workload management.
- Expert-level SQL development and query optimization skills.
- Strong working knowledge of Oracle databases.
- Proficiency in Python scripting, automation, and data engineering utilities.
- Advanced experience with Git, branching strategies, pull requests, and CI/CD pipelines.
- Ability to lead technical discussions, conduct code reviews, and mentor engineering teams.
- Strong analytical, troubleshooting, and problem-solving skills.
- Experience working on enterprise-scale data transformation and modernization initiatives.
- Ability to work onsite at the client location in Oaks, Pennsylvania.
Preferred Qualifications
- Experience leading or contributing to Denodo-to-dbt migration projects.
- Experience with enterprise ETL modernization or data virtualization migration initiatives.
- Knowledge of dimensional modeling methodologies, including Kimball and Data Vault.
- Experience with Astronomer Cosmos, dbt Cloud, or dbt Semantic Layer.
- Experience running Airflow on Kubernetes, Astronomer, AWS MWAA, or Google Cloud Composer.
- Knowledge of data quality and observability platforms such as Great Expectations, Monte Carlo, or Elementary.
- Experience working in financial services or other regulated environments.
- Experience establishing data engineering standards, governance, and operational best practices.
Certifications
- dbt Analytics Engineering Certification.
- Astronomer Certification.