Data Factory Engineer
Job Purpose
The Data Engineer within the Data Factory pillar is responsible for designing, developing, testing, deploying, and operating backend data integration solutions and reusable data platform components. The role supports the current on-premises data estate while enabling the progressive evolution toward Microsoft Fabric and Databricks. Its purpose is to deliver secure, scalable, observable, and high-quality batch and near-real-time data pipelines that make trusted data available for reporting, analytics, data products, and AI use cases.
Key Accountabilities Design, build, test, deploy, and maintain backend data pipelines for ingestion, transformation, orchestration, and delivery across on-premises and cloud environments.
Develop and enhance ETL/ELT solutions for the current on-premises Data Platform, working with relational databases, enterprise data warehouses, flat files, APIs, scheduling tools, and existing integration technologies.
Contribute to the target Data Platform based on Microsoft Fabric and Databricks, using modern Lakehouse patterns and services such as One Lake, Fabric Data Factory, notebooks, Spark, Delta Lake, Databricks Workflows, and SQL endpoints, as applicable to the adopted architecture.
Apply modern data engineering patterns including metadata-driven and configuration-driven pipelines, reusable frameworks, parameterization, incremental processing, change data capture, idempotency, schema evolution, partitioning, and medallion-style data layers.
Translate logical and physical data designs into robust backend implementations, in collaboration with Data Architects, Data Factory Leads, platform engineers, analysts, and business-facing data teams.
Implement automated controls for data quality, reconciliation, lineage, logging, alerting, error handling, retry,
restartability, and operational monitoring.
Optimize data pipelines and storage for performance, scalability, resiliency, and cost efficiency, using appropriate workload patterns for batch, micro-batch, or streaming use cases.
Manage source code through standard Git-based engineering procedures. The use of branching, pull requests, peer review, version control, automated testing, artifact management, and standard CI/CD pipelines for build, release, and deployment is mandatory.
Create automated unit, integration, regression, and data validation tests to reduce production defects and support repeatable releases across development, test, and production environments.
Support secure engineering practices, including secrets management, least-privilege access, environment segregation, secure connectivity, and compliance with enterprise data governance and security standards.
Produce and maintain technical documentation for pipelines, mappings, transformation rules, interfaces, dependencies, deployment procedures, runbooks, and operational support.
Investigate incidents and data issues through structured root-cause analysis, implement permanent corrective actions, and contribute to continuous improvement of engineering standards and platform reliability.
Participate in Agile delivery ceremonies, estimation, backlog refinement, technical reviews, release planning,
and knowledge-sharing activities within the Data Factory pillar.
Leverage approved AI-assisted engineering capabilities to accelerate the software development lifecycle, including code generation and refactoring, technical documentation, test-case creation, defect analysis, code review, and release preparation. Ensure that all AI-generated outputs are subject to appropriate human validation, security controls, quality standards, and standard CI/CD governance before deployment.
Qualifications
Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related discipline, or equivalent professional experience.
5–7 years of professional experience in Data Engineering, backend data integration, ETL/ELT development, or enterprise Data Platform delivery within complex environments.
Strong SQL skills, including data analysis, query optimization, data manipulation, dimensional or relational modelling, and troubleshooting of large data workloads.
Hands‐on programming and scripting experience, preferably with Python and Py Spark; knowledge of Java or Scala is an advantage.
Experience with on‐premises data integration and database technologies. Knowledge of Oracle, Microsoft SQL Server, SSIS, enterprise schedulers, file‐based interfaces, APIs, or comparable technologies is relevant to the current landscape.
Knowledge of modern cloud and Lakehouse engineering patterns, with practical exposure to Microsoft Fabric and/or Databricks. Experience with Spark, Delta Lake, One Lake, Fabric Data Factory, Azure Data Factory, notebooks, or workflow orchestration is strongly preferred.
Mandatory experience managing code through Git‐based repositories and standard CI/CD pro
The Data Engineer within the Data Factory pillar is responsible for designing, developing, testing, deploying, and operating backend data integration solutions and reusable data platform components. The role supports the current on-premises data estate while enabling the progressive evolution toward Microsoft Fabric and Databricks. Its purpose is to deliver secure, scalable, observable, and high-quality batch and near-real-time data pipelines that make trusted data available for reporting, analytics, data products, and AI use cases.
Key Accountabilities Design, build, test, deploy, and maintain backend data pipelines for ingestion, transformation, orchestration, and delivery across on-premises and cloud environments.
Develop and enhance ETL/ELT solutions for the current on-premises Data Platform, working with relational databases, enterprise data warehouses, flat files, APIs, scheduling tools, and existing integration technologies.
Contribute to the target Data Platform based on Microsoft Fabric and Databricks, using modern Lakehouse patterns and services such as One Lake, Fabric Data Factory, notebooks, Spark, Delta Lake, Databricks Workflows, and SQL endpoints, as applicable to the adopted architecture.
Apply modern data engineering patterns including metadata-driven and configuration-driven pipelines, reusable frameworks, parameterization, incremental processing, change data capture, idempotency, schema evolution, partitioning, and medallion-style data layers.
Translate logical and physical data designs into robust backend implementations, in collaboration with Data Architects, Data Factory Leads, platform engineers, analysts, and business-facing data teams.
Implement automated controls for data quality, reconciliation, lineage, logging, alerting, error handling, retry,
restartability, and operational monitoring.
Optimize data pipelines and storage for performance, scalability, resiliency, and cost efficiency, using appropriate workload patterns for batch, micro-batch, or streaming use cases.
Manage source code through standard Git-based engineering procedures. The use of branching, pull requests, peer review, version control, automated testing, artifact management, and standard CI/CD pipelines for build, release, and deployment is mandatory.
Create automated unit, integration, regression, and data validation tests to reduce production defects and support repeatable releases across development, test, and production environments.
Support secure engineering practices, including secrets management, least-privilege access, environment segregation, secure connectivity, and compliance with enterprise data governance and security standards.
Produce and maintain technical documentation for pipelines, mappings, transformation rules, interfaces, dependencies, deployment procedures, runbooks, and operational support.
Investigate incidents and data issues through structured root-cause analysis, implement permanent corrective actions, and contribute to continuous improvement of engineering standards and platform reliability.
Participate in Agile delivery ceremonies, estimation, backlog refinement, technical reviews, release planning,
and knowledge-sharing activities within the Data Factory pillar.
Leverage approved AI-assisted engineering capabilities to accelerate the software development lifecycle, including code generation and refactoring, technical documentation, test-case creation, defect analysis, code review, and release preparation. Ensure that all AI-generated outputs are subject to appropriate human validation, security controls, quality standards, and standard CI/CD governance before deployment.
Qualifications
Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related discipline, or equivalent professional experience.
5–7 years of professional experience in Data Engineering, backend data integration, ETL/ELT development, or enterprise Data Platform delivery within complex environments.
Strong SQL skills, including data analysis, query optimization, data manipulation, dimensional or relational modelling, and troubleshooting of large data workloads.
Hands‐on programming and scripting experience, preferably with Python and Py Spark; knowledge of Java or Scala is an advantage.
Experience with on‐premises data integration and database technologies. Knowledge of Oracle, Microsoft SQL Server, SSIS, enterprise schedulers, file‐based interfaces, APIs, or comparable technologies is relevant to the current landscape.
Knowledge of modern cloud and Lakehouse engineering patterns, with practical exposure to Microsoft Fabric and/or Databricks. Experience with Spark, Delta Lake, One Lake, Fabric Data Factory, Azure Data Factory, notebooks, or workflow orchestration is strongly preferred.
Mandatory experience managing code through Git‐based repositories and standard CI/CD pro