Technical Consultant- Databricks – Amazon Store
Role Summary
We are seeking a Data Engineer with hands-on experience in Databricks and modern data engineering practices. This role will focus on designing, developing, and maintaining data pipelines, ensuring data quality, and enabling advanced analytics across the enterprise. The ideal candidate will have a strong foundation in SQL, Python, and Spark, with experience across the Databricks Lakehouse ecosystem and familiarity with modern ELT, transformation, and data governance tools.
Responsibilities- Design, develop, and maintain ETL/ELT pipelines and Databricks notebooks for data ingestion, transformation, and reporting.
- Implement and optimize Spark-based workflows using Databricks Workflows, Delta Lake, and Unity Catalog.
- Leverage Databricks partner tools (e.g., Fivetran, dbt, Coalesce) for scalable data ingestion, transformation, and orchestration.
- Conduct data reconciliation across source, staging, and target systems to ensure accuracy and completeness.
- Build and maintain SQL- and Python-based automation frameworks for regression, smoke, and integration testing.
- Validate structured and semi-structured data formats including CSV, JSON, XML, Parquet, Avro, and fixed-length files.
- Review and refactor existing code for performance, scalability, and maintainability.
- Support workflow orchestration using tools such as Airflow, Azure Data Factory, or Control-M.
- Collaborate with data governance and cataloging teams leveraging tools such as Alation or Unity Catalog to improve discoverability and lineage tracking.
- Experience with cloud platforms (Azure, AWS, or GCP).
- Minimum 2 years of experience with Databricks, Spark, Python, SQL, and Unity Catalog.
- Strong proficiency in SQL (joins, CTEs, window functions, query optimization, stored procedures).
- Solid experience with PySpark, pandas, and Python scripting for data manipulation and validation.
- Familiarity with medallion architecture and data warehouse concepts, including star/snowflake schemas, fact/dimension modeling, and reporting logic.
- Experience validating flat files (CSV, TSV, TXT) including delimiter handling, header checks, null value handling, and schema enforcement.
- Proven ability to interpret and implement complex transformation logic from source to target systems.
- Strong problem-solving, analytical, and debugging capabilities.
- Excellent written and verbal communication skills for interfacing with developers, clients, and technical stakeholders.
- Ability to work independently while collaborating effectively in a team-oriented environment.
- Exposure to Databricks ecosystem tools such as Fivetran, dbt, Coalesce, and Alation.
- Experience building CI/CD pipelines and DevOps practices for data engineering.
- Knowledge of data governance, metadata management, and security frameworks (e.g., GDPR, SOC2, Unity Catalog RBAC).
- Familiarity with data quality frameworks (e.g., Great Expectations, Deequ).