[Remote] Senior Data Engineer
Note: The job is a remote job and is open to candidates in USA. LanceSoft, Inc. is seeking a Senior Data Engineer to support their data engineering initiatives. The role involves defining testing scopes, designing testing strategies, and collaborating with cross-functional teams to ensure high-quality data deliverables.
Responsibilities
- Define the end-to-end testing scope based on solution architecture and project documentation
- Design and implement a comprehensive testing strategy and plan aligned with organizational QA standards
- Develop and maintain test scripts and frameworks for the Redshift serverless platform
- Perform testing across key technologies, including: AWS Redshift, AWS DMS (Data Migration Service), AWS Glue, PySpark Deequ, Event Bridge, Data Lakes, Python-based data pipelines, Apache Airflow, dbt (data build tool)
- Build and implement automated testing solutions to ensure: End-to-end data validation, Data ingestion accuracy, Transformation logic integrity, Data pipeline reliability
- Conduct test coverage analysis and ensure adequate validation across all data engineering workflows
- Prepare and manage test data
- Review and provide feedback on: Solution architecture, Data models, Design and technical documentation
- Collaborate with cross-functional teams (Data Engineering, BI, DevOps, Product) to: Identify testing impacts, Mitigate risks, Ensure high-quality deliverables
Skills
- Proven experience in data engineering testing / data QA / ETL validation
- Strong hands-on experience with AWS data services (Redshift, Glue, DMS)
- Proficiency in Python for test automation and validation
- Experience with Airflow and orchestration testing
- Hands-on experience with dbt and data transformation validation
- Familiarity with CDK for infrastructure validation
- Experience in BI testing in Quicksuite will be highly beneficial
- Experience with data quality tools such as PySpark Deequ or similar
- Strong understanding of: Data warehousing concepts, ETL/ELT pipelines, Data validation techniques (schema, reconciliation, anomaly detection)
- Experience designing enterprise-level test strategies for data platforms
- Knowledge of CI/CD pipelines for data and test automation
- Experience working in Agile / Scrum environments
- Familiarity with data observability frameworks
Company Overview