[Remote] Data Engineering with Python
Note: The job is a remote job and is open to candidates in USA. Diverse Lynx is seeking a Data Engineer with strong expertise in Python-based data engineering and CI/CD automation using GitHub Actions. The candidate will play a critical role in building scalable data pipelines, automating deployments, and enabling high-quality data delivery across cloud platforms.
Responsibilities
- Design, build, and maintain scalable data pipelines using Python
- Develop ETL/ELT workflows for batch and near real-time data processing
- Work with structured and unstructured data across cloud/platform ecosystems
- Implement data transformation, validation, and quality frameworks
- Build and maintain CI/CD pipelines using GitHub Actions to automate: Code builds and validations, Unit/integration testing, Deployment of data pipelines and services
- Implement branching strategies, code reviews, and version control using Git
- Enable automated testing and release management for data workflows
- Integrate CI/CD pipelines with cloud platforms and data services
Skills
- Strong expertise in Python-based data engineering
- CI/CD automation using GitHub Actions
- Building scalable data pipelines
- Automating deployments
- Enabling high-quality data delivery across cloud platforms
- Designing, building, and maintaining scalable data pipelines using Python
- Developing ETL/ELT workflows for batch and near real-time data processing
- Working with structured and unstructured data across cloud/platform ecosystems
- Implementing data transformation, validation, and quality frameworks
- Building and maintaining CI/CD pipelines using GitHub Actions
- Automating code builds and validations
- Unit/integration testing
- Deployment of data pipelines and services
- Implementing branching strategies, code reviews, and version control using Git
- Enabling automated testing and release management for data workflows
- Integrating CI/CD pipelines with cloud platforms and data services
Company Overview
Company H1B Sponsorship