Data Engineer II, PySpark / Databricks

Data Engineer II, PySpark/Databricks This is a remote position. Ad Hoc is a technology company that empowers organizations to deliver scalable, impactful digital services. Using modern, agile methods, our team creates products that meet people's needs and transform their experience of government. Our collaborations have shaped some of the defining moments in public‑sector service delivery. We’ve helped build products that connect Veterans to tailored services, help millions access affordable health care, and support important programs like Head Start. As we work with agencies to deliver critical services, we’re also changing how the government approaches technology. Our culture, communications, and tools are built for remote work, enabling us to bring together top talent nationwide. At Ad Hoc, remote life empowers our teams to design work environments that fit their lives and that foster flexibility and collaboration to achieve positive outcomes for our customers. Ad Hoc values acceptance, accountability, and humility. We aren’t heroes. We learn from our mistakes and improve the process for the next time. We build small, inclusive teams to collaborate closely with our partners to solve the right problems and deliver software that works. The Federal Civilian business unit supports many customers spanning the federal, commercial, and nonprofit space. Our customers include NASA, the General Services Administration, Office of Personnel Management, the Library of Congress, Health & Human Services, and the FDIC. We partner with these agencies to build new capabilities, deliver products, establish data as a strategic asset for informed decision‑making, modernize legacy systems, and build the digital service infrastructure necessary to scale their mission impact. This role is on a program within Health & Human Services. Primary Responsibilities:

• Build and maintain PySpark data pipelines in the Databricks environment
• Optimize Spark jobs performance and resource usage, identifying and addressing bottlenecks and inefficiencies in backend systems
• Design, develop, and maintain high‑quality backend software components and services, ensuring functionality, performance, and scalability
• Research and build proof of concepts in the data space
• Write clean, well‑structured, and maintainable code, adhering to established coding standards and best practices
• Perform thorough code reviews, providing constructive feedback to peers and identifying potential risks or areas for improvement
• Debug and resolve defects, proactively identifying and addressing potential issues before they impact users
• Create and maintain comprehensive technical documentation
• Actively participate in Agile ceremonies, such as stand‑ups, sprint planning, and retrospectives, ensuring effective communication and collaboration across the team
• Assist in the estimation, prioritization, and planning of development tasks, ensuring projects are delivered on time and within budget
• Continuously evaluate and recommend new dataframe related technologies, frameworks, and tools, helping to drive innovation and keep the team up‑to‑date with industry trends
• Engage in ongoing professional development to stay current with industry best practices, and share knowledge and insights with the team as appropriate
• Assist in the implementation and maintenance of security, compliance, and governance policies within the Databricks and AWS environment to ensure adherence to industry standards and regulatory requirements

Basic Qualifications:

• Bachelor’s degree and 8 years of experience
• Strong experience with Python / Apache Spark
• Solid understanding of data modeling, ETL process, and distributed computing
• Bachelor’s degree in Computer Science, Computer Engineering or related field
• Strong understanding of software design patterns, data structures, and algorithms
• Experience with Agile development methodologies
• Ability to work independently as well as in a team
• Strong problem‑solving and analytical skills
• Strong verbal and written communication skills
• Related experience in analytic programming, data extraction, querying databases/data warehouses and data analysis

Preferred Qualifications:

• AWS Experience (S3, EC2, Glue, Lambda, etc)
• R experience
• Professional Databricks/Apache Spark Certification(s)
• SAS experience

Benefits:

• Company‑subsidized health, dental, and vision insurance
• Flexible PTO
• 401(k) with employer match
• Paid parental leave after one year of service
• Employee Assistance Program

Ad Hoc LLC is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, national origin, ancestry, sex, sexual orientation, gender identity or expression, religion, age, pregnancy, disability, work‑related injury, covered veteran status, political ideology, marital status, or any other factor that the law protects from employment discrimination. We value the unique skills gained through military service and encourage veterans and transitioning service members to apply. In support of various state and city equal pay transparency laws, Ad Hoc job descriptions feature the starting range we reasonably expect to pay to candidates who would join our team with little to no need for training on the responsibilities we’ve outlined above. Actual compensation is influenced by a wide range of factors including but not limited to skill set, level of experience, and responsibility. The range of starting pay for this role is $90k‑$110k. Our recruiters will be happy to answer any questions you may have, and we look forward to learning more about your salary requirements. https://adhoc.team/ Apply tot his job Apply To this Job