Technical Reviewer - RL Environment Terminal Benchmarking (Agentic AI)
This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.
Role Description
Mercor is hiring a Technical Reviewer on behalf of a leading AI lab to evaluate and refine benchmarking pipelines for reinforcement learning (RL) environments and agentic AI systems. In this role, you’ll be responsible for reviewing environment design, terminal conditions, and evaluation protocols to ensure accuracy, reproducibility, and fairness in benchmarking. You’ll work closely with researchers and engineers to provide technical feedback that strengthens experimental rigor and system reliability.
Qualifications
- Background in reinforcement learning, computer science, or applied AI research
- Experience with RL environments
- Understanding of benchmarking methodologies, terminal conditions, and evaluation metrics for RL tasks
- Comfortable reading and reviewing codebases in Python (PyTorch/TensorFlow a plus)
- Strong critical thinking skills and ability to provide structured technical feedback
- Care deeply about experimental reproducibility, fairness, and standardization in agentic AI
- Detail-oriented and capable of reviewing both theoretical formulations and implementation details
Requirements
- Review RL environments and evaluate terminal conditions for correctness and consistency
- Assess benchmarking pipelines for fairness, reproducibility, and alignment with research objectives
- Provide structured technical feedback on code implementations and documentation
- Collaborate with researchers to refine evaluation metrics and methodologies
- Ensure reproducibility by validating results across different runs, seeds, and hardware setups
- Document findings and recommend improvements for environment design and benchmarking standards
Benefits
- Directly influence the reliability of benchmarking in agentic AI research
- Work on cutting-edge RL environments that test the limits of intelligent agents
- Help establish standards for evaluation and reproducibility in a fast-moving field
- Collaborate with researchers shaping the future of agentic AI systems
Pay & Work Structure
- Classified as a full-time hourly contractor to Mercor
- Paid weekly via Stripe Connect, based on hours logged
- 40 hours/week commitment with flexible scheduling
- Remote and flexible working style