Generalist Evaluator Expert

 Seeking detail-oriented writing experts to contribute to a high-impact AI research project with a leading lab. Freelancers will author prompt–golden answer pairs that train and evaluate advanced language models. This is a short-term, flexible opportunity for professionals with strong academic backgrounds and a knack for instructional clarity. Ideal for those who enjoy distilling complex concepts into well-crafted text.

Job Details:

Design and Optimize Prompts: Create detailed prompts with multiple constraints and instructions.

Define and Document Evaluation Standards: Establish high-level expectations for correct responses in general consumer contexts, and develop comprehensive rubric.

Conduct Model Testing and Grading: Run prompts through models and assess preliminary outputs against expectations.

Support Benchmarking and Quality Assurance: Collaborate in QA review processes to ensure prompt tasks and rubrics meet rigor, maintaining consistency and reliability before integration into official benchmarks.

Minimum Qualifications:

BS or BA from a reputable institution completed or in progress

Strong writing and critical thinking skills.

Ability to work independently and meet deadlines.

Significant familiarity with Chat GPT or similar tools for personal decision-making or hobbies / general interests.

Preferred Qualifications:

• 2+ years of experience in teaching or research.

Application & Onboarding Process:

Complete an AI-led interview, this should take around 15 minutes.

Complete a 45-minute written assessment that will guide you through writing rubrics.

If selected, you will be invited to work on the project.

More Details About This Role:

This is a remote and asynchronous role — work on your own schedule.

Expect to contribute at least 20 hours per week.

Expect a commitment of around 1 month.

You’ll be working in a structured project environment with clear goals and tools.