Data Annotation & AI QA Specialist

Role Overview

As a Data Annotation & AI QA Specialist, you will help build and maintain our "golden benchmark dataset"—the foundational ground-truth dataset used to evaluate AI-generated project briefs, listing claims, and expert proof submissions across our platform.

This role is ideal for students or recent graduates interested in gaining direct experience in AI Quality Assurance, Natural Language Processing (NLP), and Data Operations. You will work at the intersection of human evaluation and machine learning model validation.

Key Responsibilities

• Benchmark Dataset Creation: Annotate, clean, and structure text and multi-modal data pairing complex business requirements with 9-line JSON project briefs.
• Claim & Proof Verification: Audit AI-generated listing claims against raw technical proof records (code snippets, API logs, and execution outputs).
• Quality Assurance & Guidelines: Help refine annotation guidelines and schema rules to ensure high inter-annotator agreement (>95%).
• Model Output Evaluation: Perform systematic human-in-the-loop (HITL) quality audits on outputs generated by fine-tuned LLMs and automated tools.
• Edge Case Flagging: Identify ambiguous, misleading, or hallucinated AI outputs and feed corrected samples back into our training pipeline.

What We’re Looking For

• Educational Background: Currently pursuing or recently completed a Bachelor’s or Master’s degree in Computer Science, Data Science, Information Systems, Linguistics, or a related quantitative field.
• Attention to Detail: Meticulous approach to data precision—you take pride in identifying subtle errors or inconsistencies in data.
• Technical Literacy: Basic familiarity with Python, JSON structures, regular expressions, or spreadsheet/data management tools (Pandas, Excel, Airtable).
• Analytical Thinking: Ability to interpret technical documentation, system logs, and business workflows quickly.
• Communication: Strong written English skills for writing clear annotation instructions and feedback reports.

Bonus Qualifications (Not Required)

• Prior exposure to data annotation platforms (e.g., Label Studio, Prodigy, Scale AI) or QA testing.
• Understanding of basic machine learning concepts (precision, recall, ground truth, fine-tuning).

Benefits & Career Growth

• Hands-on AI Experience: Work directly with state-of-the-art LLM pipelines and real-world training/evaluation datasets.
• Flexible Remote Environment: Work asynchronously from anywhere with structured milestone goals.
• Mentorship & Mentoring: Direct collaboration with senior machine learning and automation engineers.

How to Apply

Apply with your updated Resume / CV and a link to your GitHub, LinkedIn, or portfolio project demonstrating your interest in AI/data projects.