AI Agent Evaluation Analyst

For an AI lab we are looking for AI Agent Evaluation Analyst to train an AI model (Large Language Model - LLM).

You help AI to make sense of the world. As consultant, you may be invited to take part in online projects to train the model in your domain of expertise.

This flexible role accommodates both experts seeking part-time engagement (minimum few hours/week) and those interested in full-time opportunities

• Reviewing evaluation tasks and scenarios for logic, completeness, and realism.
• Identifying inconsistencies, missing assumptions, or unclear decision points.
• Helping define clear expected behaviors (gold standards) for AI agents.
• Annotating cause-effect relationships, reasoning paths, and plausible alternatives.
• Thinking through complex systems and policies as a human would to ensure agents are tested properly.
• Working closely with QA, writers, or developers to suggest refinements or edge case coverage.

• Excellent analytical thinking: Can reason about complex systems, scenarios, and logical implications.
• Strong attention to detail: Can spot contradictions, ambiguities, and vague requirements.
• Familiarity with structured data formats: Can read, not necessarily write JSON/YAML.
• Can assess scenarios holistically: What's missing, what’s unrealistic, what might break?
• Experience with policy evaluation, logic puzzles, case studies, or structured scenario design.
• Background in consulting, academia, olympiads (e.g. logic/math/informatics), or research.
• Exposure to LLMs, prompt engineering, or AI-generated content.
• Familiarity with QA or test-case thinking (edge cases, failure modes, “what could go wrong”).

Application Process:

• If you are being selected you will by invited to an interview by Mindrift.