Evaluation Scenario Writer (m/f/d)

We’re looking for someone who can design realistic and structured evaluation scenarios for LLM-based agents. You’ll create test cases that simulate human-performed tasks and define gold-standard behavior to compare agent actions against. You’ll work to ensure each scenario is clearly defined, well-scored, and easy to execute and reuse. You’ll need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Although every project is unique, you might typically:

• Designing structured test scenarios based on real-world tasks.
• Defining the golden path and acceptable agent behavior.
• Annotating task steps, expected outputs, and edge cases.
• Working with devs to test your scenarios and improve clarity.
• Reviewing agent outputs and adapting tests accordingly

• Bachelor's and/or Master’s Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
• Background in QA, software testing, data analysis, or NLP annotation.
• Good understanding of test design principles (e.g., reproducibility, coverage, edge cases).
• Strong written communication skills in English.
• Comfortable with structured formats like JSON/YAML for scenario description.
• Can define expected agent behaviors (gold paths) and scoring logic.
• Basic experience with Python and JS.
• Curious and open to working with AI-generated content, agent logs, and prompt-based behavior.
• You are ready to learn new methods, able to switch between tasks and topics quickly and sometimes work with challenging, complex guidelines.
• Our freelance role is fully remote so, you just need a laptop, internet connection, time available and enthusiasm to take on a challenge.

Nice to Have

• Experience in writing manual or automated test cases.
• Familiarity with LLM capabilities and typical failure modes.
• Understanding of scoring metrics (precision, recall, coverage, reward functions).