Ayush Rai-Senior AI Engineer | RLHF & LLM Evaluation Specialist | Bhopal, India

Python, LangGraph, FastAPI, Qdrant, MongoDB, Streamlit

• Built an agentic RAG system that dynamically routes queries across indexed document retrieval, general LLM reasoning, and real-time web search via a modular LangGraph orchestration pipeline.
• Implemented query analysis, route classification, relevance grading, query rewriting, answer generation, and Tavily search fallback integrated with Qdrant vector search, MongoDB session memory, and OpenAI-compatible LLMs.

• Executed RLHF and SFT workflows for frontier LLM training: evaluated model responses for correctness, reasoning quality, and instruction adherence; ranked competing outputs using structured quality criteria.
• Delivered advanced prompt engineering and code-generation datasets across Python and JavaScript, improving contextual accuracy by 25% and reducing iteration cycles by 10%.
• Conducted systematic error analysis identifying hallucinations, factual inaccuracies, and logical inconsistencies and authored detailed rationales to calibrate preference selections and maintain dataset quality.

• Designed and shipped production AI systems spanning LLM evaluation pipelines, RAG architectures, and AI-assisted decision platforms for startup and enterprise clients, contributing to 40L+ in cumulative client revenue.
• Built and operated RLHF and human-feedback workflows for frontier AI evaluation programs, delivering structured datasets, grading rubrics, error taxonomies, and alignment feedback loops that improved AI output reliability by 30%.
• Led AI readiness audits and maturity assessments covering data constraints, model risks, and deployment feasibility for 10+ client engagements; worked directly with founders and CXOs to align AI strategy with business outcomes.