Engineer | DATA

Vedi l’offerta su Go Lavoro: golavoro.it/annuncio-lavoro/8iu4v0

Our partner is looking for an AI Evaluation Engineer based in Italy. As a founding AI Evaluation Engineer, you will design and build the evaluation, testing, and observability infrastructure that ensures production-grade AI agents perform reliably, safely, and efficiently. You will develop automated testing systems for complex reasoning, multi-turn conversations, and agent interactions involving APIs, databases, and code execution. Working across AI engineering, software testing, and infrastructure, you will create scalable evaluation pipelines that go beyond traditional QA and manual prompt testing.
You will establish robust regression testing, statistical evaluation methods, and adversarial scenarios to identify weaknesses before deployment. This is an opportunity to take ownership of an evaluation architecture from the ground up while working with modern AI technologies in a fully remote, globally distributed environment. Your work will directly contribute to the reliability, security, and continuous improvement of advanced AI systems.
Design, build,



and maintain automated CI/CD testing pipelines to validate AI agent behavior before deployment and throughout the development lifecycle.
Implement fast, cost-effective, deterministic evaluation checks before introducing LLM-as-a-judge approaches, including JSON schema validation, regular expressions, API status checks, and embedding similarity.
Create comprehensive test suites for multi-turn agent conversations, evaluating context retention, state consistency, reasoning behavior, and safeguards against infinite tool-calling loops.
Design and implement adversarial testing scenarios to assess resilience against prompt injection, malformed inputs, unexpected tool responses, and other potential security vulnerabilities.
Build statistical evaluation frameworks to measure non-deterministic AI behavior using pass rates across multiple runs, confidence intervals, and other quantitative reliability metrics.
Integrate evaluati

Vedi l’offerta su Go Lavoro: golavoro.it/annuncio-lavoro/8iu4v0