Building successful AI apps is hard. There are plenty of statistics out there showing failure rates ranging from 65% to 95% of AI agents failing in production.

The main reasons are:

• Unclear requirements as to what the application or agent should do. The C-suite says "Do AI" without clearly defining what that means.
• The UI for AI is a chatbot, so there are infinite ways a user can interact with the AI
• AI apps are non-deterministic, so traditional testing methods no longer work

These problems have given rise to a new discipline - Eval Engineering, the use of AI powered evaluations to test your AI. This is both a function of AI engineers, and also fast becoming a job title in its own right, like the SDET of AI engineering.

This demo heavy session covers the role of the eval engineer. It looks at:

• What are Evals?
• Defining requirements for your AI application
• Observing AI apps and analyzing failures
• Coding and defining failure themes
• Building eval prompts
• Creating datasets of inputs and outputs for testing your app in a CI/CD pipeline

By the end of this session, you will know enough to start your journey building your own evals and fixing up your AI apps.