AI Agents in Action Foundations for Evaluation and Governance 2025

For organizations adopting AI, understanding and clearly defining the operational context is essential to ensure effectiveness in actual deployment.An example of an operational context could be a fraud detection agent in online banking. The agent accesses transaction data and user history, but cannot fully observe external factors like user intentions or hidden fraud tactics. It functions stochastically, with outcomes influenced by unpredictable variables such as varying fraud methods or user behaviours, rather than guaranteed results. The setup is sequential, refining risk assessments with each detection. Operating in a fast-changing environment, it requires continuous monitoring by human reviewers and other security systems. For organizations adopting AI, understanding and clearly defining the operational context is essential to ensure effectiveness in actual deployment settings. Potential issues can be mitigated by adjusting agent parameters, like autonomy and authority, and/or by constraining the context in which the agent operates. Examples include limiting a robot to a controlled zone or confining a software agent to a sandbox. An AI agent’s role, autonomy, authority, predictability and operational context collectively shape its overall impact, defined as the degree of benefit or harm it may generate. Highly autonomous, authorized and non-deterministic behaviour in a complex operational context may deliver strong performance but also carry greater risks. The following example illustrates how these dimensions can be applied in practice through the classification of a basic AI agent, a robot vacuum cleaner. AI Agents in Action: Foundations for Evaluation and Governance 15