Building Trust in Self-Optimizing Agents

AI agents that can detect and correct their own failures – and improve with limited human intervention – are moving closer to practical use. For Chief Data & Analytics Officers (CDAOs), preparing for this next generation of autonomous systems means establishing a trust infrastructure for how agents operate and improve. This foundation should provide the right context, monitor agent performance and behavior, and evaluate outputs so that reinforcement loops support reliable improvement rather than amplify errors.

Against this backdrop, CDAOs from the Gartner San Francisco CDAO Community recently gathered for a Town Hall on building trust in self-optimizing agents. In small, peer-led discussion groups, executives explored the role of trust infrastructure in self-improving AI, the importance of a strong foundation for reinforcement loops, and what organizations can learn as autonomous agents continue to develop.

The program was moderated by Barr Moses, CEO and Co-Founder of Monte Carlo. Community members Alan Ma, SVP, Data & Analytics at Happen Bank; Flo Oswald, VP, Data Strategy and Governance at Rate; and Brandon Thomas, Chief Data Officer at Zions Bancorporation, led the small-group discussions.

Here are the key takeaways from the Town Hall:

1. Trust starts with a strong data and context foundation

The discussion highlighted that trust in self-optimizing agents begins well before an agent makes a decision. Data and analytics executives connected trust to the quality of the underlying data, the processes used to manage it, and the context provided to the agent. As one CDAO shared, “The trust is not about the model. It's the level of the process and quality of the data.”

Participants also emphasized the importance of a common semantic layer and shared terminology for creating a consistent understanding of data and agent behavior. For many executives, the immediate priority is to establish a reliable foundation for AI – one that improves data quality, enriches metadata, defines rules, and provides agents with the context they need to operate effectively.
 

1. Governance will determine how much autonomy is acceptable

CDAOs emphasized the importance of explainability – being able to understand why an agent made a particular decision. They also noted that security, regulatory requirements, and organizational policies will determine which actions an agent can take independently. From this perspective, autonomy is not an all-or-nothing condition. Different components of an agent or process may operate with different levels of autonomy, depending on their risks and applicable controls. As one executive said, “Autonomy is not simple. Some parts of the agent are highly autonomous and other parts are highly restricted by regulations.”

Participants also noted that data regulation and information security requirements are slowing progress in some organizations. The discussion raised a related question about performance expectations: Do organizations hold AI systems to a higher standard than human decision-makers – and should they always? As one CDAO observed, “We hold AI to a higher bar than humans even if we're okay with a human making that mistake.”
 

1. Self-improvement introduces new oversight questions

Self-optimizing agents raise oversight questions that extend beyond their initial deployment. Participants discussed how reinforcement loops should operate, who should define the rewards that shape an agent’s behavior, and how organizations should evaluate updated versions before moving them into production. As one participant asked, “The reinforcement loop is even more important. Who will decide the reward mechanism in this loop?”

Executives also discussed using simulation to evaluate an agent before deployment, while acknowledging that important questions remain about their reliability. One participant questioned when simulations would be reliable to put into production and how organizations should approve each new version of a self-improving agent. Together, these questions highlight that trust infrastructure must address not only an agent’s behavior at launch, but also how that behavior may change as the system continues to learn and improve.
 

1. Most organizations are still building toward this future

Although CDAOs recognize the potential importance of self-optimizing agents, most organizations represented in the discussion were not yet preparing to deploy them. Instead, many are focused on building the foundational capabilities needed for more advanced AI, including improving data quality, strengthening data platforms, enriching metadata, and validating AI-generated queries and models. As one participant described, “At the moment, we are more focused on setting up the right foundation for AI, starting with ensuring that data quality reaches acceptable levels.”
 

To continue the conversation on advanced AI and other critical issues, apply to join a CDAO community. If you are already a community member, sign in to find upcoming opportunities to connect with your peers.