The rapid integration of agentic AI into the banking sector promises unprecedented efficiency, yet it brings a subtle, psychological risk that many institutions are only beginning to grasp. Priya Jaiswal, a leading expert in banking risk management and international finance, joins us to discuss the phenomenon of automation complacency—a state where human oversight fails not because of incompetence, but because the technology appears too reliable. With years of experience navigating market volatility and portfolio management, Jaiswal provides a deep dive into why even the most seasoned professionals might “tune out” when faced with near-perfect automated systems. This conversation explores the cognitive traps of modern fintech, the hidden costs of verification, and the strategies banks must employ to ensure their “human in the loop” remains vigilant against the rare but catastrophic errors that AI can produce.
When automated systems appear reliable for long periods, human reviewers often struggle to maintain vigilance over rare errors. Why is the human brain so poorly equipped for this type of oversight?
The core of the problem lies in what we call automation complacency, a psychological state where the brain essentially adapts to a system’s high reliability by lowering its own defensive posture. If you come to work every single day and perform a task where the AI is right 99.9% of the time, your neural pathways begin to see the review process as a mere formality or a boring box-checking exercise. Patrick Hall from George Washington University points out that our brains are simply not wired to maintain high-alert vigilance for that “one-in-a-million” error when the previous 999,999 instances were perfect. This isn’t a lack of professionalism; it is a predictable cognitive response where experts use their pattern recognition skills to assess work quickly, often signaling “acceptable” if the tone and structure look right, without triggering a deeper analysis. Paradoxically, those who try to force themselves into constant, high-level vigilance often end up exhausted, which further degrades their judgment and makes them even more susceptible to missing a critical, expensive mistake.
A recent industry survey suggests that bank professionals are increasingly worried about human-centric factors in AI safety. What does the data tell us about the specific fears weighing on the minds of modern bankers?
The data from the Wolters Kluwer survey of 230 U.S. bank professionals is quite telling, as it reveals that “automation bias”—the tendency to defer to algorithms rather than exercise independent judgment—is the top concern for 34% of respondents. It is fascinating because even experts like Elaine Duffus expected “skills gaps” or “misaligned incentives” to lead, yet those came in lower at 21% and 27% respectively. There is a growing realization that when you incentivize employees to speed through high volumes of work, they will naturally lean on the AI as a crutch to meet those productivity targets. We are also seeing a significant 17% of professionals worried about “shadow AI,” which suggests that unauthorized tools are entering the workflow, further complicating the safety framework. These numbers highlight a “glazed-eyed employee” hazard that has existed for a long time but is now being supercharged by the sheer speed and convincing nature of agentic AI outputs.
There is often a “verification burden” associated with AI that companies are hesitant to discuss. Can you elaborate on how checking an agent’s work might actually negate the time-savings promised by the technology?
This is the “fun part of the story” that many tech advocates tend to sweep under the rug to keep the narrative of AI efficiency alive. Consider a scenario where an agentic AI model can prepare complex loan documents in a mere three minutes, which sounds revolutionary on paper. However, if a human reviewer then has to spend 11 hours meticulously verifying every data point to ensure accuracy, the bank is actually spending more time and money than if a person had just done it the old-fashioned way. There is immense commercial pressure to report the three-minute processing time as a success, while the hundreds of hours of human verification are conveniently ignored in the ROI reports. If we aren’t careful, the drive to save money by eventually removing humans from the loop will lead to a system where we have high-speed processing with zero meaningful oversight, creating a ticking time bomb of financial risk.
As banks move toward more autonomous systems, how do human managers maintain accountability without “outsourcing” their professional judgment to the software?
This is perhaps the most dangerous frontier, as Ryan Hildebrand from Bankwell Bank noted: the real danger isn’t the AI agents themselves, but rather their managers outsourcing the judgment for which they are ultimately accountable. In high-stakes areas like risk and credit, nuance is everything, and seasoned bankers still possess a depth of contextual knowledge that an algorithm cannot replicate. To prevent this outsourcing of judgment, institutions must ensure that the human “in the loop” feels they have true ownership of the mission rather than just acting as a secondary validator. If the employee feels the AI is a tool helping them achieve a critical sign-off, they are much more likely to stay engaged with the process. Accountability must be reinforced through a culture where the human’s role is seen as the primary safeguard, not a redundant hurdle in a high-speed digital assembly line.
What practical strategies can financial institutions implement to foster “sustainable skepticism” among their staff?
To combat the natural drift toward complacency, companies need to get creative with how they keep their reviewers’ brains “switched on,” such as periodically injecting intentional errors into the AI’s output. By doing this and tracking whether the reviewers catch these “planted” mistakes, you can provide anonymous feedback to the whole team, which serves as a sharp reminder that the system is not infallible. I also highly recommend Leigh Coney’s idea of rotating the role of a weekly “AI skeptic” who is specifically tasked with finding verification errors that others might have missed. Implementing confidence scores—where reviewers rate their certainty before and after a manual check—can also expose gaps where high confidence is masking low accuracy. Finally, bringing in external industry specialists can provide a fresh set of eyes, as these outsiders lack the daily familiarity that breeds complacency and are often the only ones who can see the errors hiding in plain sight.
What is your forecast for the future of human-AI collaboration in high-risk banking environments?
I believe we are heading toward a period of “corrective friction,” where banks will realize that the “human-in-the-loop” model is significantly more expensive and labor-intensive than originally advertised. Within the next few years, the industry will move away from the “black box” approach and toward systems that require humans to perform “mission-critical” sign-offs at specific, high-risk junctions rather than reviewing every minor output. We will see a shift in hiring, where the most valued bank employees won’t be those who can process data the fastest, but those who possess the “sustainable skepticism” needed to challenge an AI that looks and sounds perfect. Ultimately, the banks that survive the next wave of automation will be those that treat human attention as their most precious—and most limited—resource, rather than an infinitely scalable safety net.
