The Calibration Bug That Taught Our Fraud Agent
Discover how a fraud investigation agent built on LangGraph and TigerGraph encountered a critical calibration bug.
Stock photo for illustration only, not from the actual event
- Utilized 590,742 credit card transactions from the IEEE-CIS dataset via TigerGraph Hacker House Goa.
- Designed a cyclical agent architecture to evaluate and request additional evidence dynamically.
- Isolated core deterministic logic from LLM components to ensure strict regulatory compliance.
- Identified a calibration bug that inverted behavioral risk patterns during agent training.
Building an agentic fraud investigator on TigerGraph highlights fascinating architectural challenges in automated financial systems. The project originated from the TigerGraph Hacker House Goa challenge, providing six months of card transactions totaling 590,742 records from the IEEE-CIS dataset without an explicit isFraud flag. Instead, each transaction carried a bank-issued risk score along with 5,565 closed investigation files spanning four months.
The primary task required deep investigation rather than simple classification: determining the nature of fraud, its scale, and appropriate actions. An implicit requirement dictated the system architecture: recording next-best actions and approval routes both before and after requesting additional evidence. This necessitated a cyclical topology moving from retrieval to belief assessment, provisional action, targeted questioning, and revised decision-making.
Stock photo for illustration only, not from the actual event
A crucial architectural choice kept the decision boundary outside the LLM. Every decision requiring regulatory justification resides in a core/ package that strictly prohibits importing LLM libraries. Automated tests verify Abstract Syntax Trees across files in that directory to fail builds if unauthorized imports appear. Consequently, the agent operates end-to-end without any API key, producing bit-identical verdicts and approval routes.
For case memory, the system fused semantic and entity-based searches within a single GSQL query inside the database. Furthermore, Vesta engineered features were compressed using incremental PCA into a 64-dimensional behavior vector stored on every transaction, enabling precise similarity queries grounded in actual data rather than language model approximations.
Developing automated investigation systems that must comply with strict financial regulations emphasizes the necessity of maintaining clear boundaries between generative AI and deterministic business logic. Prohibiting LLMs from accessing regulatory decision boundaries ensures that financial institutions can audit and explain every automated verdict to external regulators, which remains a fundamental requirement for deploying artificial intelligence in banking.
Nevertheless, a profound calibration bug emerged during likelihood ratio measurements. When replaying closed investigations, the extractor incorrectly learned that transactions on known devices in known regions were significantly more likely to be fraudulent, while large purchases from brand-new devices indicated legitimacy—a completely inverted reality.
The root cause stemmed from population bias within the closed-case files. Cleared cases were anomalies that turned out legitimate, whereas confirmed fraud cases often originated from minor unrecognized charges reported by cardholders. Fixing this required incorporating a stratum of ordinary, untouched transactions from the same period to accurately represent non-fraud baselines.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment