Awas: Scoping a Scam Ad Checker Around Quotes
Developer shares how the Awas scam ad checker built for the Nebius Global AI Hackathon was rescoped after its web coverage gate missed benchmarks.
Stock photo for illustration only, not from the actual event
- Awas was built for the Nebius Global AI Hackathon to analyze overseas job ads.
- The coverage gate failed its benchmark, surfacing 8 out of 20 cases instead of the target 12.
- The developer shifted the design so verdicts rely strictly on rules quoted from official pages.
- Awas remains a research prototype available for testing at awas.edycu.dev.
A job offer lands in a family WhatsApp group featuring a Cambodian posting with a 15,000 IDR/month salary, a tourist visa, and covered tickets. Someone in the family needs to decide tonight whether to let their relative board the plane. While rules to stop them exist across various ministry, police, and embassy pages, no one can read all of them in a single evening.
To address this, Edy Cutjong built Awas for the Nebius Global AI Hackathon, designed to answer chat queries with pages instead of a simple score. Users paste ad text, and Awas returns an evidence file with one of three outcomes: STOP, VERIFY FIRST, or NO RED FLAGS FOUND, explicitly noting that it is never a guarantee of safety.
Stock photo for illustration only, not from the actual event
Before the first benchmark run, the developer set a pass condition requiring Tavily to surface a relevant advisory or news item for at least 12 out of 20 known scam cases. This promise formed the core of the original pitch. However, when tested, the system successfully surfaced evidence for only 8 out of 20 cases.
A secondary issue compounded the problem: published reports almost never name the specific recruiters. Out of the benchmark cases, only 1 in the development set, 0 in the first tranche, and 1 in the second tranche named a company at all, making a company-matched success rate impossible to measure consistently.
The failure of web search agents to surface specific recruiter names highlights the gap between real-world fraud patterns and published institutional advisories. Government reports often focus on broad risk categories rather than specific operational entities. Pivoting from dynamic web search retrieval to hard-coded rules derived from official policy pages represents a necessary architectural shift to ensure reliability in safety-critical automated tools.
Faced with the choice of staying quiet or rescoping, the developer chose to rescope and publish the exact metrics. Verdicts now rely entirely on rules quoted from official pages, while web searches serve exclusively as contextual background for families rather than direct inputs for rule enforcement.
The benchmark's second tranche was frozen and hashed prior to running as a Nebius Serverless Job against the production application. All metrics are documented in DEMO.md and the bench/ directory, with complete call records available in FEEDBACK.md in the repository.
Ultimately, Awas is a research prototype rather than legal advice or an official government service. Anyone who encounters suspicious overseas job ads in group chats can remove phone numbers and names, paste the text at awas.edycu.dev, and provide feedback on what the system missed.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment