I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data
Frustrated by anti-bot walls and layout shifts, a developer switches to official Romanian open registry data to power an AI agent MCP server.
Stock photo for illustration only, not from the actual event
- Abandoned traditional web scraping prone to bot blocks and layout changes
- Utilizes official Romanian open-data registry containing 4.2 million companies refreshed monthly
- Built a self-hosted FastMCP server on a homelab box to feed clean data to AI agents
- Exposes three dedicated tools for company search, contact extraction, and domain validation
For a long time, building B2B lead-generation tools followed the standard approach: scrape a directory, fight anti-bot protections, re-scrape whenever layouts changed, and hope the data remained current. Everything changed upon discovering that the Romanian government publishes its entire company registry as open data—encompassing 4.2 million firms, refreshed monthly, complete with legal statuses, activity codes, and directors, requiring zero scraping.
This post chronicles that shift, detailing how an official open-data snapshot transformed into a self-hosted Model Context Protocol (MCP) server equipped with three tools tailored for AI agents. Building a targeted directory of Romanian companies—such as every SRL in Cluj holding a CAEN software service code—relies on official open-data programs that publish monthly CSV snapshots directly from the source.
The entire registry is structured and licensed for reuse, completely bypassing logins, API keys, rate limits, and anti-bot systems. Fragility disappears since the government maintains the data freshness, allowing a simple monthly re-download of the snapshot. However, open data does not equal clean data; these CSV files are massive (the firm dataset alone spans roughly 690 MB), utilize caret delimiters, carry BOMs, encode Romanian diacritics, and store statuses and activities as raw codes requiring nomenclator tables for decoding. That represents the core engineering workload.
Stock photo for illustration only, not from the actual event
Transitioning from web scraping to utilizing official government open data represents a highly robust and sustainable architecture for data engineering tasks. By pairing this approach with Model Context Protocol (MCP), developers can seamlessly expose structured national registry data to AI agents in a standardized, secure manner, significantly reducing maintenance overhead caused by unstable web user interfaces.
The established pattern scales to any CKAN-backed government portal, encapsulated within a lightweight Python package comprising a loader script, a nomenclator parser, and a FastMCP server. Running via Streamable HTTP on a homelab setup, the server exposes three distinct tools:
- search_companies: Queries by name or CUI tax ID, executing exact index hits or case-insensitive matching enriched with decoded CAEN activities and director data.
- extract_contacts: Crawls corporate websites prioritizing contact and about pages while aggressively filtering out placeholders and noise to harvest valid emails and social profiles.
- validate_domain: Wraps WHOIS, DNS, and SPF/DMARC checks so AI agents can verify domain health prior to lead aggregation.
"Honest limitations: corporate homepages often carry no public email... and obfuscated emails need decoders."
Project Author
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment