Greenfinger 2.0: A Distributed JVM Web Crawler
Explore Greenfinger 2.0, a distributed JVM web crawler requiring no initial database or search server, offering instant searchable text and image archives.
Stock photo for illustration only, not from the actual event
- Greenfinger 2.0 is a distributed web crawler built for the JVM.
- Requires no database, search server, or API key for the initial crawl.
- Supports text search, semantic queries, and local AI image description matching.
- A leaderless, decentralized architecture ensures resilient crawling if nodes drop out.
In software development, harvesting and searching through massive website data traditionally demands complex infrastructure setups. Recently, the Greenfinger 2.0 project introduced a streamlined approach to building a distributed web crawler on the JVM (Java Virtual Machine). It is engineered to operate out-of-the-box without requiring pre-installed databases, search engines, or API keys, capturing entire sites along with every image and version into a searchable archive.
Architecturally, Greenfinger 2.0 organizes processed data into three distinct layers:
- File: Stores fetched web pages, article text, and all images on local disk, MinIO, or any S3-compatible store.
- Index: Manages full-text search capability using embedded Lucene or Elasticsearch.
- Vector: Handles text chunks and image embeddings, supporting embedded Lucene, Qdrant, Weaviate, and Elasticsearch.
Stock photo for illustration only, not from the actual event
A key design choice is that the file layer remains permanently active while the database stores metadata exclusively, allowing the other two layers to rebuild effortlessly from the raw files if an outage occurs or analyzers change. Furthermore, swapping object storage backends to MinIO, AWS S3, or Google Cloud Storage requires modifying just three lines of configuration.
Regarding execution, Greenfinger 2.0 completely eliminates central coordinators or scheduler bottlenecks. Nodes communicate peer-to-peer; when Node A discovers a URL, it hands off ownership dynamically to the responsible node. Completion is determined collectively by checking shared counters against fetch limits rather than relying on a single leader, preventing stalled crawls if a node unexpectedly terminates midway.
"Query: what happens when a star runs out of fuel . Top answer: the supernova remnants page, which never contains that sentence."
Greenfinger Documentation
The ability to retrieve relevant pages based on semantic intent—even when the exact query terms do not appear literally on the target page—highlights the immense power of integrating local vector embeddings into modern archiving tools, bridging the gap left by rigid keyword-matching search engines.
Document parsing covers text, markdown, and CSV natively, with PDF, Word, and Excel supported via configurable modular dependencies. Embedding models run locally with zero account registrations required, utilizing multilingual-e5-small for text and SigLIP 2 for images via ONNX. The web dashboard is built with Angular 21, Signals, and Material 3, communicating directly through the same REST endpoints available to custom applications.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment