How to get full article text from RSS feeds tested on 32
Analysis of 32 public feeds reveals common RSS teaser issues and methods to extract full article text using standard Python code.
Stock photo for illustration only, not from the actual event
- Tested across 32 public feeds with 13 requiring full web page scraping.
- The content:encoded rule proves inaccurate in both directions frequently.
- Most feeds support ETag or Last-Modified headers to save bandwidth.
- GUID management and date parsing work smoothly without complications.
This article is based on an experiment conducted by Claude Code, which issued requests to 32 public feeds two seconds apart on October 12, 2026, and repeated it on November 11, 2026. The core challenge when feeding blog posts into a RAG index, LLM summary, or newsletter is determining whether a feed provides the complete post or just a teaser.
Standard advice usually suggests checking for content:encoded, but across these 32 feeds, that rule was incorrect in both directions. Four feeds placed the entire post inside the description field without any content field at all, while three feeds featured a content field containing only a teaser for nearly every item. The formats consisted of 22 RSS 2.0 feeds, 8 Atom feeds, 2 JSON Feeds, and no RSS 1.0 feeds.
Stock photo for illustration only, not from the actual event
Teasers within these feeds announce themselves at the end, though each site words them differently. Inspecting the last 80 characters successfully caught every teaser except for Stratechery, where length remains the only indicator. Two traps affected the initial version, leading to a rule that strips the WordPress footer and trailing link markers before evaluating text length.
Extracting data reliably from diverse RSS feeds requires robust conditional request handling and caching mechanisms. Dealing with irregular markup structures across various publishing platforms is essential for maintaining efficient ingestion pipelines without overloading network resources.
For the 13 feeds requiring web page retrieval, fetching the item URL and isolating the article text is necessary. Cache validation plays a major role since 26 out of 32 feeds sent ETag or Last-Modified headers, and returning them yielded 304 responses with empty bodies from 24 of those feeds, significantly reducing data transfer overhead.
Regarding identifiers, 15 feeds did not use the item URL as the GUID for their entries, while 16 feeds utilized the URL for every item. Meanwhile, dates caused no parsing issues whatsoever, with all 1,991 timestamps successfully processed using standard library datetime parsers.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment