Handling Nested JSON Files Easily with jq Streams
Learn how to use jq tostream and fromstream functions to easily clean and filter deeply nested JSON files without writing complex Python scripts.
Stock photo for illustration only, not from the actual event
- Large and deeply nested JSON files cause issues when trying to delete specific fields at various depths.
- The tostream function flattens documents into [path, value] pairs for easier filtering.
- The fromstream function reconstructs the filtered stream back into a valid JSON file.
- This method saves time and avoids writing complex recursive dictionary-walking Python scripts.
Many developers face the challenge of dealing with massive JSON files, such as a 500KB OpenAPI spec, where every description field needs to be stripped regardless of its depth. While regular jq paths like .. | .description? work for reading data, deleting those fields is difficult because the path doesn't indicate where to place the result back.
The solution lies in using tostream and fromstream within jq, which flattens the entire document into a stream of [path, value] pairs. For example:
- jq 'tostream' spec.json outputs pairs like [["openapi"], "3.0.1"]
- [["info", "title"], "Petstore"]
- [["info", "description"], "This is a sample server..."]
Stock photo for illustration only, not from the actual event
Once the data is flattened into a stream, standard tools like grep or jq's select filter can be used to exclude unwanted keys before piping the data back to fromstream to reconstruct the document in a single line, such as jq 'tostream | select(.[0][-1] != "description") | fromstream' spec.json > spec-clean.json. The stream format uses [path_array, leaf_value], allowing .[0][-1] to easily target the final key in the path array.
jq's streaming filters have been available since version 1.6 released in 2019. Although not entirely new, they remain underused by developers tackling complex JSON structures. Leveraging streams not only speeds up data manipulation tasks but also eliminates the need to spin up general-purpose programming languages like Python for quick data sanitization in shell scripts.
This approach was successfully tested on a 2MB Kubernetes CRD manifest to remove all x-kubernetes-* vendor extensions, completing the task in just 3 seconds—far more efficient than writing a custom Python script utilizing recursive dictionary walking.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment