How to Grade Search Relevance Before You Touch Ranking Weights
Learn how to freeze your search corpus and score ranking relevance using graded judgments before modifying Postgres weights.
Stock photo for illustration only, not from the actual event
- Never tune ranking weights against a live, constantly changing database table.
- Freeze your corpus by creating a dated snapshot table as a consistent benchmark.
- Source 30 to 50 real queries directly from search logs and support tickets.
- Use nDCG@k metrics to evaluate and gate ranking ordering quality.
Before you change a single ts_rank weight, freeze your corpus and score the current ranking against a set of graded judgments. Otherwise, you are comparing two moving targets—new weights and new documents—and you will never know which one moved the needle. A relevance regression harness turns search complaints into a number you can diff, the same way a test suite turns broken code into a red check.
A previous discussion on load-testing a Postgres full-text index highlighted a critical point: if you tune weights against a live table, you are comparing two systems that are both changing at once. This post builds out the practical solution of how to actually collect judgments, freeze the corpus, and compute the scores.
Search quality lacks a single-row assertion. While a standard unit test asserts a concrete total like 42, relevance is about ordering. For example, for the query postgres connection pool, the pgbouncer guide should rank above an unrelated changelog entry. These orderings degrade quietly when a title weight fix inadvertently worsens three other queries without immediate notice.
Another common trap is the moving corpus where articles tables gain rows hourly. If you score today's ranking against today's table and last week's ranking against last week's table, any difference becomes contaminated by the documents themselves. You must hold the corpus still to isolate ranking changes.
The solution is to snapshot the exact columns your ranker reads into a dated table as your evaluation fixture:
-- One-time: capture the corpus as it was on this date.
CREATE TABLE eval_corpus_2026_08 AS
SELECT id, title, body, search_vector
FROM articles;
-- Run any candidate ranker against the frozen table, not the live one.
SELECT id
FROM eval_corpus_2026_08
WHERE search_vector @@ websearch_to_tsquery('english', :query)
ORDER BY ts_rank_cd(search_vector, websearch_to_tsquery('english', :query)) DESC
LIMIT 10;When testing new weightings, you only change the ORDER BY clause and never the FROM clause. Keep the snapshot table in repository fixtures or a durable bucket so the baseline remains reproducible across any machine.
Stock photo for illustration only, not from the actual event
You need a set of query, document, and grade rows where grades range from 0 for irrelevant up to 3 for perfect. Practical rules to manage this efficiently include:
- Source queries directly from your logs and support tickets instead of imagination, as thirty to fifty real queries beat a thousand invented ones.
- Judge a pool instead of the entire corpus by taking top results from every candidate ranker and grading only that union.
Pick the metric matching your user interface usage, such as nDCG@k for typical article or documentation search. It rewards proper ordering of relevant grades while discounting gains down the page, normalized cleanly into a 0 to 1 range.
"relevance is a property of an ordering over a fixed corpus, so you cannot measure it without freezing the corpus first."
Dev.to
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment