Semantic search finds items by meaning instead of by matching the words typed, so a search for “strike” can return a letter that says “the smelter men walked out.” Over a digitized collection it adds the most where the records use different words from the searcher, where recognized text is noisy, and where photographs have no captions. Its risks are matches nobody can explain, bias carried in the model, generated answers that misstate the records, and restricted material surfacing where it should not. The safer design keeps keyword search, combines the two (hybrid search), adds filters, shows why each item matched, never shows a generated answer without links to its sources, and keeps restricted material out of the index entirely.
This article explains the three kinds of search in collection terms, reports what published tests on historical text measured and under which conditions, and turns each risk into a requirement you can write into a project plan, a request for proposals or a contract. It is written for directors and digital collections staff deciding how the public will search a digitized collection.
A note on interest. Dastok is a Colorado Springs firm that digitizes historical documents, photographs and museum collections and builds collection search, including meaning-based search, so we have a stake in this question. The findings below come from published studies and institutional reports, cited where they are used, not from our own projects.
How keyword, semantic and hybrid search differ
Keyword search matches the words in your query against the words in the text and records. Semantic search compares meanings, represented as lists of numbers. Hybrid search runs both and merges the two ranked lists. Each has a different strength, and each fails in a different way.
Keyword search: the words you type
Keyword search, also called full-text search, builds an index of every word in the recognized text and the catalog records, then returns the items that contain the query words. Many search engines rank those items with a formula called BM25, which favors rarer words and words that appear often in a short passage. BM25 is, for example, the default ranking in Elasticsearch, according to its similarity documentation.
Keyword search is predictable and explains itself. A result contains your word, and the viewer can highlight it on the page. It is the right tool for surnames, place names, file numbers and exact phrases. Its weakness is vocabulary: it cannot find “strike” in a letter that says “walked out,” and it misses any word the recognition engine misread.
Two older techniques soften the spelling problem. Fuzzy matching accepts words within a small edit distance of the query; Elasticsearch’s fuzzy query counts one-character changes such as changing, removing or inserting a letter, or swapping two neighbors. Phonetic codes group names that sound alike. The National Archives’ census Soundex files SMITH and SMYTH under one code “so that you can find a surname even though it may have been recorded under various spellings.” Both help with names. Neither helps when the record uses a different word for the same thing.
Semantic search: passages close in meaning
Semantic search, also called meaning-based, embedding or vector search, uses a language model to turn each passage of text into an embedding. An embedding is a long list of numbers (hundreds or thousands, depending on the model) placed so that passages with similar meaning sit close together. The query gets an embedding too, and the search returns the passages whose numbers sit nearest to it.
One widely used method for producing such embeddings was described in the Sentence-BERT paper by Reimers and Gurevych (EMNLP 2019), which produced “semantically meaningful sentence embeddings that can be compared using cosine-similarity.” Cosine similarity is simply a measure of how closely two of these number lists point in the same direction.
Because the model reads passages rather than whole items, long items are first split into chunks: a letter page, a day’s entries in a ledger, a paragraph of a newspaper column. How you split them matters. A ledger page cut in the middle of an entry, or a letter separated from its date line, produces a chunk that means less than the page did.
Semantic search also never runs out of answers. Every passage sits at some distance from every query, so the nearest passages come back even when nothing in the collection is relevant, unless the system sets a cut-off. That is why a semantic search can return confident-looking results for a question the collection cannot answer.
Hybrid search: both lists, merged
Hybrid search runs a keyword search and a semantic search on the same query and merges the two ranked lists. A common merging method is reciprocal rank fusion (RRF), described by Cormack, Clarke and Büttcher in a 2009 SIGIR paper. Each item scores 1 ÷ (k + its rank) in each list, and the scores are added. The authors fixed k at 60, and Elasticsearch’s RRF documentation uses 60 as its default.
The effect is easy to picture. An item near the top of both lists rises to the top. An item found by only one list still appears, a little lower.
| Keyword (full-text) | Semantic (embedding) | Hybrid | |
|---|---|---|---|
| What it matches | The words typed, with optional fuzzy or phonetic variants | Passages close in meaning, whatever words they use | Both, merged into one list |
| Best for | Names, places, numbers, exact phrases | Topics, paraphrase, questions in everyday words | Public search over mixed material |
| Shows why it matched | The matched word, highlighted | Only if the interface shows the passage and labels it | Per result, if each list’s part is kept |
| Fails when | The record uses other words, or recognition misread the word | The query is a rare name or number, or nothing relevant exists | Settings lean too far toward one list |
| “No results” | Possible and meaningful | Rare unless a cut-off is set | Meaningful only for the keyword part |
What semantic search adds to a digitized collection
Semantic search adds reach. It finds items described in other words, tolerates some spelling variation and recognition noise, can search photographs by description, and accepts questions in everyday language. Each gain comes with a condition.
Items described in other words
Researchers rarely know the vocabulary of the people who wrote the records. The authors of a 2026 study of Swiss newspaper archives, discussed below, put the problem plainly: exact word matching fails “when spelling conventions shift across time, when users do not know the specific historical terminology used in the source material, or when relevant content is expressed through paraphrase rather than direct lexical overlap.”
Example (fictional). Box 3 of the Aster County Historical Society holds the Okerlund family papers and the Wren Gulch Mercantile ledger, 1893 to 1896. Nobody in the box wrote the word “strike.” A keyword search for it finds nothing. A semantic search finds Ida Okerlund’s letter of February 3, 1894 (“The smelter men walked out on Monday”) and the Wren Gulch Courier notice of March 1, 1894 (“The trouble at the smelter ended Thursday”). A search for “school supplies” finds ledger page 112, which reads “chalk 1 box, slates 6, lamp chimney 1.” The search on our example collection page imitates this with a small hand-made table of related words; a real index uses a language model’s embeddings.
The condition: the model has to know that “walked out” and “strike” are related. Models learn such links from large amounts of mostly modern text, so they handle common paraphrase well and local or period terms less predictably. A mining camp’s slang, a county’s own office names or a church’s internal terms may sit nowhere near the words a searcher uses.
Spelling variants and recognition errors
Recognized text from old print and handwriting contains errors, and a misread word is invisible to keyword search. In the same fictional box, the uncorrected text of Ida’s letters read “Wren Gulch” as “Wren Gutch” eleven times. A keyword search for the town misses all eleven. Fuzzy matching with one edit catches them. A semantic search may still rank those letters near a query about floods in Wren Gulch, because the rest of each passage is about the same thing.
Embeddings are not immune to noise, though. Zhuang and Zuccon found that typos in queries “can lead to a significant drop in retrieval and ranking effectiveness” for a dense retriever (a semantic search model) and a re-ranker tested on a large web passage collection, in a 2021 EMNLP paper. Better text helps every kind of search. Our article on historical newspaper OCR quality covers what improves recognition on difficult print, and text recognition for print and handwriting describes how page text is produced and labeled.
Photographs without captions
A second family of models places images and text in the same space, so a text query can retrieve pictures. The best known is CLIP, described in a 2021 paper by Radford and colleagues at OpenAI, trained to predict which caption goes with which image “on a dataset of 400 million (image, text) pairs collected from the internet.” With such a model, a search for “horse and wagon in snow” can return an uncataloged print that nobody has described yet.
The condition: the model describes what is visible, in the vocabulary of modern internet captions. It does not know who is in the picture, which studio made it, where it was taken or when. A search can find “two girls in matching dresses,” but only a catalog record can say they are Ida and Marit Okerlund. The model also carries social bias, covered under the second risk below.
Questions in everyday language
Some systems use a large language model to turn a question into a structured search. The Victoria and Albert Museum tested this against its collections API and described the results in AI Assisted Collections Exploration (Richard Palmer, November 2023). In one test the model picked out “early Victorian” as a style, looked it up in the Getty vocabulary, and searched by the style’s identifier; the post notes that a vocabulary identifier “avoids any confusion with jet being used with a different meaning, e.g. jet planes.” The search found William Mulready’s painting “An Interior.”
The condition: the same post reports a test that went badly wrong, described under the third risk. Palmer’s suggested safeguard is to show users, as they type, what the assisted search would be, so they can see whether it understood the question.
Across languages
Multilingual embedding models place a sentence and its translation close together. Reimers and Gurevych described a training method in which “a translated sentence should be mapped to the same location in the vector space as the original sentence,” and demonstrated it for more than 50 languages (EMNLP 2020). For a collection with letters in Norwegian, German or Spanish, an English query can reach them. How well this works on 19th century spelling and dialect has to be tested on your own pages.
What published tests on historical text have measured
Published tests on historical collections show hybrid search beating either method alone, and show that semantic search on its own does not reliably beat keyword search. They were run on large newspaper and parliamentary corpora, not on a small archive’s boxes, so treat them as direction, not as a forecast for your collection.
Two measures appear below. Recall at 10 is the share of the known relevant items that appear in the first ten results. NDCG at 10 is a score that rewards putting the most relevant items nearest the top; 100 percent means the ideal order.
A general benchmark, 2021
The BEIR benchmark tested ten retrieval systems on 18 public datasets drawn from different retrieval tasks and domains (Thakur et al., NeurIPS 2021). The authors found BM25, the classic keyword ranking, to be a strong baseline, and found that dense and sparse retrieval models “often underperform other approaches” on data unlike their training data. Embedding models have changed a great deal since 2021, but the lesson stands: a semantic model is not automatically better than keyword search on unfamiliar material.
Swiss newspapers, 1762 to 2001
Konstantinidis and colleagues at the University of Lausanne tested search over 500,000 randomly sampled segments from nine Swiss periodicals spanning 1762 to 2001, provided by the Cantonal and University Library of Lausanne (arXiv 2607.03440, a preprint, July 2026). They wrote 384 natural-language test queries by hand and compared keyword ranking, semantic ranking, both combined, and both combined with a re-ranker. A re-ranker is a second model that reads the query and each of the top candidates together and puts them in a better order.
Three points matter more than the top score. First, semantic ranking alone scored no better than keyword ranking (63.7 against 66.0 percent, within the stated margins), which the authors describe as BM25 providing “a competitive lexical baseline.” Second, combining the two lifted the score by about eight points. Third, the re-ranker added the most, and cleaning the OCR text with a language model added a little more.
The limits are the authors’ own. They state that they had no ground-truth question and answer pairs for the new dataset, so the scores come from an automated evaluation framework (RAGAS) with a language model as judge, not from people marking relevant items. The paper is a preprint, and the dataset cannot yet be redistributed.
French debates and newspapers from 1887
Bian, Puren and Cafiero tested retrieval over French parliamentary debates and newspapers from 1887, from digitized collections of the Bibliothèque nationale de France, with 1,752 questions whose factual coherence a historian validated (TRACE, CIKM 2026). Here semantic retrieval beat keyword ranking overall (recall at 10 of 0.734 against 0.643), the reverse of the Swiss result.
The finding that matters most for planning is how little the two methods overlapped. When the authors ran their system’s search queries through each method separately, the keyword and semantic top 20 lists had a mean Jaccard overlap of only 0.19, a measure on which 1 means identical lists and 0 means no items in common. Of the relevant documents found by combining them, 28.8 percent were reachable through only one of the two. Combining them retrieved 78.9 percent of relevant documents, against 70.1 percent for the better single method. In the authors’ words, keyword search “matches tokens that survive digitisation, while dense retrieval captures semantic paraphrases of corrupted text.”
Text quality and what users judge useful
The National Library of Finland ran a user study on one Finnish newspaper, Uusi Suometar (1869 to 1918), showing 32 users the same articles with either older or improved OCR, without telling them which (Kettunen et al., IRCDL 2022). The improved text scored 7.94 percent higher on average in users’ judgments of usefulness. Search quality is not only about the ranking method; what the result shows matters too.
What the tests do not tell you
None of these studies tested a small historical society’s mixed boxes of letters, ledgers and photographs. Two are preprints or new conference papers. One scored results with a language model as judge. And the two archival tests point in opposite directions on semantic search alone. The consistent message is narrower: combining keyword and semantic search found more than either alone in both archival tests, and the only way to know what works on your collection is to test on your collection.
Risk one: matches nobody can explain
A keyword result explains itself because the matched word is on the page. A semantic result often does not: the item came back because its numbers sat close to the query’s, and nothing on screen says why. Users then cannot tell a good match from a near neighbor that happens to be irrelevant.
This matters most to the researchers archives serve best. A genealogist checking whether a name appears anywhere in a collection needs to know that a search was exhaustive. Semantic search cannot promise that. It returns the nearest passages, not every passage containing the name, and it rarely says “no results,” so an empty answer from the keyword part is the only one that means something.
The library field has asked for explanation for some time. OCLC Research’s Responsible Operations research agenda (Padilla, 2019) argued that transparency “must go hand in hand with practices that encourage ‘explainability’” and suggested public interfaces that let users “adjust the parameters of an algorithm being applied to a collection, foregrounding rather than occluding the subjective nature of collection representation.” For a search page, that translates into plain features:
- Show the passage that matched, not only the item title.
- Label each result by how it was found: matched words (highlighted) or matched by meaning.
- Offer an exact-words mode that switches semantic matching off.
- Keep exact matches for names and numbers above results matched by meaning.
Risk two: bias in the model and gaps in the collection
Embedding models learn from large collections of mostly modern internet text and images, and they carry that material’s associations into your search results. The gaps in what has been digitized shape results too.
What the model brings
The clearest published measurement comes from CLIP’s own authors. In the bias section of the CLIP paper, they asked the model to classify 10,000 face images with a label set that included crime-related and non-human categories. They found that 4.9 percent of images were misclassified into one of the non-human classes; images in the dataset’s “Black” category had the highest rate, approximately 14 percent, while all other groups were under 8 percent. They also found that 16.5 percent of male images were put into crime-related classes, against 9.8 percent of female images.
The authors add that results “can change based on the class categories one chooses to include as well as the specific language one uses.” In a collection search, the users choose the words. Anyone can type a loaded description into a photo search, and a model with these associations will return its nearest pictures of real people. For portrait collections, decide whether image search by description should cover photographs of people at all, and test it with the queries you would least want to see answered.
Text models raise a related problem. Historical records and older catalog descriptions contain terms that are now offensive. A model can link a respectful modern query to those terms, which may help a researcher, but the results then appear without the context a reading room would give. The V&A’s post notes that text taken from the internet “may contain ‘offensive content and language’” which a model can replay.
What the collection leaves out
A search can only find what has been digitized and indexed. Amelie Roper of Cambridge University Library, interviewed by the university’s ai@cam initiative in October 2025, put it this way: “Most institutions have digitised only a fraction of their holdings, so results may be biased or unrepresentative.” The Santa Barbara Statement on Collections as Data, since succeeded by the Vancouver Statement, makes the same point about scale: “The scale of some collections may also obfuscate what is hidden or missing.” A search page that reads as complete can hide this. Say on the page what has been digitized and what has not.
Risk three: generated answers that misstate the records
Some search systems go further than ranking. They pass the top results to a large language model and show a written answer, a pattern called retrieval-augmented generation (RAG). Retrieval reduces invented content compared with a chatbot answering from memory, but it does not remove it.
The V&A post shows the failure in a collection setting. Asked to list John Constable objects from the 18th century and tabulate them, the model built a workable query and received the museum’s records, then, in the author’s words, “seems to ignore the data entirely” and listed six Constable paintings held by other institutions, “with mostly invented data,” none of them the V&A’s. The output was tidy and wrong.
Outside heritage, the most careful measurement so far comes from legal research. Magesh and colleagues ran a preregistered evaluation of commercial legal research tools built on retrieval-augmented generation and found that they “each hallucinate between 17% and 33% of the time” in the May 2024 version of their study, later published in the Journal of Empirical Legal Studies (2025). These were products from large publishers with curated case law. A local history collection with noisy text and thin records gives a model less to stand on, not more.
A generated answer also summarizes, and summary loses what archivists care about: who wrote the record, when, in what voice, and with what gaps. Our fictional example shows the risk in small.
Example (fictional). A visitor asks a generated-answer search over Box 3: “Who ran the Wren Gulch store during the strike?” The index returns Ida’s letter of February 3, 1894 (“Walter says the store will carry them on credit”) and ledger page 77. A model could fairly answer that Walter Okerlund was connected to the store. It could just as easily write that “Walter Okerlund owned the Wren Gulch Mercantile,” which no record in the box says. With links to the two items under the answer, a reader can check in seconds. Without them, the sentence becomes a fact on the internet.
If you offer generated answers at all, require that every sentence carries a link to the item it came from, that quotations are quoted exactly, that the system says so when the records do not answer the question, and that the answer is labeled as machine-written.
Risk four: sensitive and restricted material
Semantic search is good at finding things by topic, and that includes topics an archive handles with care: illness, adoption, crime, welfare, sacred or ceremonial knowledge. A keyword search for a sensitive subject finds only records that use the word; a semantic search finds records that describe it in any words. If restricted items are in the index, they are findable.
Hiding restricted items in the results is not enough. Cambridge’s Roper lists material that is “sensitive for many reasons, including copyright, data protection, licensing restrictions, subject matter, materiality, and provenance,” and notes that “controlling how uploaded data is subsequently used by AI engines - and ensuring it is not retained for training - remains a much more complex task.” Generated answers, snippets and “more like this” links can all surface text from an item whose page is blocked.
Security guidance for these systems says the same. OWASP’s 2025 list of risks for language model applications includes vector and embedding weaknesses: inadequate access controls “can lead to unauthorized access to embeddings containing sensitive information,” and the recommended mitigation is “permission-aware vector and embedding stores.” Embeddings are not anonymized copies, either. Morris and colleagues showed that a multi-step method could recover 92 percent of 32-token text inputs exactly from their embeddings and could recover full names from clinical notes (EMNLP 2023).
Four practical rules follow:
- Leave restricted material out of the index. Items closed for privacy, donor terms, legal holds or community protocols should not be embedded at all, rather than embedded and filtered.
- Re-index when status changes. When an item is closed, its text and embeddings come out of the index the same day, along with any cached answers.
- Read the terms of any hosted model. If passages are sent to an outside service for embedding or answering, find out what it keeps and whether it trains on them. OCLC’s agenda recommended assessing whether such terms “breach expectations of privacy” or allow “corporate data reuse beyond the context of original use.”
- Follow community decisions. Material held under community protocols, including tribal materials, needs the community’s agreement before any machine processing. Our page on community archives describes working under those agreements.
Requirements to set before you launch
The risks above become requirements you can write into a project plan, a request for proposals or a hosting contract. The table pairs each with a way to check it before launch.
| Requirement | Why | How to check it |
|---|---|---|
| Hybrid search, with keyword search always available | Combined search found more than either alone in both archival tests | Run your test queries in each mode and compare |
| An exact-words mode | Exhaustive name and number searches need it | Search a surname you know appears five times; expect all five |
| Filters on date, type, place, rights status and review status | Narrows meaning-based results to what the user asked for | Filter to one year; check no item outside it appears |
| A visible reason for each match | Users must judge relevance themselves | Every result shows its passage and how it matched |
| No generated answer without links to source items | Generated text misstates records | Ask a question the records cannot answer; expect a plain “not found” |
| Restricted items excluded from the index | Filtering after retrieval still leaks | Search for a closed item’s distinctive phrase; expect nothing anywhere |
| Machine-made text and descriptions labeled in results | Users should know what a person checked | Uncorrected text and unreviewed fields show a label |
| Model name, version and settings recorded | Results change when the model changes | A written record exists, and test queries are rerun after any change |
| Data terms for any hosted model | Material may be kept or reused | Written terms on retention and training |
| An exit plan | The index is rebuilt elsewhere one day | You receive the text, records and a description of the index settings |
Labels for machine-made text in the index
Semantic search indexes whatever text you give it: corrected transcriptions, uncorrected recognition, catalog records, and sometimes machine-written descriptions of photographs. The searcher sees them mixed together. Each kind should carry its status into the results, so “matched in uncorrected text” or “description suggested by a model, not reviewed” appears where a person decides whether to trust it. The companion article on labeling AI-generated metadata covers field-level labels, and our AI policy sets out the labels we use.
Where IIIF fits
IIIF, the International Image Interoperability Framework, solves a different problem. Its Content Search API 2.0 searches the text annotations inside a single item, such as the OCR of one book or newspaper issue, so a viewer can highlight hits and jump between them. The specification says that “searching for metadata or other descriptive properties is not in scope.” Collection-wide semantic search therefore sits beside IIIF, not inside it: the search finds the item and the passage, and the IIIF viewer opens the page with the words marked, using the word positions stored in the recognized text. Whether a small institution needs IIIF at all is the subject of Does a small museum or archive need IIIF?.
How to test semantic search on your own collection
You can judge semantic search for your collection with a few dozen real questions and an afternoon of checking. The aim is to see what it finds that keyword search misses, and what it gets wrong, before the public does.
- Collect real questions. Pull 30 to 50 from reference emails, reading room requests and staff knowledge. Mix names, places, dates, topics and questions phrased in everyday words.
- Mark the right answers. For each question, list the items a careful staff member would want found. This is your reference set, and it takes most of the effort.
- Run each question three ways. Keyword, semantic and hybrid, on the same text and records.
- Score the first page. For each mode, count how many of the marked items appear in the first ten results (recall at 10), and note any result that is plainly irrelevant.
- Read the failures. Look for missed names, wrong dates, near neighbors ranked above exact matches, and photographs returned for loaded descriptions.
- Probe the risks. Search for phrases from closed items, ask questions the records cannot answer, and try offensive and modern terms for historical subjects.
- Record the setup. Write down the model, its version, the date, the chunk size and the merge settings, and rerun the same questions whenever any of them changes.
Example (fictional). The Aster County Historical Society tests search over Box 3 with 40 questions drawn from a year of reference emails. Suppose keyword search finds 60 percent of the marked items in its first ten results, semantic search 58 percent, and hybrid 75 percent. The numbers matter less than the misses. Keyword search misses the strike letters and anything misread as “Wren Gutch.” Semantic search, asked for the teacher Thea Lund, returns letters about the school but ranks the cabinet card that names her fourth. Hybrid puts the card first and keeps the strike letters. The society keeps hybrid search, adds an exact-words switch for genealogists, and corrects the town name in the recognized text before launch.
The Library of Congress offers a structure for this kind of trial. Its LC Labs AI Planning Framework moves through three phases, “Understand,” “Experiment” and “Implement,” and looks at the data, the models and the people involved, with a use case risk worksheet you can adapt. A search test like the one above is the experiment phase in miniature.
What to do next
Semantic search is worth adding when your records use words your users do not, when the text is imperfect, or when photographs lack captions. It is not a replacement for keyword search, and it is not a reason to skip cataloging. A practical sequence:
- Decide who will search the collection and for what: names, topics, images or questions.
- Improve the text first, since every kind of search depends on it.
- Separate restricted material before anything is indexed.
- Write the requirements in the table above into your plan or request for proposals.
- Test with 30 to 50 real questions in keyword, semantic and hybrid modes, and read the misses.
- Label machine-made text and descriptions in the results, and record the model and settings.
- Decide whether to offer generated answers at all; if you do, require links under every one.
To see labeled records and meaning-based search on a fictional box, open the example collection. Our collection search service describes how we build hybrid search and IIIF viewing, or deliver files for your own system. If you would rather test search on your own material first, a pilot box runs a small batch of your real items through every stage, including search, before anything larger is decided.