Plan a pilot box
Menu

Can AI read old handwriting? What HTR and LLMs get right

Reader question: How accurately can handwritten text recognition and large language models transcribe 18th and 19th century handwriting, and how much human review is still needed?

By Dastok Editorial TeamIssue: Published

Yes, with conditions. In the most directly relevant published test, on 18th and 19th century English handwriting, the best general-purpose language model read about 93 to 94 percent of characters correctly with no training on the hands, and a language model correcting a dedicated handwriting engine’s draft cut the character error rate to between 1.8 and 3 percent. That is often good enough to search a collection. It is not good enough to quote, publish or extract figures from without a person checking, and every one of those numbers comes from a 50-page test set in a preprint, so your own pages may do better or much worse.

This article turns the research into expectations a collections manager can plan with. It explains what error rates mean, reports what five recent studies found and under which conditions, lists the mistakes language models make that a single accuracy number hides, and sizes human review to the way the text will be used. Matching recognition methods to each kind of record (print, typescript, forms, ledgers, letters) is covered in OCR or HTR? Matching text recognition to each kind of record. Organizing volunteers around machine drafts is covered in Volunteers, AI or both? Planning a transcription project.

A note on interest. Dastok is a Colorado Springs firm that digitizes historical documents, photographs and museum collections and runs text recognition for print and handwriting, so we have a stake in this question. The figures below come from published studies, cited where they are used, not from our own projects.

What accuracy means: character and word error rates

Accuracy in handwriting recognition is measured against a careful human transcription of the same page, called the reference or ground truth. The character error rate (CER) counts every character the machine substituted, left out or added, and divides that count by the number of characters in the reference. The word error rate (WER) does the same for whole words.

Written out, CER = (substitutions + deletions + insertions) ÷ characters in the reference. The studies below compute it this way, as an edit distance between the machine text and the reference. Transkribus, a widely used handwriting recognition platform, puts the idea plainly in its help page on character error rate: “a 5% CER means that the text model has automatically transcribed correctly 95 characters out of 100.”

Five percent sounds small until you scale it to a page. Suppose, as a hypothetical, a letter page holds 1,200 characters. A 5 percent CER means about 60 errors on that page, and any one of them can sit inside a surname that someone will later search for.

Why word error rate usually runs higher

One wrong letter spoils a whole word, so WER usually runs well above CER on the same text. Humphries and colleagues note in their 2024 study of language models on historical handwriting that WER is normally “roughly 3 to 4 times” the CER, citing earlier work. For search, WER is often the more honest figure, because people search for words, not characters.

Strict and normalized scoring

The same transcription can score very differently depending on what counts as an error. A strict score counts every difference in capitalization, punctuation and spelling. A normalized score ignores some of them, for example treating “employ’d” in the original and “employed” in the transcription as a match.

Humphries and colleagues reported both. On their test set, about a third of the word errors made by Transkribus’s Titan model, and 38 percent of those made by its older PyLaia model, were differences of capitalization, punctuation or historical versus modern spelling. Romein and colleagues show the same effect in a 2025 comparison of HTR engines: one engine’s CER on 17th century printed text was 1.50 percent under the base measure and 0.82 percent once punctuation, case and spaces were set aside. When anyone quotes you an error rate, the first question is how it was measured.

What the vendor rules of thumb say

Transkribus offers its own guidance, which is useful as long as you read it as one vendor’s rule of thumb rather than a standard. Its help page says results with a CER of 10 percent or below “can be considered very efficient for automated transcription,” that a CER of 20 to 30 percent is “sufficient to work with powerful searching tools like Smart Search,” and that a good CER for handwriting “ranges between 2 and 8%, depending if the model has been trained on one or multiple hands.” The same page says these figures are measured on validation pages not used in training, and that models “may perform worse” on hands not seen in training.

Example (fictional). A passage from Ida Okerlund’s letter of April 12, 1893, in Box 3 of the Aster County Historical Society, has 137 characters and 25 words: “The creek came up in the night and took the footbridge below the schoolhouse. Father says the county will not rebuild it before the snow.”

An HTR engine returns “tooh” for “took,” splits “footbridge” and “schoolhouse” into two words each, capitalizes “County” and drops the final period. That is 5 character errors, a CER of 3.65 percent, and 7 word errors, a WER of 28 percent. The text looks rough, and a reader sees every problem.

A language model returns the line perfectly except for the last word, which it reads as “spring.” That is also 5 character errors and a CER of 3.65 percent, but only 1 word error, a WER of 4 percent. The text looks clean, and the one error changes what the letter says about the bridge and the season. Same CER, very different risk.

What published tests found, and under which conditions

Recent studies show language models matching or beating dedicated HTR engines on some historical handwriting when neither has been trained on the hands, but not consistently, and the best results come from combining the two. Every figure belongs to a particular test set, language and scoring method, so read each one with its conditions attached.

Study Material tested Systems compared Headline result Main limits
Humphries et al., 2024 (preprint) 50 pages of English letters, diaries, memoranda and legal papers, 1761 to 1827, 33 hands GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro; Transkribus PyLaia and Titan Best model 7.3% strict CER, 5.7% normalized; Titan draft corrected by Claude 3.0% strict, 1.8% normalized Small, English only, mostly fur trade records; authors built the tool they tested
Crosilla, Klic and Colavizza, 2025 (preprint) Modern and historical handwriting in English, French, German and Italian Proprietary and open-source language models; Transkribus models No consistent advantage for either approach; better on English Preprint; results differ by dataset and language
Kim et al., 2025 (preprint) 20 pages of 1921 French-language Belgian estate declarations in table form GPT-4o, Claude 3.5 Sonnet; four OCR and HTR tools without training Language models read names the untrained tools could not; digits were hard One record type, 20 pages
Romein et al., 2025 (peer reviewed) 17th century print, Dutch resolutions 1576 to 1795, Glagolitic script, German shorthand Five dedicated HTR engines, trained or fine-tuned Best base CER from 1.50% (print) to 9.29% (shorthand) Some authors work for vendors whose engines were tested
Semnani et al., 2025 (CHURRO, EMNLP conference paper) 99,491 pages from 155 corpora, 46 language clusters Commercial and open-weight vision-language models, OCR systems Best commercial model 63.6% similarity on handwriting, averaged over languages Conference paper; different metric; averages hide large differences by language

The study closest to an American or Canadian archive’s holdings is the one by Humphries and colleagues at Wilfrid Laurier University, posted to arXiv in November 2024. They built a test set of 50 pages (9,911 words) of letters, memoranda, diary entries and legal documents written in English between 1761 and 1827, in 33 hands. Thirteen pages were scans from black-and-white microfilm, 26 were scans or photographs of medium to high quality, and 11 were taken with a phone or similar handheld device.

They ran three commercial multimodal models (GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro) through their own open-source tool with a simple instruction and no examples, and compared them with two Transkribus models. Each system transcribed the full set ten times, and the averages are reported.

Character error rate, lower is better Strict (every difference counts) Modified (ignores case, punctuation, modern spelling) 0% 4% 8% 12% Transkribus PyLaia 10.3% 7.6% Transkribus Titan 8.0% 6.6% GPT-4o 11.0% 9.0% Gemini 1.5 Pro 8.5% 6.7% Claude 3.5 Sonnet 7.3% 5.7% Titan, then Claude 3.0% 1.8%
Average character error rates over ten runs on one 50-page test set of English handwriting, 1761 to 1827. "Titan, then Claude" is a Transkribus Titan transcription corrected by Claude 3.5 Sonnet with the page image. Source: Humphries et al. (2024), arXiv 2411.03340, Tables 1, 3 and 10, a preprint; checked November 2026.

Three things in that chart matter more than the ranking. First, one of the three language models (GPT-4o, at 11.0 percent strict CER) did worse than both Transkribus models. Second, the best language model beat Transkribus Titan by less than one percentage point on strict CER (7.3 against 8.0), although the gap in strict WER was wider (15.9 against 19.7 percent). Third, the largest gain came from combining the two kinds of system, which a later section covers.

The authors state the limits themselves: the dataset “is relatively small and is focused solely on English language documents from the 18th and 19th centuries,” and it came mainly from records of the North American fur trade. The reference transcriptions were made by correcting a Transkribus draft, which may favor that engine’s conventions. The authors also built and released the software they tested. The paper is a preprint, not a peer-reviewed article.

Four languages, and no consistent winner

Crosilla, Klic and Colavizza benchmarked proprietary and open-source language models against Transkribus models on modern and historical handwriting in English, French, German and Italian (preprint, last revised June 2025). Their abstract reports that proprietary models, “especially Claude 3.5 Sonnet,” did best among the language models; that the models “exhibit a preference for the English language due to their pre-training dataset composition”; and that “comparisons with Transkribus show no consistent advantage for either approach.” That last finding is the useful corrective to headlines about language models replacing HTR.

Handwritten tables from 1921

Kim and colleagues tested GPT-4o and Claude 3.5 Sonnet on 20 scanned pages of 1921 estate declarations from Nivelles, Belgium: French-language tables with typed column headings and handwritten entries (preprint, January 2025). They compared the language models with four conventional tools used without training. The untrained tools “could not return any meaningful outputs” for the handwriting, while the language models produced the records “with correctly read names.”

The same paper is candid about problems. Giving the model two example pages with their transcriptions gave the best results, and cutting each page into rows worked better than sending the whole page. The authors also found that “LLMs have a hard time reading digits,” and that the models changed their behavior several times over the months of the experiment.

Dedicated engines across scripts

Romein and colleagues compared five dedicated HTR engines, trained or fine-tuned on each dataset, in a peer-reviewed article published in May 2025. The best base CER ranged from 1.50 percent on 17th century Dutch and French print to 1.78 percent on Dutch States General resolutions written between 1576 and 1795, 4.54 percent on Glagolitic script and 9.29 percent on German shorthand. Transkribus’s Titan model showed strong results without extra training on Latin-script material. Some of the authors are affiliated with READ-COOP, which runs Transkribus, and with Planet AI, whose engine was tested, so read it as expert but interested work.

Many languages, one average

The CHURRO paper by Semnani and colleagues, presented at EMNLP 2025, assembled 99,491 pages from 155 historical corpora in 46 language clusters. The best commercial model it tested, Gemini 2.5 Pro, reached 63.6 percent normalized Levenshtein similarity on handwritten pages averaged across languages, and 80.9 percent on printed pages. That metric is close to one minus CER but not identical, so do not compare it directly with the figures above. What it shows is that an average over many languages and scripts is far below what the best English tests report.

Why results swing so much between collections

The same system can be nearly right on one box and close to useless on the next, because recognition depends on how close your pages are to what the system learned from. The hand, the language, the layout and the image all move the error rate, often more than the choice of engine does.

Factor What the research shows What to test on your own pages
The hand Transkribus warns that models may perform worse on hands not seen in training. Humphries et al. cite earlier work reporting CERs of 8 to 25 percent for general Transkribus models on unfamiliar hands. Include the hardest hand in the box, not only the neat one
Language and script Crosilla et al. found a preference for English. Romein et al. found much higher error rates on Glagolitic and shorthand than on Latin-script print. Test each language and script separately
Spelling and abbreviation About a third of word errors in the Humphries tests were case, punctuation or modernized spelling. Decide in advance whether you want spelling as written or modernized
Layout Kim et al. found whole-page outputs did not always return the right number of table rows. CHURRO reported reading-order errors, often from columns. Count rows and lines in the output against the image
Image source The Humphries set mixed microfilm, scans and phone photos. CHURRO notes character errors near the spine of a bound letter. Test pages captured the way the whole batch will be
Amount of text CHURRO found that accuracy generally fell as the amount of text on a page rose. Include your densest pages
Digits and proper names Kim et al. found digits hard. CHURRO found look-alike character errors frequent in proper names. Score numbers and names separately
Whether the page is already online Humphries et al. found signs that published collections were in model training data. Test on pages that have never been posted

The last row deserves a sentence of its own. Humphries and colleagues found that one model refused to transcribe nearly every document they tried from well-known online collections, with a stop reason indicating the material had appeared in its training data. Test sets built from such material produced error rates “several standard deviations below” their clean set. A test on pages that are already online can flatter a model, because it may be recalling text rather than reading it.

How HTR engines and language models work differently

A dedicated HTR engine finds the lines on a page and predicts the characters in each line from patterns learned on transcribed examples. A multimodal language model looks at the whole image and writes out the most probable text, drawing on everything it has learned about language. That difference explains both why language models cope well with unfamiliar hands and why their mistakes are harder to spot.

Dedicated HTR engines learn from your transcriptions

An HTR engine’s results improve most when you train or fine-tune it on transcribed pages of the same hand. Transkribus’s data preparation guidance says that, depending on the material and the number of hands, “between 5,000 and 15,000 words (around 25-75 pages) of transcribed material are required to start,” and advises training on “at least 10,000 words for each hand” for handwritten documents. Models trained on more than 100,000 words from many hands of the same period and region “should be capable of recognising hands not seen in any way during the training,” though the guidance expects results somewhat worse than the model’s measured CER.

That training cost is the reason fine-tuned engines suit large runs in one hand: a clerk’s 40 years of minute books, a diarist’s lifetime. For a box of letters from twenty correspondents, transcribing 10,000 words per hand is rarely realistic, and a general model or a language model becomes the practical choice.

HTR errors tend to look like misreadings. Humphries and colleagues give the example of an engine producing “furr5” for the historical spelling “furrs.” A reader notices that kind of error.

Language models need no training, and write fluent text

A multimodal language model can transcribe a hand it has never seen without any training on your pages. The studies above used it zero-shot, or with one or two example pages in the instruction. Its errors are real words in plausible order, because the model writes the most likely text, and that is exactly what makes them easy to miss.

Dedicated HTR engine Multimodal language model
Training on your pages Helps most; thousands of words per hand for best results None needed; one or two examples can help
Unfamiliar hands Often weaker without training Often stronger
Typical error Misread characters that look wrong Fluent words that look right
Output Text tied to the lines it found on the page Usually plain text for the page
Repeatability Stable for a fixed model version Hosted models can change behavior over time
Where the images go Depends on whether the engine runs locally or as a hosted service Usually a hosted service; read the provider’s terms

The failure modes specific to language models

Language models make mistakes that read well. An error rate counts characters; it does not tell you that a sentence was invented, a row was dropped or a spelling was quietly modernized. These are the patterns the research documents, and each needs a specific check.

Plausible invented text

When a model cannot read the page, it can write something that fits instead. The CHURRO authors found major hallucinations in 36 percent of 50 sample outputs from a small open model used without training, which they describe as content that “seems contextually plausible.” In one case the model, unable to read a 19th century English newspaper column that mentioned India and the Queen, produced a passage about the Queen traveling to Calcutta that was not on the page.

That was a small model, and the sample does not tell you how often larger models do the same. The mechanism, though, applies to any model that writes the most probable text when the image is faint, torn or blotted. Your check is simple and tedious: compare the text with the image wherever the image is hard to read.

Silent spelling normalization

Language models tend to write modern spelling. The CHURRO authors added explicit instructions to their prompt after they “noticed most VLMs try to normalize the text to modern English characters.” Humphries and colleagues give the example of a model that might choose “furs” over the historical “furrs” because the modern form is the more probable word.

For a historian, the original spelling can be evidence. A surname spelled three ways across a family’s letters may be how a genealogist proves a connection. Decide whether you want a diplomatic transcription (as written) or a normalized one, say so in the instruction, and check a sample for drift.

Skipped lines, rows and small text

Models can return less text than the page holds. Kim and colleagues found that when they sent a whole table as one image, the output did not always have the correct number of rows, and that outputs mostly came back with fewer characters than the reference. In the CHURRO examples, an untrained model picked up the large title characters and missed most of the smaller text.

Repetition loops and reading order

A model can get stuck. CHURRO documents an untrained model repeating “Das Wetter ist hier sehr angenehm” (the weather here is very pleasant), a common opening for a letter, more than a dozen times over a 19th century German letter it could not read. The same paper found major reading-order errors in 42 percent of a sample of outputs from that untrained model and 16 percent from the authors’ fine-tuned model, most often where pages had two or more columns.

Numbers

A wrong digit reads as naturally as the right one, so number errors are easy to miss. Kim and colleagues flag digits as a persistent difficulty, and the correction instruction Humphries and colleagues used warns the model that “numbers also easily misread.” For ledgers, registers and census-style tables, numbers deserve a separate check.

Self-correction barely helps, and models change

Asking the same model to check its own transcription barely helps. In the Humphries tests, self-correction changed scores by about 1 percent on average, and GPT-4o’s scores got slightly worse. Hosted models also change: Kim and colleagues saw behavior shift several times during their experiment. Record the model name, version and date with every batch, and measure again when any of them changes.

Failure mode How to catch it
Invented text Read the image beside the text on faint, damaged and blotted passages
Modernized spelling Compare a sample with a diplomatic reference; search for known period spellings
Missing lines or rows Compare line and row counts with the image
Repeated lines Search the output for any line that appears more than twice
Wrong digits Check totals, dates and amounts against the image or against column footings
Model drift Re-measure a fixed sample whenever the model or its version changes

Combining an HTR engine with a language model

The best published result so far comes from combining the two: an HTR engine makes the first transcription, and a different language model, given the page image and that draft, corrects it. In the Humphries tests, Transkribus Titan drafts corrected by Claude 3.5 Sonnet reached a strict CER of 3.0 percent and WER of 9.0 percent, and a normalized CER of 1.8 percent and WER of 3.5 percent.

The authors explain the gain this way: HTR engines make character-level errors that a language model can recognize as improbable, such as “furr5,” while a model looking at its own output tends to see its own mistakes as probable text. They also report that nearly four fifths of the errors left after correction were punctuation, capitalization and historical spelling, which matter for publication and much less for search.

Treat this as a promising method to test, not a settled result. Boros and colleagues tested fourteen language models on post-correction benchmarks covering different languages, periods and document types, and concluded that “LLMs are anything but efficient at this task.” Crosilla and colleagues likewise report that language models “demonstrate limited ability to autonomously correct errors in zero-shot transcriptions.” The setups differ, and the Humphries correction step had the page image and used a different model from the one that made the draft. The way to find out which result holds for your collection is to measure it on your pages.

How much human review you need, by intended use

Size the review to what the text will be used for, not to one accuracy target for the whole collection. Uncorrected text with a moderate error rate can serve keyword search well; text that will be quoted, published or turned into data needs a person to read every line against the image.

Intended use What an error costs Review level What the reviewer checks
Finding pages inside your own catalog Some searches miss a page Uncorrected, measured on a random sample Error rate per batch; invented or missing lines on sample pages
Public search and research access A wrong name or date misleads a researcher Targeted correction Names, places, dates, numbers, and lines that read oddly
Publication, exhibition, text of record An error becomes a citation Full correction Every line against the image, to written conventions
Data from ledgers, registers, tables One digit changes an amount, age or date Full check of numbers and structure Row counts, digits, totals, which name sits on which row

Vendor guidance leans generous on the first row. Transkribus says a CER of 20 to 30 percent is “sufficient to work with powerful searching tools like Smart Search,” which describes its own search. Whether that holds in your catalog depends on how your search handles misspellings, so test it with names you know are in the box.

Whatever the level, people review best when the task is narrow. “Check every name, place, date and number on this page” is a task a volunteer or staff member can finish reliably. “Make this perfect” is not, and it drifts toward full correction without anyone deciding to pay for it.

Example (fictional). The Aster County Historical Society tests Box 3 before deciding. A staff member transcribes 6 pages of Ida Okerlund’s letters exactly, giving a reference of 5,400 characters and 1,020 words. Three approaches are scored against it:

  • A general HTR model with no training: 518 character errors, a CER of 9.59 percent and a WER of 28.43 percent.
  • A language model working alone: 335 errors, a CER of 6.20 percent and a WER of 17.65 percent.
  • The HTR draft corrected by a different language model: 214 errors, a CER of 3.96 percent and a WER of 11.57 percent.

The society keeps the third approach. At about 36 errors per page and six seconds per fix, fixing every error would take roughly 36 minutes per 10 pages, plus the time to read every line. Because the letters will serve research and search, it chooses targeted correction of names, places, dates and numbers.

The Wren Gulch Mercantile day book (212 pages) gets different treatment. Its cramped columns and abbreviations score far worse, and on the pilot pages the whole-page output dropped rows, so the society chooses row-by-row processing and full correction. After correction it inspects a random sample of 55 day book pages, the number the QC sample size calculator gives for 212 pages at 95 percent confidence, a 5 point margin and an expected 5 percent defect rate. The fictional example collection follows the same box through every stage.

How to test AI transcription on your own pages

Published error rates tell you what is possible; a small test on your own pages tells you what to expect. The method is the same whatever system you try, and it takes a few days of staff time, most of it spent making the reference transcription.

  1. Decide the use and the conventions first. Write down whether you want spelling as written or modernized, how to handle abbreviations, line breaks, insertions and struck-out words. Transkribus’s guidance makes the same point for training data: choose conventions and “be consistent.”
  2. Choose pages that represent the batch. Include every hand, the faint and damaged pages, the densest pages and any tables. Prefer pages that have never been published online, for the memorization reason above.
  3. Make the reference by hand. A person transcribes the sample to your conventions, and a second person checks it. Avoid starting from a draft made by one of the systems you are testing, because the reference will lean toward that system’s habits; Romein and colleagues report exactly this bias in one of their datasets.
  4. Run more than one approach. A general HTR model, a language model and the combination are a sensible set. Record each system’s name, version, date, settings and the exact instruction used.
  5. Score strict and normalized error rates. The OCR and handwriting accuracy calculator turns your counts of substitutions, deletions and insertions into CER, WER and an estimate of correction hours.
  6. Read the errors, not just the rates. Look for invented phrases, missing lines and rows, loops, changed numbers and modernized names. One invented sentence matters more than many small misreads.
  7. Choose a review level per batch and use. Letters for search and ledgers for data rarely deserve the same treatment.
  8. Measure again during production. Check a random sample of every batch, and run the same reference pages again whenever the model, its version or the instruction changes.

If you want a wider structure for the decision, the Library of Congress’s LC Labs AI Planning Framework moves through three phases, “Understand,” “Experiment” and “Implement,” and looks at the data, the models and the people involved, with a use case risk worksheet you can adapt. A transcription test like the one above is the experiment phase in miniature.

Recording what the machine did

Whatever method you choose, store with every page of text a short record of how it was made and checked. A researcher in ten years should be able to tell machine text from text a person verified, and judge it accordingly.

At minimum, record for each page or batch:

  • Status: uncorrected, corrected for names, places, dates and numbers, or fully corrected.
  • How it was made: the engine or model, its version, the date it ran, and the instruction or custom model used.
  • How good it is: the CER and WER measured on that batch’s sample, and the conventions the reference followed.
  • Who checked it: the reviewer and the date, where a person was involved.

Keep the raw machine output as well as the corrected text, so later corrections can be compared and a better model can be run against the same reference. Our AI policy sets out the labels we use for page text and catalog fields, and the companion article on labeling AI-generated metadata covers how such labels sit in catalog records.

Before sending images to any hosted model, read the provider’s terms on retention and training, and check donor agreements and any restrictions on the material. Records about living people, and material held under community protocols, may not belong on an outside service at all without permission.

What to do next

You can answer the question for your own collection in a short test before committing a budget or a volunteer program. A practical sequence:

  • List the kinds of handwritten material you hold and what each will be used for: search, research, publication or data.
  • Pick representative pages from each, including the worst hand, and make a checked reference transcription.
  • Run at least two approaches and record versions, dates and instructions.
  • Score strict and normalized CER and WER, then read every error on a few pages.
  • Set a review level per batch, label the text by status, and re-measure on a random sample during production.

The fictional example collection shows what labeled page text and records look like at the end of this process. If you would rather have the measuring done on your own material, a pilot box runs a small batch of your real pages through recognition and reports the error rates before anything larger is decided.

Working through this with a real collection?

If this article describes a decision you are facing, tell us about the collection. A pilot box answers most questions with your own material.

Plan a pilot box

Studio and contacts

24 South Weber Street, Suite 225
Colorado Springs, CO 80903
(719) 244-8450sales@dastok.com