Use optical character recognition (OCR) for printed and cleanly typed pages, and handwritten text recognition (HTR), or a multimodal language model, for handwriting. Treat printed forms filled in by hand, and ledgers and registers, as layout problems first, because there the hard part is keeping each name, date and amount attached to its field or row. Ask for plain text plus a layout file with word or line positions (ALTO XML or PAGE XML), add a searchable PDF where you want one file per item, and keep typing by people for small runs, for data where every value counts, and for pages no machine reads well.
That is the short version. Most collections are not one kind of record, though. A single archive box can hold printed clippings, a typed carbon of a reply, a telegraph blank with a written message, a day book in columns and a bundle of letters in three hands. This article works through those kinds one at a time. For each it names the main difficulty, the method to try first, the output to ask for and the point at which people should do the typing.
Two companion articles go deeper on narrower questions. How accurate handwriting recognition and language models really are, and how much review they need, is covered in Can AI read old handwriting?. Why local newspapers so often come out garbled is covered in Historical newspaper OCR: why it fails and what improves it.
OCR, HTR and language models: what each one reads
OCR recognizes printed or typed characters, and HTR recognizes handwriting. Both turn an image of a page into text. Multimodal language models, which read an image and write out text, now do some of both, with their own kinds of mistakes.
Optical character recognition (OCR) reads machine-made letterforms: type, typewriting, print on forms. Modern OCR engines recognize whole lines rather than isolated letters. The open-source Tesseract engine, for example, says its version 4 added “a new neural net (LSTM) based OCR engine which is focused on line recognition.” Its documentation lists more than 100 languages “out of the box” and does not describe handwriting recognition.
Handwritten text recognition (HTR) reads handwriting. Because every hand differs, HTR engines are trained on transcribed pages, and they work best on hands like the ones they learned from. Platforms such as Transkribus first find the text regions and baselines on a page, then recognize each line. Its help pages describe the baseline as “the most important reference point for text recognition.”
Multimodal language models look at a whole page image and write out the text they judge most probable. They need no training on your hand and can read print, typescript and handwriting. They can also invent plausible words, modernize spelling and drop lines, which is why the accuracy article treats them separately.
The useful distinction for planning is not OCR against HTR. It is how regular the letterforms are, and how much the layout matters. Regular letterforms in a simple layout are an easy job for OCR. Irregular letterforms need HTR or a language model. Any page where meaning depends on position, such as a form or a table, needs layout analysis before character recognition matters at all.
The decision matrix by kind of record
The matrix below is the article in one table. Each row is a kind of record you are likely to hold, with the main difficulty, the method to try first, the files to ask for and the point at which typing by people beats recognition.
| Kind of record | Main difficulty | Try first | Ask for | Key by people when |
|---|---|---|---|---|
| Printed books, pamphlets, reports | Old typefaces, long s, ligatures, faded impressions | OCR with a model for the typeface and language | Plain text, ALTO XML, searchable PDF | Rarely; perhaps title pages or short items for a catalog |
| Newspapers and clippings | Columns, article order, small worn type, microfilm | OCR with page layout and reading order | ALTO XML with reading order, plain text per article | Headlines and names you index |
| Typescripts | Uneven strikes, faint ribbons, strikeovers, handwritten notes | OCR, then check annotated pages | Plain text, ALTO XML, searchable PDF | Heavily annotated drafts |
| Carbons and mimeographs | Blurred, spread or broken letters | OCR test on a sample; HTR or a language model if OCR fails | Plain text, searchable PDF, accuracy report | Short runs that test badly |
| Printed forms filled in by hand | Print and handwriting together; values must stay in their fields | Layout or field detection, then OCR for labels and HTR for entries | Field-level CSV, PAGE XML or ALTO | Few forms, or fields that carry legal or vital data |
| Ledgers, registers, census-style tables | Rows and columns, abbreviations, digits | Table detection, then HTR row by row | Spreadsheet by row and column, PAGE XML | When every number must be right and the run is short |
| Letters, diaries, minutes in free handwriting | Many hands, spelling, insertions, faded ink | HTR, a language model, or both, measured on a sample | Plain text, PAGE XML or ALTO, accuracy report | A hand fills fewer pages than you would need to train on |
The rest of the article explains each row, then the output formats, then the decision to type rather than recognize.
Printed books, pamphlets and newspapers
Printed material is OCR’s home ground, and clean modern type usually needs no more than a quick check. The work is in the exceptions: old typefaces, the long s, worn type, faint impressions and pages laid out in columns.
Old typefaces and spellings
An OCR model trained on modern type will stumble over blackletter, the long s and ligatures. Tesseract publishes separate trained data for some scripts and typefaces; its list of data files includes models for Fraktur, the German blackletter, alongside its language models. When you scope a job, tell whoever runs recognition the languages and the typefaces, not just “English.”
Decide also what you want done with period spelling and letterforms. A long s can be kept as the historical character or rendered as a modern s. A diplomatic transcription keeps it; a searchable reading text often normalizes it. Either choice works if it is written down and applied the same way across the batch.
Image quality comes first
Tesseract’s own guidance on improving output quality is a useful checklist even if you never use Tesseract. It says the engine “works best on images which have a DPI of at least 300 dpi,” that a skewed page reduces “the quality of Tesseract’s line segmentation” significantly, and that the dark borders around scanned pages “can be erroneously picked up as extra characters.” Most of those problems are fixed at capture, which is why recognition planning belongs in the scanning specification, not after it. The scanning resolution calculator helps set capture resolution from the size of the smallest type.
Newspapers are a layout problem
On a newspaper page, recognizing the letters is the easier half. The engine must also find the columns, separate articles and advertisements, and put the text in reading order, or a search hit will land in a sentence stitched from two different stories. ALTO, the Library of Congress layout format discussed below, added explicit reading-order support in version 4.3.
Ask for ALTO XML with the position of every word, so a viewer can highlight hits on the page image. The Library of Congress’s ALTO page links a page of The Winchester News from Chronicling America as a live example. Microfilm generation loss, article segmentation and re-processing old OCR are covered in the newspaper article linked above.
Where language models fit for print
Language models are not only for handwriting. In the CHURRO benchmark, published at EMNLP 2025, the authors report that “multiple VLMs outperform OCR systems” on both their printed and handwritten historical pages. For a large print run, though, a conventional OCR engine remains the sensible first choice: Tesseract is open source under the Apache 2.0 license, it can write word positions, and its errors look like misreadings rather than invented sentences. Try a language model where OCR fails on a particular typeface or a damaged page, and measure the result.
Measure print as well
Print is easy, not error-free. Transkribus’s guidance on error rates, a vendor’s rule of thumb rather than a standard, puts a good character error rate for printed text “between 0.5 and 2%.” Even 1 percent is about a dozen errors on a page of 1,200 characters, and they cluster in the places that matter for search: proper names, numbers, hyphenated words split across lines and words in italic or small capitals. Measure a sample of your oldest and most worn print, not the cleanest pamphlet in the box, and decide whether names and dates need a correction pass.
Typescripts, carbon copies and mimeographs
Clean typescripts read well with OCR, because typewriter letterforms are regular. Carbon copies, mimeographs and faint ribbons are another matter: blurred or broken letters can defeat OCR, and some of these pages are better treated as handwriting or simply typed again.
Top copies and ribbon typescripts
A typed top copy has even letterforms in a single typeface, often in a monospaced grid, and OCR engines generally handle it well. The usual troubles are local: a dry ribbon leaving pale letters, keys that strike unevenly, letters typed over each other to correct a mistake, and words struck out with a row of x’s. Tesseract notes that “Bold characters or Thin characters (especially those with Serifs) may impact the recognition of details and reduce recognition accuracy,” which describes both heavy and faint typing.
Typescripts in archives are seldom clean, however. Drafts carry handwritten corrections between the lines, signatures, initials and marginal notes. OCR will either skip the handwriting or turn it into noise. If the annotations matter (and in a draft they often matter most), plan a second pass for them, by HTR, a language model or a person.
Carbons, flimsies and mimeographs
A carbon copy is made by pressure through carbon paper, so its letters are often softer and wider than the top copy’s, and thin onionskin “flimsies” can show text from the other side. Mimeographs and other duplicated pages can be patchy, with filled-in counters in letters such as e and a. These defects change from page to page, so no published rule tells you in advance whether a set of carbons will read well.
Test before you commit. Run OCR on a sample of the worst carbons, measure the character error rate against a careful transcription, and look at the errors. Tesseract’s guidance notes that heavy “ink bleeding from historical documents can be compensated for by using an Erosion technique,” a form of image processing that thins swollen letters, so preprocessing can help. If OCR still fails, try an HTR model or a language model, which do not depend on clean, regular letterforms. If the carbons are few and important, such as the only surviving copy of outgoing letters, typing them may be the cheapest accurate answer.
What to keep together
Carbon copies are often the outgoing half of a correspondence whose incoming half is handwritten. Keep the recognized text of both in the same folder order, with the same identifiers, so a reader can follow an exchange. The recognition method can differ by page; the arrangement should not.
Printed forms filled in by hand
Printed forms combine two kinds of text on one page: printed labels and handwritten entries. The job is not just reading both, but keeping each entry attached to the label it answers, so treat forms as a layout or field-detection problem first and a recognition problem second.
Certificates, registration cards, applications, telegraph blanks and census-style schedules all work this way. If the engine reads the page as running text, “Name of deceased” and the handwritten name beside it can end up on different lines, or the name can be attached to the next field’s label. The text is all there, but its meaning is lost.
What the research shows
Kim and colleagues tested two language models and four conventional tools on 20 pages of 1921 Belgian estate declarations: tables with typed column headings and handwritten entries (preprint, January 2025). The conventional tools, used without training, read only some of the typed heading words and could not read the handwriting. The language models read names that the conventional tools could not, and did better when each page was cut into rows than when it was sent whole.
The same study shows a measuring trap. The typed headings, 152 words repeated on every page, are easy to recognize and easy to leave out, and they swing the scores. For one approach, setting the headings aside dropped the median character error rate from 0.72 to 0.17. When you measure accuracy on forms, score the handwritten entries separately from the printed boilerplate, or the boilerplate will flatter or punish the result.
Approach and output
The working approach has three steps. First, detect the fields: either draw them once on a template, when every form is the same edition, or train a model to find them. Transkribus offers trainable “field models” and recommends tagging “at least 50 pages, depending on the complexity of your document’s layout” before training one, according to its field models help page. Second, recognize each field with the right method: OCR for printed labels and typed entries, HTR or a language model for handwritten ones. Third, export by field.
The output to ask for is a table with one row per form and one column per field, plus a layout file (PAGE XML or ALTO) that records where each value sits on the image. Ask that every value carry the identifier of its page image, so a reviewer can check a doubtful entry against the original in one click.
Card indexes
Card indexes are a special kind of form, and many historical societies, libraries and county offices hold one: obituary files, cemetery cards, accession cards, membership rolls. Typed cards of one design read well with OCR once the card’s fields are mapped. Handwritten cards behave like short forms in many hands. Because each card holds only a few lines, a long run of handwritten cards is often typed rather than recognized, and the typing can go straight into the columns of a spreadsheet that later imports into your catalog.
When to type forms instead
Forms are where typing by people is most often right. If you hold a few hundred cards of one design, setting up field detection may take longer than keying them. If the fields hold vital data (names, dates of birth and death, ages, amounts), a person will have to read every value anyway, and typing directly into a form-shaped spreadsheet may be quicker than correcting machine output.
Ledgers, registers and other handwritten tables
In a ledger or register, layout analysis matters more than character recognition: the hard part is finding the rows and columns so that each name stays with its date, item and amount. Get the table structure right first, then recognize cell by cell or row by row, and check the numbers separately.
General OCR tools struggle here. Tesseract’s documentation says plainly that “it is known tesseract has a problem to recognize text/data from tables … without custom segmentation/layout analysis.” Handwritten tables add abbreviations, ditto marks, running totals and entries that cross column lines.
Table detection
Specialized platforms train models to find a table’s structure. Transkribus’s table models help page says such models “are not an all-in-one or out-of-the-box solution”: they are trained on a particular document or collection. It recommends about 20 pages of ground truth for easy tables, 50 for difficult ones (“uneven rows, skewed tables, a table across two slightly off-set pages”) and between 50 and 100 for a mix of table types. It also warns that table models “may encounter difficulties with very narrow rows and columns.”
Those numbers decide a lot. Fifty pages of hand-drawn table structure is reasonable for a run of 2,000 register pages in one format. It is a quarter of a 212-page day book.
Row by row, and the digits
Once rows are found, recognizing them one at a time keeps each entry together. Kim and colleagues found their language models did better when the table image was cut into rows. They also report that “LLMs have a hard time reading digits,” and that sending a whole page did not always return the right number of rows.
So check the structure and the numbers separately from the words. Count rows in the output against rows on the image. Where the ledger has column totals or carried-forward balances, add up the recognized amounts and compare; a mismatch points straight to a misread digit or a dropped line.
Output for tables
Ask for a spreadsheet with one row per ledger line and one column per ledger column, with the page identifier and row number on every row. Transkribus, for example, can export recognized tables to a spreadsheet, with an option to include “a single column containing image snippets of the table cells” for checking, according to its export documentation. Also keep the layout file, so the table can be re-read later with a better model without redrawing it.
Letters, diaries and other free handwriting
Free handwriting is HTR’s main job, and increasingly a job for multimodal language models too. Choose between them by measuring on a sample of your own pages, and decide how much correction the intended use needs before the work starts.
Letters, diaries, journals, minute books and commonplace books share the problems of handwriting: many hands, personal abbreviations, period spelling, insertions above the line, crossed letters written in two directions to save paper, and faded or bleeding ink. The deciding question is usually how many pages each hand fills.
Long runs in one hand
When a hand fills hundreds of pages, such as a clerk’s minute books or a diarist’s volumes, training an HTR model on that hand usually pays off. Transkribus’s data preparation guidance says “between 5,000 and 15,000 words (around 25-75 pages)” of transcribed material are required to start, and recommends “at least 10,000 words for each hand.” Once trained, the model reads the rest of the run quickly and consistently.
Many hands, few pages each
A box of family letters from twenty correspondents is the opposite case. Transcribing 10,000 words per hand to train models would cost more than typing the letters. Here a general HTR model, a language model, or a language model correcting an HTR draft is the practical choice, with a careful check on names, places and dates. The accuracy article reports published results for each of these, with their conditions.
Output for handwriting
Ask for plain text per page and a layout file that ties each line of text to its place on the image. HTR engines work line by line, so line positions are reliable; word positions are usually estimated. Transkribus notes in its export documentation that a highlighted search term in its PDFs “may not perfectly match the word in the image because the word coordinates are determined from the lines with a certain degree of fuzziness.” A language model used on its own returns plain text with no positions at all, so if hit highlighting matters, the text has to be aligned to the image in a separate step.
Which output to ask for: text, searchable PDF, hOCR, ALTO or PAGE XML
Ask for plain text for every page, a layout file with positions (ALTO XML for most library and archive systems, PAGE XML where handwriting and complex layouts dominate), and searchable PDFs only where you want one file per item for reading or download. hOCR is a sound alternative layout format if your tools use it.
| Format | What it is | What it holds | Typical use |
|---|---|---|---|
| Plain text | One text file per page or item | The words, in reading order, nothing else | Full-text indexing, quick reading, reuse in other tools |
| Searchable PDF | Page images with a hidden text layer | Images and text, sometimes positions | One file per item to read, download and search in a browser |
| hOCR | OCR results written as a subset of HTML | Text with bounding boxes, word confidence, layout elements | Pipelines built on Tesseract; viewable in a browser |
| ALTO XML | XML layout standard maintained by the Library of Congress | Text with page, block, line and word positions, word and character confidence, reading order | Library and archive repositories, newspaper programs, viewers that highlight hits |
| PAGE XML | XML format from the PRImA Research Lab | Regions, text lines, words, glyphs, reading order, text content | HTR platforms, complex layouts, research and model training |
Plain text
Every delivery should include plain text, one file per page, in reading order. It is the format every search engine and every future tool can read, and it is what a person copies into a citation. Its weakness is that it says nothing about where words sit on the page.
Searchable PDF
A searchable PDF holds page images with an invisible text layer, so a reader can search and copy text while looking at the original. The Internet Archive, for example, builds PDFs “containing MRC-compressed images and the OCR result as a hidden (selectable, searchable) text layer,” according to its OCR documentation. Tesseract can write both a full PDF and an “invisible-text-only PDF.”
Searchable PDFs are convenient access copies. They are not a substitute for preservation masters or for a layout file: the text layer is hard to correct and re-use, and a PDF compresses the images. Ask for them in addition to the other files, not instead.
hOCR
hOCR describes OCR results as HTML. The hOCR specification 1.2, a “Living Standard” dated February 26, 2020, defines “an open standard for representing document layout analysis and OCR results as a subset of HTML.” Each line or word carries a bounding box, and engines can record a word confidence. The Internet Archive chose hOCR when it moved its OCR to Tesseract, a transition completed near the end of 2020, and its hOCR files carry “text, bounding boxes, and confidence at the word level.”
ALTO XML
ALTO (Analyzed Layout and Text Object) is the layout format most library and archive systems know. It was developed in the METAe project for use with METS, and the Library of Congress became its official maintenance agency in August 2009, with an editorial board, according to the About ALTO page. Version 4.4 is the current official schema.
ALTO records the position of each block, line and word on the page image. Its schema also allows a word confidence (the WC attribute, from 0 for unsure to 1 for sure) and a per-character confidence (CC, from 0 for sure to 9 for unsure; note the reversed scale). Those confidence values are what let a reviewer jump straight to the words the engine doubted. Ask that they be included.
A confidence score is the engine’s own estimate, not a measurement. Before you use it to decide which words a person checks, compare it with the errors found on your sample pages: if most real errors sit in low-confidence words, targeted review by confidence works for that batch; if many errors carry high confidence, it does not. Language models used on their own usually give no comparable per-word score, which is one more reason to keep a conventional engine’s output alongside theirs.
PAGE XML
PAGE XML comes from the PRImA Research Lab and is common on HTR platforms. Its format collection describes it as covering “regions, text lines, words, glyphs, reading order, text content” and more. Transkribus lists PAGE XML among its standard exports, and ALTO and TEI among its advanced exports on some plans, as of November 2026. That makes it a good fit for handwriting and complex layouts, where regions and lines matter more than a simple column of words.
TEI for editions
If the goal is a scholarly edition rather than search, the encoding most editions use is TEI XML, which records deletions, insertions, expansions of abbreviations and editorial notes. Transkribus describes its TEI export as “suitable for academic research and digital humanities projects.” A TEI file is usually built from corrected text by an editor, so treat it as a product of the editing stage, not of recognition.
Which engines produce what
Check that the engine or service you use can produce the format your system needs before you start. Tesseract lists “plain text, hOCR (HTML), PDF, invisible-text-only PDF, TSV, ALTO and PAGE” as outputs. Transkribus exports PAGE XML, plain text, Word files, PDFs with embedded text and, on some plans, ALTO, TEI and spreadsheet tables. Converting one layout format into another adds a step that can lose detail, so it is better to receive the format you need. The collection search you publish through will decide which layout format matters most, and viewers that show search hits on the page image need word positions.
When typing by people is still the right call
Recognition is not always cheaper or better than having a person type the text. Typing wins when the run is short, when every value must be right, when the page defeats machines, or when the transcription needs editorial judgment that a person would have to apply anyway.
Use these decision rules:
- The run is shorter than the training it would need. If a hand or a table format fills 30 pages, and a model would need 25 to 75 pages of ground truth, type the 30 pages.
- Every value must be exact. Vital records, legal descriptions and financial figures will be read in full by a person in any case. If correcting machine output takes nearly as long as typing, type.
- The items are short. Inscriptions on photograph mounts, captions, labels and envelope addresses are a few words each. Typing them during cataloging is faster than routing them through recognition and review.
- The page defeats machines. Very faded, water-damaged or heavily crossed pages produce error rates so high that correction becomes retyping.
- The text needs editorial decisions. Scholarly editions record deletions, insertions, uncertain readings and expanded abbreviations. A person makes those decisions, and machine drafts may get in the way.
Typing has costs of its own. People make errors too, so check a sample of typed text just as you would check machine text, and record who typed it and when. For larger keying projects, you can have two people type the same pages independently and compare the results; whether that is worth it depends on how much an error would cost.
Volunteers are another way to type, and they bring local knowledge of names and places that no model has. Planning volunteer work around machine drafts, or instead of them, is the subject of Volunteers, AI or both?.
Sorting a mixed box before recognition
Most collections need several methods at once, so the practical first step is to sort what you hold by kind of record, count pages for each kind, and choose a method and an output per group. A short inventory pass does most of this work.
Example (fictional). Box 3 of the Aster County Historical Society holds the Okerlund family papers and the Wren Gulch Mercantile day book, 1893 to 1896. Sorted by kind of record, it becomes five recognition jobs:
- Letters (41 items, 96 pages) from Ida and Walter Okerlund: free handwriting in two hands, with Ida writing 35 of the 41 letters. HTR with a language-model correction pass, measured on a sample of each hand. Output: plain text and PAGE XML per page, labeled by correction level.
- Store day book (1 volume, 212 pages): a handwritten table of dates, customers, goods and amounts. Table structure first, then row-by-row recognition and full correction, with amounts checked against page totals. Output: a spreadsheet with one row per entry and the page and line on every row.
- Clippings (8 items) from the Wren Gulch Courier: small newspaper type. OCR, delivered uncorrected and labeled as such, because the printed text reads cleanly. Output: plain text and ALTO XML.
- Telegram (1 item): a printed telegraph blank with a message of eight words. Typed by a person.
- Cabinet cards (18 items): studio imprints on the mounts and short pencil or ink notes on the backs. Typed into the catalog record by the cataloger, because each inscription is a few words.
Two of the five jobs end with a person typing. That is normal for a mixed box, and it keeps the recognition budget for the pages that need it.
The fictional example collection follows this same box through capture, recognition, cataloging and return, with the error rates measured for each batch. A collection inventory worksheet gives you columns for format, dates, item counts and condition, which is the information this sorting needs.
What to put in your request
Write the decisions above into whatever document starts the work: a statement of work for a vendor, a project plan for your own staff or a grant narrative. A short list per batch prevents most disappointments.
For each batch, state:
- The kind of record and its page count, using the groups in the matrix above.
- The languages, scripts and typefaces, including blackletter, shorthand or anything unusual.
- The method, or that the method will be chosen on measured results from a sample.
- Transcription conventions: spelling as written or modernized, abbreviations, line breaks, insertions and deletions.
- The outputs: plain text per page, ALTO or PAGE XML with positions and confidence values, searchable PDFs if wanted, spreadsheets for forms and tables.
- The accuracy measure: character and word error rates on a sample transcribed by a person, reported per batch and per kind of page. The OCR and handwriting accuracy calculator shows the arithmetic.
- The correction level: none, targeted (names, places, dates, numbers) or full, chosen per batch by intended use.
- What people will type, and who checks it.
- How machine text is labeled, so later users know what was checked.
Mistakes that cost the most
A few scoping mistakes cause most of the trouble later:
- One method for the whole collection. A single price per page for “OCR” hides the ledgers and letters that OCR cannot read.
- Measuring on the cleanest pages. A sample of the neat hand and the crisp print predicts nothing about the faded carbons and the worst hand.
- Scoring forms with their boilerplate. Printed labels repeated on every form inflate accuracy figures, as the estate declaration study showed.
- Asking only for PDFs. Text locked in a PDF layer is hard to correct, re-index or move to a new system. Always get plain text and a layout file too.
- Leaving conventions unwritten. Two people correcting the same batch to different spelling rules produce text that searches inconsistently.
If you are comparing vendors, the same list makes their proposals comparable. Our own text recognition service describes how we sample, measure and correct by kind of page. A pilot box of your hardest material, a carbon, a ledger page and a letter in the worst hand, is the quickest way to see which row of the matrix each of your records really belongs in.