Plan a pilot box
Menu

Historical newspaper OCR: why it fails and what improves it

Reader question: Why is the OCR on our digitized local newspapers so poor, and what can we do about it when we digitize or reprocess?

By Dastok Editorial TeamIssue: Published

Poor OCR on old newspapers usually comes from the source, not the software. Small, worn type on acidic paper, microfilm made decades ago, pages of six or seven columns read in the wrong order, and scans made for reading rather than for recognition all do more damage than the choice of engine. What improves it, in the order a project meets each step: capture from the best source you can get at 300 to 400 ppi in grayscale, get the columns and reading order right in an ALTO file with word positions, re-run recognition with a current engine, and correct the words that searches depend on (headlines, names and places).

This article is written for a public library, historical society or county archive with its own town paper: a run of bound volumes in the back room, reels of microfilm made in the 1960s or 1970s, or a set of page images that someone digitized years ago and that nobody can search well. It draws on what national programs measured when they did the same work at scale, from the National Library of Australia, the British Library and the Library of Congress’s National Digital Newspaper Program (NDNP).

How newspapers compare with letters, ledgers and typescripts is covered in OCR or HTR? Matching text recognition to each kind of record. This article stays with newsprint.

Why newspaper OCR is harder than book OCR

Newspapers combine almost everything that makes recognition hard: tiny type, dense columns, mixed typefaces, advertisements, cheap paper and, for most runs, a microfilm copy standing between the page and the scanner. Each of these problems is manageable on its own; together they explain why newspaper text lags behind book text from the same period.

The Library of Congress sums up the main causes in an April 2025 post on reprocessing Chronicling America: “Original newsprint can be damaged or suffer from deterioration before it was microfilmed,” “poor quality microfilming practices can lead to subpar images,” and “historical newspapers often exhibit tight columns and tiny text sizes that further complicate the OCR process.” The same post reminds users “that achieving 100% accuracy in an automated OCR process is virtually impossible.”

The paper

From the mid-19th century, newspapers were printed on cheap wood-pulp paper. The Library of Congress’s guide to preserving newspapers describes these papers as “highly acidic and prone to swift degradation.” By the time a run was microfilmed or reaches your scanner, pages may be yellowed, torn at the folds, missing corners or stained. Every one of those defects removes contrast between ink and paper, and contrast is what OCR reads.

The type and the layout

Newspaper body type was small, set in narrow columns with little space between them, and often printed from worn type with uneven inking. Headlines, mastheads and advertisements mix display faces, sizes and decorative borders on the same page. An OCR engine must first find the columns and blocks, decide their reading order, and only then recognize characters. If it misjudges the layout, the words can be right and the text still wrong.

Period typefaces

Older runs bring older letterforms. English printing of the 18th and early 19th centuries often uses the long s, which looks much like an f; Tanner and colleagues single it out as the reason the British Library’s Burney collection searches so poorly. Blackletter faces such as Fraktur, which NDNP notes “were predominately used for German language text,” need their own handling. NDNP requires that text “printed in Fraktur/black letter fonts must incorporate technical processing that includes Fraktur/black letter specific tools,” and warns that this “may represent additional cost.” Tesseract, for example, publishes separate trained models for Fraktur. Tell whoever runs recognition which typefaces and languages appear in which years.

The microfilm

Large newspaper programs mostly scan from microfilm. NDNP’s technical approach is “primarily from microfilm,” and Holley notes that digitizing from original hard copies costs “considerably higher than for using microfilm.” Film adds its own losses: reduction to a small frame, the quality of the original filming, density that is too heavy or too light, scratches, and generation loss each time a copy is made from a copy. As Holley put it in a 2009 report on Australia’s newspaper program, quoting a Dutch National Library survey, “the quality of the OCR text says more about the condition of the original materials than it does about the performance of the OCR software.”

What national programs measured

Large programs that tested their own newspaper OCR found accuracy that varied widely by title and period, and was often far below vendors’ claims. Their measurements are old, but they remain the clearest published evidence of why local newspaper text disappoints, and of which words suffer most.

The British Library’s 19th century newspapers

In 2009, Tanner, Muñoz and Ros published an assessment of the British Library’s 19th Century Newspapers database: 48 national and regional titles, about 2 million pages. They sampled roughly 1 percent of the pages, took two zones of 100 words from each, had people key those zones twice, and compared the keyed text with the delivered OCR.

Their averages for the 19th century collection were 83.6 percent character accuracy and 78 percent word accuracy. For “significant words” (the content words a person might search for, leaving out common words such as “the” and “it”) accuracy fell to 68.4 percent; for words starting with a capital letter, mostly names and places, to 63.4 percent; and for groups of numbers to 64.1 percent. The older Burney collection of 17th and 18th century papers did worse on every measure, and the authors note that its long s “reduces word accuracy to such a low point that searching can become very difficult.”

Average OCR accuracy, higher is better 19th Century British Library Newspapers Burney Collection (17th and 18th century) 0% 50% 80% 100% Characters 83.6% 75.6% Words 78.0% 65.0% Significant words 68.4% 48.4% Capitalized words 63.4% 47.4% Number groups 64.1% 59.3% 80% word accuracy
Average OCR accuracy measured on samples of two British Library newspaper collections, by kind of word. The study sampled the clearest zones of each page, so these are best-case figures. Source: Tanner, Muñoz and Ros (2009), D-Lib Magazine, checked November 2026.

Two details make these figures worse than they look. First, the sampled zones were chosen from “amongst the clearest on the page image,” because the study wanted a “high water mark,” the best accuracy the material was likely to reach. Second, the words that matter most for search, names and places, are exactly the ones recognized worst. The authors suggest that “should the word accuracy be greater than 80%,” most fuzzy search engines can fill the gaps, and that below 80 percent “the search capacity will drop off steeply.” In their sample, only about a quarter of the titles had significant-word accuracy above 80 percent.

Australia’s newspaper program

Rose Holley’s 2009 report on the Australian Newspapers Digitisation Program at the National Library of Australia tested 45 pages chosen to represent its collection from 1803 to 1954. Raw OCR “varied from 71% to 98.02%,” measured as character confidence with no correction. At 71 percent, the report notes, “there would be 145 incorrect characters in an average paragraph of 500 characters.”

The report also records a rough consensus among libraries and contractors at the time: 98 to 99 percent counted as good newspaper OCR, 90 to 98 percent as average, and below 90 percent as poor. It adds the honest caveat that there was no agreement on whether those figures meant characters or words, pages or articles.

Chronicling America today

The National Digital Newspaper Program, run by the National Endowment for the Humanities and the Library of Congress since 2005, supports Chronicling America, which now holds “over 23 million newspaper pages,” according to the Library’s April 2025 post. That post announced a new workflow, built on the open-source Tesseract engine with “customized post-processing tools,” to reprocess pages digitized in the program’s early years, with over 170,000 pages done at the time of writing. If a national program with its scale finds early OCR worth redoing, a local run digitized in the 2000s probably is too.

Why one accuracy percentage hides the problem

A single figure such as “95 percent accurate” says almost nothing about whether people will find what they search for. It may count characters rather than words, it may be the engine’s own confidence rather than a measurement, it may come from the cleanest part of the page, and it cannot see text that was put in the wrong order or invented.

Characters are not words

Tanner and colleagues give the arithmetic. On a page of 1,000 words and 5,000 characters, 90 percent character accuracy means 500 wrong characters. Depending on how those errors fall, that “might convert to a maximum of 900 correct words (90% word accuracy) or a minimum of 500 correct words (50% word accuracy).” Ask for word accuracy, and better still for accuracy on names, places and numbers, measured separately.

Confidence is not accuracy

OCR engines attach a confidence score to each character or word, and those scores are easy to report. Holley warns that “OCR contractors often talk about OCR confidence levels and OCR accuracy as if they were the same thing,” while “the OCR software doesn’t know whether any character is correct or not.” True accuracy needs a person: a careful transcription of sample text compared with the OCR. Confidence values are still useful, for pointing reviewers at doubtful words, but they are not a measurement.

Layout failures do not show up in character counts

A recent survey makes the structural point sharply. Beyene and Dancy’s 2026 review of OCR evaluation, prepared for the ACM FAccT conference, found that most evaluations “emphasize character accuracy and task success on modern layouts” and “rarely capture structural failures common in historical newspapers, including column collapse, typographic errors, and hallucinated text.” In their worked example of a 19th century Black newspaper, Tesseract 5 “misinterpreted vertical column rules as text” and “defaulted to left-to-right” reading across columns, merging separate items, while a vision-language model “overwrote” what was printed with likely-sounding text, replacing names with fabricated ones.

Column collapse is the failure local users notice first: a search hit lands in a sentence that runs from a council report straight into a patent-medicine advertisement. Every character can be right and the text still be useless for reading, quoting or text mining.

Where the errors cluster

Errors are not spread evenly across a page or a run. Tanner and colleagues avoided “graphic, tabular or list type content” when they chose their zones, so their averages do not describe market tables, shipping lists or columns of names. They also note that titles rich in “lists of traders, births and deaths listings” show higher apparent error rates, because names are harder for OCR engines. Those are often the very columns genealogists come for, so measure them separately.

Paper or microfilm: choosing the source to scan

Scan the best source you can get, and judge “best” by a test, not by assumption. Original newsprint, where it survives in good condition, gives the scanner the most information; a clean early-generation microfilm negative is the usual practical choice; a worn service copy of film is the worst starting point.

The national guidance points both ways, which is why a test matters. Holley’s table of factors recommends using “original hard copies if budget allows,” noting that “digitisation costs will be considerably higher than for using microfilm,” and that hard copies should be “the most complete and cleanest version possible,” preferably not bound. Yet the same report lists switching from microfilm to hard copies as not an option for its program and says it “will also not necessarily give better results.” Its advice on film is specific: use microfilm made “after establishment and use of microfilm imaging standards (1990’s or later),” and use “master negative microfilm only (first generation)” rather than later copies.

NDNP’s technical guidelines for 2026 to 2028 awards, dated August 1, 2025, base the program on grayscale images scanned “primarily from microfilm,” from “a clean second-generation duplicate silver negative.” NDNP accepts a second-generation duplicate negative where Holley preferred the first generation, and its guidelines allow pre-approved amounts scanned “from the best extant copy including original newsprint, positive microfilm, microfiche, and preservation facsimile or reprint copies.”

Source When it is the better choice Watch for
Original newsprint, loose or bound Film is poor or missing; pages are complete and not too brittle to handle Handling risk; bound volumes curve into the gutter; higher capture cost
Camera negative or a clean duplicate negative The usual best film source; the filming met standards Density problems, scratches; you may need permission to use a master
Positive service copy Nothing better exists Generation loss, scratches from years of reader use
Existing page images Images are grayscale TIFF at adequate resolution Bitonal or heavily compressed images limit what re-OCR can do

For a local run the choice is often forced: the originals were discarded after filming, or the film was never made. Where both exist, scan the same few pages from each and run them through the same OCR. Our document scanning service covers newspapers and bound volumes when originals are the better source, and FADGI’s guidelines say plainly that “fragile newspapers should only be imaged on scanners that do not stress the paper.”

Capture settings that help OCR

Capture newspapers in 8-bit grayscale at 300 to 400 pixels per inch relative to the original page, keep an unenhanced master, and do any cropping and de-skewing before OCR so the text coordinates match the image. Those settings come from both national guidelines and are within reach of any project.

What the guidelines specify

The FADGI Technical Guidelines (third edition, 2023) set newspaper sampling rates of 300 ppi at three stars and 400 ppi at four stars, at 8 bits, and state: “Three-star performance specifications are recommended for newspaper digitization. This performance level has been tested with OCR and found to provide excellent results with a variety of languages and conditions of the newspaper.” The FADGI star system is explained in more detail in our article on FADGI star ratings.

NDNP’s summary of scanning guidelines asks for the “maximum resolution possible between 300-400 dpi, relative to physical dimensions of the original material; 8-bit grayscale; TIFF 6.0 uncompressed.” For film, that means knowing the reduction ratio: 400 ppi on a page that was filmed at a large reduction requires far more resolution at the film plane. NDNP also asks for pages filmed two to a frame to be split into separate images, and for pages with more than 3 degrees of skew to be straightened, noting that “greater skew leads to less accurate OCR.”

Grayscale masters, and what OCR sees

Two details from the national programs are worth copying. NDNP requires that all operations that change image geometry (cropping, de-skewing) be made “to the TIFF before OCR, since the OCR output is expected to include bounding-box coordinates,” and that the grayscale master have “no other enhancements (bitonalization, sharpening, contrast enhancement, etc.) used in the OCR-creation process.” In other words, enhance a working copy for the OCR engine, but keep the master clean.

Holley’s tests are a useful corrective to vendor claims. Running OCR on grayscale rather than black-and-white (bitonal) files made “no significant improvement”: 93.8 percent for bitonal against 94 percent for grayscale on the program’s average pages. Trying other contractors’ image optimization gave differences the program judged “negligible.” The lesson for a small project is not that grayscale is unnecessary (it is the right master format, and Tesseract, for one, converts to black and white internally) but that preprocessing tweaks rarely rescue a poor source.

Bound volumes, color and the files to keep

Bound newspaper volumes need care at capture. Pages curve into the gutter, so text near the spine distorts and loses focus, and brittle paper can crack at the fold. FADGI lists planetary scanners and digital cameras among recommended technologies for newspapers, and warns that fragile papers should only be imaged on equipment that does not stress them. Never cut a volume apart to make scanning easier without a conservation decision; the binding is part of the record.

Grayscale suits most newsprint. NDNP permits color, with approval, for “materials that merit color imaging (i.e. use of color inks or high-end graphic techniques),” such as color supplements and comics. FADGI’s newspaper table allows grayscale or color at three stars and calls for color at four.

Alongside the master and the ALTO file, NDNP asks for a searchable PDF for each page, an “Image with Hidden Text” for printing and download, and a JPEG 2000 file used for zooming in its viewer. Those are useful access copies for any project, but the search itself runs on the ALTO text, and the PDF should be regenerated whenever the OCR improves.

Layout, reading order and ALTO files

Ask for ALTO XML with word coordinates and text ordered column by column, one file per page. Decide separately whether you need articles marked as units, because article segmentation is a real added cost and national programs differ on it.

What NDNP asks for

NDNP requires that “each page digitized for NDNP must be accompanied by OCR text encoded using the ALTO (Analyzed Layout and Text Object) XML schema.” Its summary lists one OCR file per page image, named to match the image; UTF-8 text; “OCR text ordered column-by-column (that is, in a natural reading order)”; and “bounding-box coordinate data at the word level.” If possible, the files should also carry confidence data and font information. Text must carry a language code by text block, and pages printed in Fraktur or blackletter “must incorporate technical processing that includes Fraktur/black letter specific tools.”

The word coordinates are what let a viewer highlight a search hit on the page image. The Library of Congress’s ALTO standard page links a Chronicling America page as a live example, and ALTO version 4.3 added explicit reading-order structures for complex pages.

Article segmentation is a choice, not a given

Many people assume newspaper OCR divides pages into articles. NDNP deliberately does not. Its technical approach is “OCR with word-bounding boxes, with recognition of columns, but without segmentation of pages into articles,” and it states that “zones for articles will not be used in the interface,” asking awardees to remove such data before delivery.

Other programs chose differently. The Australian program delivered text by article and paid its contractor to correct “article title headings, subtitles and the first four lines of the article to 99.5% accuracy,” so that search results showed readable headlines. Article-level text is easier to read, cite and clip, and it keeps a search for two words from matching across unrelated stories. It also costs more to produce and check. For a small run, column-ordered page text with good coordinates is the sound baseline; add article segmentation where your users need to cite or share individual items, and price it separately.

Re-running OCR on pages you already have

If your newspaper was digitized ten or more years ago, re-running recognition on the existing images with a current engine is often the cheapest improvement available. It works only if you still have good images, and it should be tested on a sample before the old text is replaced.

The Library of Congress’s 2025 reprocessing of early Chronicling America pages shows the approach at scale: new OCR from existing page images, using Tesseract and “customized post-processing tools,” so that articles whose text “was mostly unrecognizable” now appear in search results. For a local project the steps are the same, on a smaller scale.

  1. Check what images you have. Grayscale TIFF masters at 300 ppi or more relative to the page are ideal. Bitonal images, small JPEGs or PDFs with heavy compression limit what any engine can recover, and may mean rescanning is the better investment.
  2. Keep the geometry. New OCR must be made from images with the same cropping, size and orientation as the images people will see, or the word coordinates will not line up. NDNP’s guidelines make the same point for corrections: “Any text correction must retain the integrity of the ALTO positional coordinates.”
  3. Set the languages and typefaces. Tell the engine which languages to expect, and use models suited to Fraktur or blackletter where the paper used them.
  4. Test before replacing. Run the new engine on a random sample, measure it against keyed text as described below, and compare with the old OCR on the same zones. Replace only when the new text is measurably better.
  5. Keep the old files. Store the previous OCR alongside the new, with dates and engine versions, in case a later comparison is needed.
  6. Re-index search. New text helps only once the search index is rebuilt from it. If a vendor hosts your search, ask how reprocessed text is loaded.

Re-OCR is the right first step when the images are good and the text is old. Rescanning is the better investment when the images are bitonal, small or heavily compressed, when the film used was a worn service copy and a better negative exists, or when the originals survive in better condition than the film suggests. A side-by-side test on a dozen pages usually settles it.

Language models can also produce text from newspaper pages, and sometimes read damaged type better than conventional OCR. They also bring the risk the 2026 survey documents, text that reads well but is not on the page. If you try one, keep its output separate, label it, and measure it the same way; the companion article on AI handwriting recognition accuracy explains the failure modes to check for.

Correcting the words search depends on

Full correction of a newspaper run is rarely affordable, but correcting the right words is. Headlines, names, places and the first lines of articles carry most of the value for search, and they are the words OCR gets wrong most often.

Targeted correction

Tanner and colleagues found accuracy on capitalized words, mostly names and places, about five points below accuracy on significant words generally, and significant words about ten points below all words. Those are the words people type into a search box. The Australian program acted on this early: its contractor corrected headings, subtitles and the first four lines of each article to 99.5 percent accuracy, so that search results were readable even where the body text was poor.

For a local paper, good targets are mastheads and dates (so pages file and display correctly), headlines, the local news and personal columns, obituaries, marriage and birth notices, and legal notices that list names. If your OCR files carry word confidence values, use them to point correctors at doubtful words in those areas, after checking on a sample that low confidence really does track errors.

Systematic errors

Some OCR errors repeat across a whole run. Holley gives two examples from Australian newspapers: “tlie” for “the” in about 1 of 8 articles, and “30,000 articles with ‘Sydnoy’ instead of ‘Sydney’.” In a local paper, the town’s own name, the main street and the leading families’ names are candidates for the same treatment. Find the common misreadings of your most important names, check a sample of each in context, and correct them in bulk where they are unambiguous.

Test any automatic fix before applying it widely. Holley reports that adding a gazetteer of Australian place names as a secondary dictionary produced the worst results of the three settings tested, an outcome the program could not explain. Changes that look sensible can make text worse.

Volunteers

Holley’s program also opened its text to public correction. In the first twelve weeks after a quiet launch in July 2008, about 1,200 people (868 registered and about 390 unregistered) corrected 700,000 lines of text in 50,000 articles. A local paper has a smaller audience, but a committed one: family historians and local history volunteers often know exactly which names they want to fix. Planning that kind of work, and deciding how to record who corrected what, is the subject of Volunteers, AI or both?.

Measuring your own newspaper run

Measure word accuracy and name accuracy on a random sample of your own pages, and check reading order page by page. Those three numbers tell you more than any accuracy figure a vendor quotes, and they let you compare sources, engines and correction plans fairly.

A practical method, adapted from the British Library study:

  1. Draw pages at random across the whole run, with some from each decade, and include front pages and inside pages.
  2. Choose two zones per page, one in ordinary body text and one in a names-heavy area such as local items or notices. Choose them before looking at the OCR, and do not pick only the clearest areas, or you will measure a best case.
  3. Key each zone exactly, about 100 words, and have a second person check it.
  4. Compare with the OCR for word accuracy, for accuracy on capitalized words and for numbers. The OCR accuracy calculator handles character and word error rates from your counts.
  5. Check reading order on every sampled page: does the text of one column run into the next column in the right place, or across columns? Count pages with collapsed columns as a separate defect.
  6. Report by decade and by source, not as one number for the run.

How many pages to sample depends on how precise you need the answer to be. For page-level defects such as column collapse, the QC sample size calculator gives the number of pages to inspect for a chosen confidence and margin.

What counts as good enough

There is no standard threshold, but the two 2009 studies give useful reference points. Holley records the rough consensus of that time: 98 to 99 percent counted as good newspaper OCR, 90 to 98 percent as average and below 90 percent as poor, with no agreement on whether those were characters or words. Tanner and colleagues suggest a search-based test instead: above about 80 percent word accuracy, fuzzy search can usually fill the gaps because important words repeat; below it, search “will drop off steeply.”

For a local paper, set the target by use. If the main use is finding names for family history, measure accuracy on names and aim to raise it, through better capture or targeted correction, until test searches for known names in known issues succeed. If researchers will mine the text for words and phrases, word accuracy and reading order matter most.

Example (fictional). The Aster County Historical Society holds the Wren Gulch Courier from 1891 to 1912, about 4,576 pages. A project in the 2000s scanned a worn positive service copy of the film to bitonal images, and its OCR finds few of the names families ask about. The society tests three options on the same random zones:

  • Re-running OCR on the existing bitonal images: word accuracy rises from 72 to 76 percent, capitalized words from 55 to 58 percent. Small gains, because the images are the limit.
  • Rescanning the camera negative, found in the county’s storage, in 8-bit grayscale at 400 ppi relative to the page: 86 percent word accuracy, 72 percent on capitalized words.
  • Scanning the bound originals, which survive only for 1905 to 1912: 91 percent word accuracy, 80 percent on capitalized words.

The society rescans 1891 to 1904 from the negative and 1905 to 1912 from the originals, asks for ALTO files with column-ordered text and word coordinates, and pays for targeted correction of headlines, the local news column and death notices. To estimate how many pages still have collapsed columns, it inspects 135 random pages, the sample the QC calculator gives for 4,576 pages at 95 percent confidence, a 5 point margin and an expected 10 percent defect rate.

What to do next

Start by finding out what you have and how good it really is, then spend money where the measurement says it will help. A short sequence for a local newspaper run:

  • List every source for the run: originals (loose or bound), each generation of microfilm, existing page images and existing OCR files. Note gaps and condition.
  • Measure the current OCR on a random sample: word accuracy, accuracy on names and places, and the share of pages with collapsed columns.
  • If you have good grayscale images, test re-running OCR before anything else.
  • If the images are the limit, test rescanning from the best film or from originals on a few pages, at 300 to 400 ppi in grayscale relative to the page.
  • Ask for ALTO XML with word coordinates and column-ordered text, one file per page, with confidence values; decide separately whether article segmentation is worth paying for.
  • Plan targeted correction of headlines, names, places and notices, and decide who does it: staff, a vendor or volunteers.
  • Rebuild the search index and tell users what changed.

Our text recognition service measures and corrects newspaper OCR in this way, and our collection search shows hits highlighted on the page image from ALTO coordinates. A pilot box of a few reels or volumes, with pages from your best and worst decades, is a practical way to see the numbers for your own paper before deciding.

Working through this with a real collection?

If this article describes a decision you are facing, tell us about the collection. A pilot box answers most questions with your own material.

Plan a pilot box

Studio and contacts

24 South Weber Street, Suite 225
Colorado Springs, CO 80903
(719) 244-8450sales@dastok.com