Nyshporka 0.24.0: new models for 18th–19th century manuscripts

Version 0.24.0 updates all three handwriting recognition models. In 18th-century parish registers, a surname lands in the correct entry six times more often than in the previous version. In Polish chancery cursive there are 38% fewer wrong characters, and 1.6 times as many words are read without a single error. The speed of processing a file on a single Tesla V100 card has hardly changed, and in cloud mode on two such cards it has risen by about 70%.

What has changed

• Pysar (Cyrillic, the main model) — version 19. Trained with additional hand-labelled lines from 18th-century parish registers.
• Diak-Litopysets (Cyrillic, the second model) — a new generation of Diak on the open PP-OCRv6 base.
• Skryba PPv3 (Latin script: Polish and Latin cursive) — a new model on the same base, trained on 28 hands.

New reading engine: kraken 7.1

The new models are built on a different architecture (PP-OCRv6), and it only works correctly on kraken 7.1. That makes this update bigger than usual: after nysh update you also need to update the reading engine itself (nysh htr install).

Kraken in Nyshporka is patched: six speed patches, verified on the new version, keep processing a file on a single Tesla V100 card at the previous level, and Latin script on the same card is read almost twice as fast. The splitting of pages into lines calculated by the old version is read by the new one without recalculation.

Recognition quality

The previous and new versions were compared on the same pages, on the same machine, with recognition redone from scratch.

“Above chance” is the share of entries where a surname from the hand-made register is found in the same entry of the recognised text, minus the matches a random page would give. The reference is the researcher's hand-made abstracts, compiled independently of the recognition.

In 19th-century books, quality has stayed at the previous level. The second model, Diak-Litopysets, confuses the surname being searched for with a similar surname of a neighbouring family less often: on the control pages there was one such error against four.

Speed and cost

The new Cyrillic models are larger, but the longest stage of processing a page — finding the lines — has not changed, so a whole file on a single Tesla V100 card is processed at practically the same speed (+2.5%). In cloud mode, the new version fits twice as many parallel processes on a card. For files with mixed Cyrillic and Latin script, where all three models work, the speed on the same machine with two Tesla V100 cards (16 GB) is about 5000 pages per hour.

Without a graphics card

Intel Core i7-9750H processor (6 cores), a dense page (~128 lines), seconds per page:

The slowdown applies only to the processor: on a Tesla V100 card, Latin script, on the contrary, is read almost twice as fast. A large file is quicker to read in the cloud (nysh cloud go).

What the models were trained on

• Hand-labelled lines from 18th-century parish registers (ЦДІАК, fond 224, and other files).
• For Latin script — 28 hands: notarial books of ДАВіО, fond 792; castle court books of the Kyiv Voivodeship (ЦДІАК, fond 2); minutes of the Podillia Assembly of the Nobility (ДАХмО, fond 230); soul lists of the Chechelnyk deanery (ДАК, fond 312); registers of a Roman Catholic church (fond 685); and an open corpus of 19th-century Dutch notarial deeds.
• The base of the new models is PP-OCRv6 medium (Benjamin Kiessling, Inria; Zenodo, DOI 10.5281/zenodo.21788410), licence Apache-2.0.

Limitations

• In parish registers from the second half of the 19th century, Pysar v19 finds surnames about 1 percentage point less often than the previous version.
• Some tests are small (143 lines of Latin script; one page of the 1802 minutes), so differences of 1–2 points on them are not significant. The main changes (six times better for 18th-century surnames, 38% fewer errors in Latin script) far exceed this margin of error.
• For small files (up to 100 pages), the cloud speed on a machine with two Tesla V100 cards is lower, about 2000 pages per hour, because of the start-up time for each file.

Other changes in this version

Less disk space

The torch library is no longer installed twice — next to the app and in the reading engine. Now it is only in the reading engine; on Linux this frees up several gigabytes.

Cloud

• A batch of files is prepared in minutes. To choose a machine, Nyshporka estimates how many lines there are on a page of a file that has not been read yet. Now, for a batch, this estimate is taken from files of the same inventory that have already been read — 0.3 s instead of ~85 s per file. A batch of 25 parish registers from one inventory used to spend 35 minutes of its 50-minute preparation on this estimate.
• Ready segmentation is not calculated a second time. Finishing a file, a retry after an interruption or a new model in the same folder all take the segmentation from the previous run.
• You can see which files did not go. The batch output prints “N of M files will go” and the reason for each file that dropped out.

Downloads

• An interrupted download resumes from where it stopped instead of starting from scratch.
• When the request rate is limited (429), Nyshporka waits exactly as long as the server asks.
• A file from Wikimedia Commons is checked against its SHA-1 checksum.

Supriaha

• If a file is in Supriaha as several packages, Nyshporka takes the one read by the newest production models; if there is none, by the previous ones; among equals — the most recent.
• New genres for catalogue entries: complete and incomplete parish register, marriage inquiries, reference publications (address calendars, commemorative books), court files, estate files (nobility, genealogy books), visitations, estate inventories, censuses, service records.
• A single failed connection to the storage is no longer treated as a block: Nyshporka tries again.
• If a folder contains only part of a file's scans, packaging asks for the number of scans in the whole file. Previously, a catalogue entry could show coverage above 100%.

Questions and answers

Why release new models if the old text is usable?

The old text is usable, there is nothing wrong with it. New models come out because more accurate text makes searching cheaper and more accurate.

An assistant searching for a family in what has been read has to look through the candidates and discard false matches. The more errors in the text, the more context it has to sort through and discard. A more accurate model gives fewer false candidates: the assistant finds the right place sooner, spends fewer tokens and misses something less often.

We miss little even now, but each new model reduces that share. New models will keep coming out.

Are files and contributions read by the old models still valuable?

Yes. Search works on text from any of the models: the daily family search reports that we review produce finds on files read by previous versions too. Every find is checked by eye on the scan, so the accuracy of the model affects how much work it takes to reach a find, not whether you can trust it.

That is why your contributions to Supriaha are needed now, with the models we have. We keep reading files with the existing models; when newer ones come out, the part where it is worth the effort can be re-read.

The work already done is not lost when re-reading either: splitting pages into lines — the longest part of reading — is taken from the cache, and the new models read the same lines.

Do I now have to re-read all the files I have read?

No. Everything you have read stays in search and does not go out of date.

Re-reading is only needed where you want the most from search for less cost: 18th-century parish registers, Latin and mixed files, and files where the family search found nothing. In 19th-century books the quality is almost the same (55.2% against 54.9% of surnames found), so there is no need to re-read them for the sake of the new models.

Re-reading is cheaper than the first reading: the ready splitting of pages into lines is taken from the cache. On a single Tesla V100 card it takes most of a page's time (5.5 s out of 8.9 s), so re-reading is roughly two to three times faster. Simply run nysh read for the file again — the new models will be picked up on their own; in the cloud, nysh cloud go takes the ready splitting automatically.

Can I stay on the old models?

Yes. The old weights stay on disk: until the new ones are downloaded (nysh models get), Nyshporka reads with them, and as soon as the new ones appear it takes the newest production models. Any file can be read with an explicitly named model: nysh read <справа> --model pysar_cyr_v17.pt.

After updating Nyshporka itself, you need to update the reading engine (nysh htr install): without it, reading will not start. The old models work on the updated engine as before.

Which text does Nyshporka take from Supriaha?

If a file is in Supriaha as several packages, Nyshporka takes the one read by the newest production models; if there is none, by the previous ones; among equals — the most recent. The same applies to a single file and to a whole fond or inventory. Next to each file in the catalogue you can see which models read it. An accepted file comes with all its readings (voices), and search runs across all of them.

If a file has already been taken and a package of the same file read by newer models appears in the catalogue, you can replace what you took with the command nysh share pull … --force.

For those updating from versions before 0.22: moving the file records

If you are updating from a version older than 0.22, the update includes moving the file records to new keys. This is the only step that changes your data, so it is worth doing carefully.

The key of every file now always contains the inventory: REPO/фонд/опис/справа, for example DAHMO/315/1/8433. If the inventory is unknown, _ stands in its place. Previously the inventory was part of the key only for some fonds, and two books with the same number in different inventories (for example, ДАДнО 193-1-213 and 193-3-213) had one key.

What is moved: the page store, run links, file and line verdicts, the search log, Supriaha catalogue entries, the file queue and proofreading profiles.

Steps

1. Stop other sessions and background processes that write to the records. While they are running, the move will not start.
2. Look at the plan: nysh cases rekey. It shows what will be renamed, what will be merged and what cannot be moved.
3. Check the merges. If one file had verdicts under two old keys, the later one becomes current, and the other is kept alongside as replaced.
4. Run the move: nysh cases rekey --apply. Before any changes, an archive is created in data/cases/rekey/<час>/; a failure midway returns the workspace to its previous state.
5. For files with an unknown inventory (_), add the inventory to the passport (nysh case <тека> --shifra "…") and repeat nysh cases rekey --apply.

Good to know

• Before the move, reading and search work, but writing to the records is refused with a hint about which command to run.
• Rollback is nysh cases rekey --rollback. If the records have already changed after the move, the rollback is refused with a list of the changed files; use --rollback --force only deliberately.
• Old keys in notes and messages still lead to their file through the relocation map (data/cases/key_moves.json).

How to update

After nysh update, two commands are needed: nysh htr install (updates the reading engine in place) and nysh models get (the new weights, about 225 MB). Nyshporka itself reads with the newest production models if they are on disk; if they have not been downloaded yet, with the previous ones. Nothing needs switching by hand.

How the comparison was made

Both versions processed the same 271 pages on one rented machine with a Tesla V100 card, with no other processes running; each page was recognised from scratch. Cloud speed was measured separately on a machine with two V100 cards on a 606-page file. Pages used to train the models were excluded from the comparison. Processor measurements were made on an Intel Core i7-9750H, on three dense pages of ДАК 312-1-27.