New models read old handwriting better

The main thing in this release is new models for Cyrillic handwriting. They were trained on the hands where the previous ones stumbled most often.

What changed in reading

Pysar is the model that reads a case first. Diak-Litopysets reads the same page a second time. Its reading helps to find a surname where the first one came out wrong.

Both models were further trained on 10,000 lines of difficult 19th–20th-century hands:

• parish registers, revision lists, confession lists;
• court and police files, prosecutors' correspondence;
• a separate selection of 18th-century books.

Every line was checked against the scan before training.

We compared Pysar v28 with the previous one on the same control pages:

• 20th century: 12% fewer errors per line. These 13 cases were not used in training.
• 19th century: 6% fewer, and surnames are read more accurately.
• 18th century: 12–20% fewer, depending on the book.

Diak-Litopysets v10 gained most on 20th-century cases.

What the old models have read stays valid; there is no need to re-read it. The new models are worth using for cases not yet read, and for those where a surname was not found.

How to install

Update Nyshporka (nysh update or a new nyshporka-setup.exe over the old one) and download the models:

nysh models get

The old models do not disappear from the disk and keep reading until the new ones are there. If an assistant set up Nyshporka for you, it will offer to do this itself after the update.

A case from Supriaha — read by the newest models

Sometimes the same case is in Supriaha as several packages: different people read it at different times. Now Nyshporka takes the package read by the newest models. Before, it simply took the most recent one, even if older models had read it.

Hundreds of cases to Supriaha — overnight

When you share many read cases with Supriaha at once, Supriaha sometimes asks you to wait: the daily limit is used up, or there were too many uploads in an hour. Before, everything stopped there, and in the morning you had to start again. Now Supriaha says how long to wait, and Nyshporka waits exactly that long and shares the rest by itself:

nysh share push --ready --wait

FamilySearch films with several cases

One FamilySearch film often holds several cases filmed in a row. Between them stands a black card with the reference number. Nyshporka now finds these cards itself and splits the film into cases. You fill in the case numbers from one sheet that collects all the cards of the film, so you do not have to go through it frame by frame.

The opposite also happens: one case is split between two reels. It can now be put together from its parts into one case.

When the disk runs out of space

nysh disk shows what is in your research:

• what must be kept (what has been read, search, the exchange log);
• what can be removed without loss, because Nyshporka will build it again itself;
• from which cases the scans can be removed.

nysh disk clean --apply removes what can be rebuilt. The scans of a case that has been read and shared with Supriaha can be removed from the disk if you say where a copy is kept. The case stays in the list marked "removed", with the address of that copy.

Model download counter

We want to know how many people install and update the models. So before downloading a model, Nyshporka tells the Nyshporka website which model it is downloading. Together with the model name it sends:

• whether this is a first installation or an update;
• the operating system and the Python version.

Nothing else: no name, no address, none of your files. The IP address is not stored. The models themselves are still downloaded from GitHub. If the website is unavailable, the installation does not stop. The full list is in the privacy policy.

Fixed

• On a case card in Supriaha, the reading by Diak-Litopysets was labelled as Pysar's. Now each is labelled with its own model.