The Ukrainian lemmatizer

Why it exists

Every word-list join depends on a lemmatizer that agrees with the list about what a headword is. The frequency list, the validated CEFR overlay and the profiler all use the VESUM convention (the infinitive for verbs, the nominative singular for nouns, the apostrophe written one way). A lemmatizer in another convention turns every token into an unknown. This endpoint is that convention, hosted, so the profiler and any third-party tool can call one thing and get the list’s own lemmas back.

Under the hood it is LanguageTool’s Ukrainian analysis pipeline, the core that nlp_uk’s TagText wraps: the tokenizer, the VESUM tagger and the rule-based disambiguator, over a pinned dictionary (ua.net.nlp:morfologik-ukrainian-lt:6.8.0). The dictionary is CC BY-NC-SA and never leaves the server; the endpoint returns analyses of your own text and nothing else.

Try it

Send a list of tokens (each analyzed on its own, what a word-list join wants):

curl -s https://www.jakelawrence.xyz/api/lemmatize \
  -H 'Content-Type: application/json' \
  -d '{"tokens": ["ім’я", "кав’ярня", "під’їзд", "м’ясо", "п’ять"]}'

Or a text (tokenized and disambiguated in context, with character offsets):

curl -s https://www.jakelawrence.xyz/api/lemmatize \
  -H 'Content-Type: application/json' \
  -d '{"text": "Вона повідомила про це вчора."}'

A response looks like this (one token shown):

{
  "tokens": [
    {
      "token": "п'ять",
      "lemma": "п’ять",
      "pos": "numr",
      "ud": "NUM",
      "tags": "numr:p:v_naz",
      "known": true,
      "ambiguous": false,
      "alternatives": [
        {
          "lemma": "п’ять",
          "pos": "numr",
          "ud": "NUM",
          "tags": "numr:p:v_zna"
        }
      ]
    }
  ],
  "meta": {
    "service": "lemmatizer 0.1.0",
    "tagger": "LanguageTool Ukrainian 6.8",
    "vesum": "6.8.0",
    "mode": "tokens",
    "chars": 5,
    "tokens": 1,
    "ms": 3,
    "cache": "miss",
    "hash": "7fb62d674239a1c0"
  }
}

Per token: lemma (apostrophe written U+2019, the list’s glyph), the coarse VESUM pos, a best-effort ud tag, the full tags string, known, ambiguous (more than one distinct lemma survived disambiguation) and every other surviving reading under alternatives. The meta block carries the tagger and VESUM versions on every answer, so a result is never separated from the dictionary that produced it. The full contract is the OpenAPI document.

Limits, and what happens at them

Does it agree with the list?

The check that matters. A deterministic sample of the list (ranks 1 to 200 in full, then every 50th rank: 436 lemmas) was sent to the service, and the answer compared with the list’s own lemma at that rank, apostrophe and stress folded. A match requires the dictionary to know the word; an unknown token echoed back is counted as a miss, not a hit.

The UD column is informational: ud is a best-effort map from VESUM tags, and the list’s own tags came from a different tagger over running text, so particles, prepositions and adverbs disagree where the two conventions draw their lines. The lemma column is the contract.

The one miss: Упл (rank 11,950, PROPN; unknown to the dictionary).

The five apostrophe lemmas the work order named, which are the words a glyph mismatch breaks first: ім'я → ім’я (rank 311), кав'ярня → кав’ярня (rank 9,089), під'їзд → під’їзд (rank 5,424), м'ясо → м’ясо (rank 2,851), п'ять → п’ять (rank 480).

Timing: 5,000 characters of Wikipedia sentences (710 tokens) in 102 ms wall time, 92 ms inside the service, median of 5 runs against a local instance; the budget was 2,000 ms. Checked 2026-09-14. The artifact behind this section is src/data/ukrainian-frequency/lemmatizer-roundtrip.json; CI recomputes the headline from its rows and fails when a pin changes without a re-run.

What it does not do

It does not ship the dictionary or any wordform table to the browser: VESUM is CC BY-NC-SA and stays server-side. It does not apply nlp_uk’s statistical disambiguation or semantic tags; the rule-based disambiguator is what the round-trip needed, and meta.pipeline says exactly what ran. It does not keep your text. And it does not guess: an unknown word comes back marked known: false with the token as its lemma, and a token that splits into several pieces is reported as such rather than collapsed.

Need something like this built?

I design and ship AI tools, full-stack apps, and data pipelines — end to end, to production. Tell me the problem in a sentence; I'll give you an honest read on fit within a day.