Pith. sign in

REVIEW 2 cited by

Tokenisation is NP-Complete

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.15210 v1 pith:VCKOJDW7 submitted 2024-12-19 cs.DS cs.CLcs.FL

Philip Whittington , Gregor Bachmann , Tiago Pimentel This is my paper · ORCID
classification cs.DScs.CLcs.FL
keywords tokenisationbottom-upcompressingdatasetdefineddeltadirectdirectly
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most $\delta$ symbols by either finding a vocabulary directly (direct tokenisation), or selecting a sequence of merge operations (bottom-up tokenisation).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Causal Estimation of Tokenisation Bias

    cs.CL 2025-06 conditional novelty 7.0 of 10

    Using regression discontinuity, the paper shows that adding a subword to a tokenizer's vocabulary can raise the model's probability for that string by up to about 17 times in small models.

  2. MultimodalHugs: Enabling Sign Language Processing in Hugging Face

    cs.CL 2025-09 conditional novelty 5.0 of 10

    MultimodalHugs provides a standardized TSV-based dataset format, modular processors, and Hugging Face integration to enable reproducible sign language and multimodal translation experiments.

Pith tools