Meta tags:
description=Compute Optimal Tokenization: Scaling Laws for Data Compression in Language Models;
keywords=scaling laws tokenization compression BLT byte latent transformer;

Headings (most frequently used words):

optimal, compute, tokenization, bibtex, f1, data, to, model, size, f2, compression, rate, f3, beyond, english,

Text of the page (most frequently used words):
the (40), #compression (33), and (29), rate (21), optimal (20), compute (15), bytes (13), ratio (11), parameter (10), for (10), loss (10), data (9), across (8), budget (8), model (8), that (7), language (7), training (7), per (7), tokenization (5), parity (5), languages (5), from (5), english (5), with (5), each (5), budgets (5), scaling (5), this (4), between (4), isoflop (4), see (4), arxiv (3), same (3), version (3), different (3), finding (3), popular (3), bpe (3), tokenizers (3), these (3), rates (3), decreases (3), law (3), plot (3), can (3), over (3), points (3), models (3), tokenizer (3), tokens (3), fixed (3), size (3), page (2), which (2), project (2), under (2), limisiewicz (2), tomasz (2), pagnoni (2), artidoro (2), iyer (2), srini (2), lewis (2), mike (2), mehta (2), sachin (2), liu (2), alisa (2), margaret (2), ghosh (2), gargi (2), zettlemoyer (2), luke (2), 2605 (2), 01188 (2), observe (2), encoded (2), higher (2), given (2), russian (2), its (2), are (2), correlated (2), differs (2), against (2), findings (2), hold (2), lets (2), find (2), but (2), there (2), power (2), relationship (2), paper (2), interactive (2), above (2), you (2), rotate (2), hover (2), check (2), shaped (2), should (2), flops (2), triangles (2), how (2), what (2), parameters (2), constant (2), when (2), not (2), amount (2), most (2), varies (2), laws (2), meta (2), was, built, using, template, based, pages, website, licensed, nerfies, adversarial, superbpe, article, limisiewicz2026cotok, title, author, year, 2026, eprint, archiveprefix, primaryclass, url, https, org, abs, bibtex, relation, shoing, sparsly, benefit, byte, length, needed, express, content, versus, sentence, translated, will, more, than, vary, both, bowls, shifted, along, axis, meaning, depends, below, compare, optima, hindi, arabic, french, run, experiments, non, separately, whether, still, analysis, beyond, they, generalize, minimizes, increases, estimating, fit, details, profiles, increasing, best, results, trained, close, optimum, 5e18, 2e21, gather, changes, pinned, down, natural, follow, target, approximately, varying, therefore, generalizing, recipe, advise, matching, remains, nearly, values, bowl, surface, shows, every, lowest, achieved, roughly, 1e20, yields, now, let, dive, train, sweep, together, determine, corresponding, interesting, here, determines, raw, text, compressed, into, discrete, research, fixes, only, happens, control, token, study, impact, value, larger, scales, increase, proportionally, count, code, university, washington, fair,

Text of the page (random words):
compute optimal tokenization compute optimal tokenization scaling laws for data compression in language models tomasz limisiewicz 1 artidoro pagnoni 1 srini iyer 1 mike lewis 1 sachin mehta 1 alisa liu 2 margaret li 2 gargi ghosh 1 luke zettlemoyer 1 1 fair at meta 2 university of washington paper arxiv code meta ai tl dr we study the impact of data compression on scaling laws we find that f1 in compute optimal scaling bytes not tokens of data increase proportionally to parameter count f2 at each training budget there is an optimal compression rate and its value decreases at larger scales f3 the optimal compression rate varies across languages and differs from the compression rate of popular bpe tokenizers tokenization determines how raw text is compressed into discrete tokens for language model training most scaling law research fixes the tokenizer and varies only model size and data amount but what happens when we can control the tokenizer s compression rate bytes token we train language models at fixed compute budgets we sweep over compression rate and model size which together determine the amount of training data under a fixed budget and the corresponding bytes per parameter ratio the relationship between compression bytes per parameter ratio and loss is the most interesting here now let s dive in f1 optimal data to model size for a fixed compute budget 1e20 flops we plot loss against compression rate and bytes per parameter ratio this yields a 3d isoflop the bowl shaped isoflop surface shows that for every compression rate the lowest loss is achieved at roughly the same bytes per parameter ratio triangles interactive version of the plot above you can rotate it and hover over points to check loss compression and bytes per parameter values we observe that the optimal bytes per parameter ratio remains nearly constant across different compression rates finding 1 the optimal ratio between bytes of data and model parameters is approximately constant across varying compute budgets and compression rates therefore when generalizing a scaling recipe to a model with a different tokenizer we advise matching the ratio of training bytes not tokens to model parameters with the data to parameter ratio pinned down a natural follow up what compression rate should we target f2 optimal compression rate for each compute budget from 5e18 to 2e21 flops we gather the optimal points triangles from f1 this lets us see how loss changes with compression across compute budgets u shaped loss profiles across increasing compute budgets for best results models should be trained at a compression rate close to the optimum interactive version of the plot above you can rotate it and hover over points to check loss compression and compute budget we fit a power law to model the relationship between compression rate compute and loss for details see the paper power law estimating loss given compression rate and compute budget we see that the optimal compression rate decreases for higher compute budgets finding 2 at each training compute budget there is an optimal compression rate that minimizes loss the optimal compression rate decreases as the training budget increases these findings hold for english but do they generalize across languages f3 beyond english we run the same experiments on non english data training on each language separately to see whether these findings still hold the isoflop analysis as in f1 lets us find the optimal bytes per parameter ratio and compression rate for each language french arabic russian hindi across languages the isoflop bowls are shifted along the compression rate axis meaning that the optimal compression rate depends on the language below we compare these optima against parity and the compression rates of popular bpe tokenizers the optimal compression is correlated with parity the optimal compression rate differs from compression of popular bpe tokenizers we observe relation between optimal compression rate and parity shoing that sparsly encoded languages benefit from higher compression finding 3 the optimal bytes to parameter ratio and compression rate vary across different languages both are correlated with parity parity is the ratio of byte length needed to express the same content in a given language versus english e g a sentence translated to russian will be encoded by 2 more bytes than its english version bibtex article limisiewicz2026cotok title compute optimal tokenization author limisiewicz tomasz and pagnoni artidoro and iyer srini and lewis mike and mehta sachin and liu alisa and li margaret and ghosh gargi and zettlemoyer luke year 2026 eprint 2605 01188 archiveprefix arxiv primaryclass cs cl url https arxiv org abs 2605 01188 this page was built using the superbpe project page template which is based on the adversarial tokenization and nerfies project pages this website is licensed under cc by sa 4 0