R0027/2026-03-26/Q002/SRC03
Lundin et al. — Tokenization as the mediating mechanism for structural challenges
Source
| Field |
Value |
| Title |
The Token Tax: Systematic Bias in Multilingual Tokenization |
| Publisher |
arXiv |
| Author(s) |
Jessica M. Lundin et al. |
| Date |
2025-09 |
| URL |
https://arxiv.org/html/2509.05486v1 |
| Type |
Research paper |
Summary
| Dimension |
Rating |
| Reliability |
High |
| Relevance |
High |
| Bias: Missing data |
Some concerns |
| Bias: Measurement |
Low risk |
| Bias: Selective reporting |
Low risk |
| Bias: Randomization |
N/A — not an RCT |
| Bias: Protocol deviation |
N/A — not an RCT |
| Bias: COI/Funding |
Low risk |
Rationale
| Dimension |
Rationale |
| Reliability |
Quantitative analysis with clear causal model. Regression analysis across multiple models. |
| Relevance |
Directly explains how morphological complexity translates to tokenization cost and accuracy loss. |
| Bias flags |
Focuses on African languages; extrapolation to Japanese/Arabic/Finnish requires caution. |
| Evidence ID |
Summary |
| SRC03-E01 |
Morphologically complex languages pay a compounding tokenization tax: more tokens per word → higher cost → lower accuracy |