How We Connected 5,700 Languages with One Semantic Hub

When we set out to build a universal linguistic database, we faced a fundamental question: how do you connect 5,700 languages in a way that actually scales?

The naive approach would be to build translation pairs. English-Spanish. Spanish-French. French-Japanese. But that's O(n²) complexity. Millions of language pairs, each requiring its own dataset. It doesn't scale, and it fragments meaning across disconnected silos.

We took a different path. Instead of connecting words to words, we connected words to meaning.

The Problem with Word-Centric Design

Consider how traditional bilingual dictionaries work:

English "water" → Spanish "agua"
Spanish "agua" → French "eau"
French "eau" → Japanese "水"

This creates several problems:

• Translation chains drift. Going English → Spanish → French → Japanese accumulates errors. The meaning shifts with each hop.
• Coverage is sparse. Most language pairs don't have direct dictionaries. Good luck finding a Swahili-Basque dictionary.
• Context is lost. "Bank" means both a financial institution and a river's edge. Word-to-word mappings can't capture this.
• It doesn't scale. With 5,700 languages, you'd need 16 million language pairs. Nobody maintains that.

The Meaning-First Architecture

Our solution: a language-neutral semantic hub.

Instead of connecting words directly, every word in every language points to a shared meaning. That meaning exists independently of any language. It's a concept that humans across cultures can express.

English "water"  ──┐
Spanish "agua"   ──┼──► MEANING:247891 (liquid H₂O)
French "eau"     ──┤
Japanese "水"    ──┘

This is powered by PanLex, a dataset containing over 2 million cross-lingual meanings derived from 7,500+ language varieties. When we import a new dictionary (say, a Yoruba-English wordlist), we don't just store word pairs. We link each Yoruba word to the same meaning that English, Spanish, and 5,000 other languages already reference.

How the Schema Works

At the core of our database sits the meaning table:

CREATE TABLE meaning (
  id UUID PRIMARY KEY,
  panlex_meaning_id INTEGER UNIQUE NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now()
);

That's it. A meaning is just an identifier. It has no language, no text, no bias. It's pure semantics.

Every lexeme (word) connects to meaning through senses:

CREATE TABLE sense (
  id UUID PRIMARY KEY,
  lexeme_id UUID REFERENCES lexeme(id),
  meaning_id UUID REFERENCES meaning(id),
  definition TEXT,
  pos VARCHAR(20)
);

A lexeme can have multiple senses (polysemy), and each sense points to exactly one meaning. The word "bank" has two senses: one pointing to the financial meaning, another to the geographical meaning.

The Translation Graph Emerges

Here's where it gets powerful. Once every word links to meaning, translation becomes a graph traversal:

To translate English "freedom" to Swahili:
1. Find English lexeme "freedom"
2. Get its sense → meaning_id = 892341
3. Find all senses where meaning_id = 892341
4. Filter to Swahili language
5. Return: "uhuru"

No English-Swahili dictionary required. The connection exists because both languages independently expressed the same meaning.

This works for any language pair. Basque to Tagalog. Quechua to Finnish. If both languages have expressed a concept, we can connect them, even if no human has ever written that specific bilingual dictionary.

The Five-Strategy Linking Process

When we ingest a new data source, we need to link each word sense to the correct meaning. This is the hardest part of the pipeline. We use five strategies, in order of confidence:

1. Direct PanLex Match (Highest Confidence)

If the source data includes a PanLex meaning ID, we use it directly. This is the gold standard.

2. Translation Graph Match

We check if the word already exists in PanLex with the same language variety. If "agua" appears in PanLex as Spanish, we inherit its meaning.

3. Definition Similarity

We compare the new word's definition to existing meaning definitions using text embeddings. If "a colorless liquid essential for life" matches meaning 247891's definition closely, we link them.

4. Concepticon Alignment

For core vocabulary (body parts, numbers, kinship terms), we align to Concepticon, a curated set of 4,000 universal concepts. If we're importing a word for "mother," we can confidently link to concept 1347.

5. Defer

When we can't confidently link, we leave meaning_id null and queue for manual review. Bad links are worse than no links.

Why This Matters for Applications

This architecture unlocks use cases that word-pair dictionaries can't support:

Cross-lingual search. Search for a concept in any language, get results in all languages. Type "freedom" and see uhuru (Swahili), liberté (French), 自由 (Japanese).

Semantic similarity. Find words with related meanings across languages, not just translations. "Liberty" and "independence" share semantic space.

Language learning. Show learners how a concept is expressed differently across languages, with cultural nuance preserved.

LLM grounding. Give AI systems a factual semantic backbone. When an LLM says two words mean the same thing, we can verify it against 2 million human-curated meanings.

Endangered language documentation. When a linguist documents a dying language, each word immediately connects to the global meaning graph. No need to create a new English-X dictionary.

The Numbers

Our current database contains:

• 2 million+ unique meanings (PanLex semantic hub)
• 15 million lexemes across 5,700 languages
• 30 million senses linking words to meanings
• 100 million word forms (morphological variants)

A query that would require traversing millions of translation pairs instead becomes a simple join through the meaning table.

Trade-offs and Limitations

This approach isn't without challenges:

Meaning granularity. Some concepts don't map cleanly across cultures. The Japanese concept of "木漏れ日" (sunlight filtering through leaves) may not have a direct meaning equivalent. It might link to a broader "sunlight" meaning, losing nuance.

Linking accuracy. Our automated linking isn't perfect. We estimate 85-90% accuracy on the automated pass, with manual review improving high-value entries.

PanLex dependency. We're building on PanLex's meaning graph. If PanLex has gaps (and it does for some language pairs), so do we.

But the fundamental architecture is sound. Words are ephemeral and culturally bound. Meaning is universal. By building on meaning, we've created a system that can grow to cover every human language: past, present, and future.

What's Next

We're working on:

• Embedding-enhanced linking to improve automated sense-to-meaning accuracy
• Concepticon expansion for better coverage of abstract concepts
• Community contributions so native speakers can verify and improve links

The goal isn't just a bigger dictionary. It's a complete map of human linguistic expression: every word humanity has ever used to express every concept we've ever had.

And it all starts with one simple idea: don't connect words. Connect meaning.