LFM — Language Faculty Model

A framework for encoding structured data as natural language.

LFM encodes arbitrary continuous representations — agent embeddings, protein features, mathematical structures, any high-dimensional vector — as linguistically structured, pronounceable IPA (International Phonetic Alphabet) utterances. The output is compositional, variable-length, and phonotactically valid: not a cipher or a learned code, but output that shares the structural properties of natural language because it is generated through a decoder pretrained on multiple typologically diverse human languages. This makes it interpretable by any multilingual LLM.

Contents

🌐 Vision — why encode structured data as natural language
🔬 The Problem — limitations of existing encoding approaches
⚙️ How It Works — three-step pipeline overview
🗣️ The Phrase Decoder — architecture, pretraining, sample outputs
🌳 Expression Generation — diffusion-based z-sequence generation through the decoder
🎯 Emergent Communication Games — Expression and dialogue game validation
📊 Structural Analysis — latent space typology and compositionality metrics
💾 Dataset Generation — HDF5 pipeline with constituency augmentation
📈 Visualization CLI — t-SNE, clustering, attention, Zipf, and more
🚀 Quick Start — install, pretrain, run
🧬 Design and Rationale — why IPA, not pretrained LM output; implementation principles
📋 Status — current capabilities and roadmap
📖 Further Reading — related docs and background

Vision

Neural systems — agents, encoders, scientific models — produce continuous representations that capture structure no human formalism was designed to express. A protein encoder's embedding of a novel fold, an RL agent's observation of a dynamical system, a GNN's representation of a molecular graph — these are empirically grounded, information-rich, and completely opaque.

Existing interpretability methods ask "does the model represent X?" — probing classifiers, mechanistic interpretability, concept bottleneck models — they all impose human ontology on the model and test whether it conforms. LFM inverts this. Instead of probing the model with human hypotheses, the model expresses its perception in its own emergent language, and a human discovers the model's categories through an LLM interpreter. The model's ontology is the output, not the input to the analysis.

LFM gives any neural system a voice. It encodes any continuous representation as a new natural language — not English, not mathematics, but an emergent language whose structure is inherited from a pretrained multilingual decoder. The language is compositional, variable-length, and phonotactically regular. A multilingual LLM learns this language through self-supervised pretraining (no paired translations), then interprets it into English — not as literal translation, but as the LLM's own understanding of what the source system perceives. The interpretation reveals the source system's learned ontology: the categories, distinctions, and relationships it has discovered.

This enables a fundamentally new kind of feedback. Current mechanisms (RLHF, DPO, fine-tuning) correct model behavior — "this answer is wrong, here's the right one" — without understanding why the model erred. If a model can express how it perceives an input, you can diagnose the perceptual error: "you're not wrong about the answer, you're wrong about what you're looking at." The feedback targets the root cause, not the symptom.

The key mechanism is a frozen linguistic bottleneck: a VAE decoder pretrained on typologically diverse languages, then frozen. Downstream systems don't learn a communication protocol from scratch — they learn to project their representations into the decoder's latent space, and the decoder's structure constrains the output to be linguistically well-formed. This is analogous to Universal Grammar in the Chomskyan sense: a fixed structural prior that constrains the space of possible languages, where only the mapping from meaning to form is learned. This avoids the known failure modes of end-to-end emergent communication (anti-Zipfian codes, degenerate protocols, non-compositional signals).

The pipeline is concrete: an input embedding is projected into a VAE latent space, decoded through a frozen multilingual transformer into IPA tokens, and the resulting utterance carries enough structure for a receiver to identify what was encoded (98.3% accuracy at 100% hard negatives in a 16-way dialogue game). An LLM learns the emergent language and interprets it into English. At every step, the information is empirically grounded and the fidelity is measurable.

The Problem

Any system that needs to express continuous representations as structured, interpretable sequences faces a choice:

Natural language imposes human ontology and semantic bias
Latent vector passing lacks structure and interpretability
Learned discrete codes (VQ-VAE, emergent protocols) produce tokens that are statistically learnable but carry no linguistic structure — an LLM can predict "token 47 follows token 12" but has no cross-lingual bridge to English interpretation
Symbolic systems are rigid and not adaptive
JSON / structured serialization is not compositional and has no linguistic inductive bias

The critical distinction is between encoding and language. A VQ-VAE can tokenize an embedding into 120 integers. An LLM can learn to predict those integers. But the integers are arbitrary — relabeling them as synthetic syllables does not create linguistic structure. The structure isn't in the surface form; it's in the generative process that produces it. A frozen decoder pretrained on real languages produces output with phonotactic constraints, morphological compositionality, Zipfian token distribution, and variable-length encoding — properties that emerge from the decoder's learned knowledge of how language works, not from label assignment.

LFM provides a frozen linguistic bottleneck that constrains any input to be expressed as compositional, variable-length, phonotactically valid natural language — without inheriting the semantics of any human language. The structure comes from the decoder; the meaning comes from whatever is being encoded. The result is interpretable not because it maps to known human concepts, but because it is structured the way language is structured — and any sufficiently capable multilingual LLM can engage with it.

Translation, not alignment

The emergent language that LFM produces is not human language — but it is language-like. It has morphology, phonotactic structure, and compositional regularity. This is by design: the structural inductive biases come from the frozen decoder, which was pretrained on typologically diverse human languages.

Because the output is in IPA — a universal phonetic representation — and because the decoder was pretrained on 12 typologically diverse human languages, the emergent language inherits the universal structural patterns that all human languages share: morphological regularity, phonotactic constraints, compositional hierarchy. A multilingual foundation LLM already has deep representations of these patterns from training on 100+ human languages. The alien language is language 101 — the LLM's existing cross-lingual transfer machinery should recognize its structure without supervised translation pairs.

The training path is self-supervised: continue training a multilingual foundation model on a corpus of alien IPA text. No paired translations. The LLM learns the distributional structure of the alien language the same way it learned every other language — from corpus statistics. Because the alien language's compositional patterns activate the same internal features that human languages activate (they were forged by the same phrase decoder), zero-shot or few-shot translation emerges naturally. Prompt with alien IPA, ask for English, and the LLM's cross-lingual transfer produces a translation — the same way multilingual LLMs translate between language pairs they were never explicitly trained on.

Crucially, this is interpretation, not translation. The LLM's English output is not a literal mapping from alien words to English words — it is the LLM's own understanding of the alien language's meaning, expressed through its existing conceptual vocabulary. The source system's ontology stays intact in the Neuroglot; the LLM interprets it through its own lens. When the source encoder perceives something for which no direct human name exists — a novel distinction in a dynamical system, an unnamed structural motif in a protein — the LLM's interpretation becomes a form of conceptual midwifery: reaching for the closest human concepts, producing novel combinations, hedging and elaborating the way humans do when encountering something they don't have a word for. Follow-up questions refine the interpretation through dialogue.

And because the LLM learns the alien language as a language — not as an encoding to decode — the interpretation is bidirectional. The same model that interprets alien IPA → English can generate alien IPA from English. This closes a full communication loop: the system speaks, a human reads the interpretation, responds in English, the LLM translates back into the emergent language, and the system receives it. Bidirectional dialogue mediated by a shared alien language, with the LLM as interpreter — no paired training data required.

Translation is inherently lossy — but language compensates for that. Unlike a fixed-size code, language can expound: use more words, more phrases, more compositional structure to progressively encapsulate meaning that a single utterance would lose. This is exactly what LFM's expression system exploits. The DiffusionZGenerator produces multiple phrases per expression, and the dialogue game extends this to multi-turn paragraphs — 4 sentences per embedding, each elaborating from a different angle — trading efficiency for fidelity the same way natural language does.

How It Works

LFM uses a generative linguistic bottleneck: a pretrained VAE decoder that produces linguistically structured IPA output from a latent space.

Step 1: Pretrain the VAE decoder

A multilingual VAE is trained on IPA-transcribed phrase-level constituents from typologically diverse languages (Leipzig Corpora Collection). The v5-leaf training set covers 12 languages spanning major morphological types:

Typology	Languages
Fusional	English, German, Portuguese, Russian
Agglutinative	Turkish, Finnish, Hungarian, Korean
Isolating/Tonal	Vietnamese, Indonesian
Introflexive	Arabic
Split-ergative	Hindi

The training corpus consists of 4M leaf-level phrase constituents (NP, VP, PP, ADJP, ADVP, S, SBAR) extracted via unified UD dependency-to-constituency conversion with Stanza dependency parsers. Only leaf constituents are kept -- the atomic building blocks of syntax. Vietnamese IPA is recovered via word-alignment of constituents against parent sentence IPA (fallback for languages where epitran's word-level IPA doesn't align with constituent spans). v5-leaf trains on standalone leaf constituent IPA sequences (no constituent_context -- each sample IS a short phrase), so the decoder learns phrase-level EOS naturally. The expression game then composes these atomic phrases into multi-phrase utterances. Text is converted to IPA via epitran (non-English) and the CMU Pronouncing Dictionary (English), tokenized with syllable-aligned sentencepiece BPE (max_seq_len=27, auto-scaled from dataset).

The decoder uses a PhraseDecoder with architectural biases for natural language:

Rotary Positional Embeddings (RoPE): translation-invariant pattern learning — a morpheme works the same way regardless of position
Multi-scale attention heads: window sizes of 3 (phonotactic), 7 (morpheme), 15 (word), and full (clause) — a multi-resolution linguistic filter bank
Weight-shared layers: 2 unique layers applied 4 times = literal recursion, mirroring syntactic Merge

Training uses cosine LR decay, DIP-VAE covariance regularization (off-diagonal penalty to encourage dimension independence), and a variance floor to prevent latent collapse.

After pretraining, the decoder is frozen. It becomes a fixed linguistic bottleneck.

Step 2: Expression game — learning to express

A DiffusionZGenerator (flow-matching denoiser, T=4 reverse steps) learns to produce K z-vectors from an input embedding. All K positions see each other via self-attention during refinement, enabling co-adaptation. Each z-vector is decoded through the frozen decoder as one phrase until EOS, with the KV cache persisting across phrase boundaries for coarticulation. The result is a multi-phrase expression — a "sentence" in the emergent language.

Training uses a referential game: the expression must discriminate the input embedding from 15 distractors drawn from the same semantic cluster (hard negatives). The loss is cross-entropy on the hidden states, scored through a learned message encoder and bilinear receiver. A z-diversity loss (cosine similarity penalty between z-vectors) prevents phrase collapse.

Only the z-generator, message encoder, and receiver learn. The decoder's linguistic structure is preserved.

Step 3: Internal dialogue — paragraph generation

Single expressions can be sufficient for LLM pretraining depending on what is being encoded. Multi-sentence documents provide richer training signal: the LLM can learn discourse patterns, progressive elaboration, and inter-sentence dependencies in addition to vocabulary and phrase structure. The dialogue game generates 4-sentence paragraphs per embedding — a self-dialogue where each turn uses a distinct learned position embedding (simplex-initialized for maximum distinguishability), elaborating on the same input from a different angle:

Each sentence is a separate expression (3 phrases decoded in one AR pass)
A ContextTransformer conditions each sentence on the hidden states of previous sentences — progressive elaboration, not repetition
Detached progressive scoring: each turn's z-generator gradient comes only from its own scoring step, preventing earlier turns from being retroactively optimized by later losses
Progressive topology matching scores the accumulated paragraph at each sentence: the dialogue so far must match the embedding cosine similarity structure (KL divergence)
Cross-entropy as primary loss drives argmax accuracy; KL shapes the full topology
Cross-turn z diversity: pairwise cosine similarity penalty on mean z-vectors pushes different sentences into different z-space regions

The result: 4 complementary sentences per embedding, each capturing a different facet of the input's semantic position. The stochastic z-gen ensures surface form diversity; the context conditioning ensures semantic complementarity. Achieved 98.3% best accuracy at 100% hard negatives (16-way, 2048 clusters).

Step 4: Corpus generation and LLM pretraining

The trained dialogue game generates a corpus of 900,000 natural paragraphs. Each document is a 4-sentence self-dialogue, isomorphically romanized to ASCII (no IPA Unicode, no hyphens, no brackets — the text looks like a natural Latin-script language):

Kexxrihng vjiska tak na zaponte prjixka eyudrtj verscxons ehvjaaojeliix.
Axrihng ttuxkxuxsihz krxieyutuxd txok aaptaashuung a lanuxrd provolmaxaxiscx
b kdhowz tuzd disuxnt dhux kawrxtihng uxvuxn. Djuxllajnixe podnzonarit iz
manjila tcjestjirexj bisa bexri vordatj skjuxjas ego tcxuxlixx domexjihada
trsh. Uhnihngxr alax horxt txownihngs nax uxkr iht vokugehdaxm bahuxpix s
infs tuxa sangfertji tcjuxjns s prxjuxnsex.

Distributional analysis of the corpus reveals recoverable grammatical structure — distinct word categories, syntactic transition patterns, and Zipfian vocabulary — all inherited from the frozen decoder's multilingual pretraining:

Left: near-perfect Zipf's law (s=1.04) over 2.8M word types — the strongest indicator of linguistic structure. Right: category-to-category transition probabilities reveal structured syntax with function/content word distinctions. Full analysis in docs/corpus-analysis.md.

Corpus statistics: 900,000 documents, 3.6M sentences, 58.7M tokens, 99.998% unique. The isomorphic romanization preserves all phonemic contrasts from the IPA while eliminating Unicode characters that would trigger the LLM's "phonetic notation" prior.

A multilingual foundation LLM (Qwen 2.5 0.5B) is trained on this corpus interleaved with English paragraphs (30% English, 70% Neuroglot) via self-supervised next-token prediction. The interleaving prevents catastrophic forgetting of English while the model learns Neuroglot's distributional structure. EMA (exponential moving average) of weights provides additional forgetting mitigation.

The Phrase Decoder

The core of LFM is a pretrained multilingual VAE decoder that produces linguistically structured IPA from a latent vector. After pretraining, it is frozen and becomes a fixed linguistic bottleneck for downstream use.

Decoder architecture

z (256-dim latent vector, 8-token memory)
  → latent_to_decoder projection
  → frozen PhraseDecoder
      |-- RoPE (translation-invariant positions)
      |-- Multi-scale attention heads (3/7/15/full token windows)
      +-- Weight-shared layers (2 unique × 4 applications = recursion)
  → variable-length IPA tokens (max_seq_len=109)

The PhraseDecoder has architectural biases for natural language:

Rotary Positional Embeddings (RoPE): a morpheme works the same way regardless of position
Multi-scale attention heads: window sizes of 3 (phonotactic), 7 (morpheme), 15 (word), and full (clause) — a multi-resolution linguistic filter bank
Weight-shared layers: 2 unique layers applied 4 times = literal recursion, mirroring syntactic Merge

Pretraining

The decoder is trained on IPA-transcribed text from typologically diverse languages (Leipzig Corpora Collection). Training uses cosine LR decay (per-epoch), DIP-VAE covariance regularization, z-variance regularization, and full resume support.

v7 (full constituency, current): 11.6M full constituency phrase constituents from 12 typologically diverse languages (eng, deu, por, rus, tur, fin, hun, kor, vie, ind, ara, hin), extracted via dep-to-constituency parsing. Syllable-aligned BPE tokenization (8000 vocab), max_seq_len=109 (auto-scaled from dataset), 8-token z memory (multi-token cross-attention), latent_dim=256, decoder_hidden_dim=512, 8-head multi-scale attention [3,3,7,7,15,15,full,full], weight-shared layers (2 unique x 4). Full constituency trees produce richer, longer phrases with syntactic depth — the decoder learns clause-level structure, not just atomic phrase boundaries.

v5-leaf-27 (leaf-level phrases): 4M leaf-level IPA phrase constituents, max_seq_len=27. Faster iteration and shorter expressions, but limited to atomic phrases (NP, VP, PP, etc.) without syntactic embedding.

Pretraining results (v7)

11.6M full constituency IPA phrases from 12 languages, syllable-aligned BPE, max_seq_len=109, 8-token z memory (latent_dim=256, lr=0.0005, 3 epochs):

Metric	Value
Val CE (short, <20 BPE)	0.001
Val CE (medium, 20-50 BPE)	0.005
Val CE (long, >50 BPE)	0.038
Val CE (overall)	0.0082
Train accuracy	99.0%
Variable-length output	mean 11.4 words, range 6-18 words
Surface diversity	100% unique (64/64 random z)
TTR	0.992
EOS rate	1.00 (always produces well-formed EOS)

Left: hierarchical clustering recovers linguistically sensible language groupings. Right: t-SNE by language — 12 distinct clusters with typologically sensible overlap zones. Full analysis in docs/structural-analysis.md.

Cross-typological Interpolation

The latent space supports smooth interpolation between typologically distant languages. Decoded IPA transitions smoothly through intermediate typological regions:

Perturbation

Adding noise scaled to the encoder's z distribution to the same z vector. sigma=0.5 preserves language-local phonotactics with content shifts, sigma=1.0 shows mixed typology, and sigma=2.0 crosses entirely into different typological families — the decoder navigates continuously through the latent manifold.

Expression Generation

LFM includes a learnable expression system for multi-phrase communication through the linguistic bottleneck. Instead of mapping one embedding to one flat utterance, an agent produces a variable number of phrases via a DiffusionZGenerator — a small transformer denoiser that refines all K z positions simultaneously using flow matching (T=4 steps), conditioned on the input embedding via cross-attention. Each phrase's z vector is decoded through the frozen decoder, with the KV cache persisting across phrase boundaries for phonotactically coherent coarticulation.

    noise ~ N(z_mean, z_std)          (K positions, from pretrained distribution)
    ┌──────────────────────────────────────────────────────────────────┐
    │  Denoiser Transformer (T=4 flow-matching steps)                 │
    │    self-attention:  z positions co-adapt                        │
    │    cross-attention: conditioned on input embedding              │
    │    activity head:   per-position sigmoid → variable length      │
    │    z_t = (1-t)*z_0 + t*noise, model predicts clean z_0         │
    └──────────────────────────────────────────────────────────────────┘
    z₀, z₁, z₂ ← refined in parallel    activity: [1.0, 0.9, 0.1, ...]

    Decode:  [BOS ðʌ kwɪk braʊn | fɑks dʒʌmpt | oʊvɝ ðʌ leɪzi dɑɡ EOS]
                  memory=z₀          memory=z₁     memory=z₂
                  (continuous KV cache — no breaks)

Key design decisions:

Flow matching: Linear interpolation z_t = (1-t)*z_0 + t*noise; the denoiser predicts clean z_0 directly. Four reverse steps (t=1.0 → 0.75 → 0.5 → 0.25 → 0.0) are fully differentiable end-to-end
Parallel refinement: All K z positions are refined simultaneously through self-attention, enabling co-adaptation that sequential generation cannot achieve
Per-position activity: A sigmoid activity head determines which phrases are active (first phrase always on). Length regularization with entropy bonus encourages variable, Zipf-like expression lengths
Noise from pretrained distribution: Initial noise is scaled to the pretrained z mean/std, and the output head is initialized to match, ensuring the denoised z vectors land in the decoder's learned manifold
Two-phase backprop: Same as referential game — no_grad generation, then parallel decoder re-run with gradients through cross-attention

Components:

Module	Role
`DiffusionZGenerator` (`lfm.agents.diffusion`)	Transformer denoiser: flow-matching refinement of K z positions with per-position activity
`ExpressionGenerator` (`lfm.expression`)	Continuous z-switching decode through frozen decoder (KV cache across phrases)
`Expression` (`lfm.expression`)	Data structure: z vectors, decoded tokens, phrase boundaries, activity weights
`ExpressionEncoder` (`lfm.expression`)	Segment pooling + bottom-up Merge composition → fixed-size message vector

Plug-and-play integration — works with any agent that produces fixed-size embeddings:

from lfm.agents.diffusion import DiffusionZGenerator
from lfm.expression import ExpressionEncoder

diff_gen = DiffusionZGenerator(
    input_dim=384, latent_dim=256, max_phrases=8, num_steps=4,
    z_mean=pretrained_z_mean, z_std=pretrained_z_std,
)

z_seq, z_weights, num_phrases = diff_gen(agent_embedding)
# z_seq: (B, K, 256) denoised z vectors
# z_weights: (B, K) per-position activity (first always 1.0)
# num_phrases: (B,) soft phrase count

expr_enc = ExpressionEncoder(hidden_dim=512, output_dim=384)
message = expr_enc(expression)  # fixed-size message vector

No decoder retraining needed. The diffusion denoiser learns to place z vectors where the decoder's pretrained manifold produces discriminative, compositional surface forms.

See docs/expression-system.md for the full design document covering motivation, architecture details, continuous z-switching decode, integration guide, and downstream applications.

Sample outputs (v1)

Reconstruction (English — all content words recovered, minor word order shuffle):

orig: mækswɛl sɛd hi meɪd fɹɛndz fɔɹ laɪf ɑn ðʌ ʃoʊ wɪtʃ ɪnkludʌd ðʌ ʌðɝ ækts
dec:  mækswɛl sɛd hi meɪd fɔɹ fɹɛndz laɪf ɑn ðʌ ʃoʊ wɪtʃ ɪnklud ðʌ ʌðɝ ækts

Interpolation (English → Portuguese — smooth typological transition):

0.00: mækswɛl sɛd hi meɪd fɔɹ fɹɛndz laɪf ɑn ðʌ ʃoʊ wɪtʃ ɪnkludʌd ðʌ ʌðɝ ækts
0.50: ɛʃtowu fɛksʌz dɛliz sɛd ðæt hi meɪd ðʌ jɪɹ fɹʌm ðʌ swʌŋ fɹeɪnz laɪf aʊtmæs wɑz viʃɪŋ
1.00: ɛʃtowu mujto fɛliz dɛ ɾɛtomɐɾ os sows dɛpowis dɛ dowis ɐnos dɛʃsɛs mudɐnsɐs ɛ tɐntɐs

Perturbation (σ=0.5 — sentence frame holds, content words shift):

σ=0.0: mækswɛl sɛd hi meɪd fɔɹ fɹɛndz ʃoʊ ɑn ðʌ laɪf wɪtʃ ɪnkludʌd ðʌ ækts ʌðɝ
σ=0.5: mækswɛl hi meɪd sɛd laɪf fɹɛndz ɑn ðʌ menlis wɪtʃ fɔɹ ðʌ æstɪʃoʊz klɑpd
σ=1.0: mækskswɛnd vlad ɔranfilt͡ɕɛ r sɛd aɪ meɪd ðætrne carne ɔ fɔɹmɝ fɹɪtel buktɨ ʌðɝ hetwʌzʔau ...

Package structure

src/lfm/
  expression/           # Expression dataclass and legacy GRU z-sequence generator
    generator.py        # ExpressionGenerator (legacy GRU z-sequence + PonderNet)
    encoder.py          # ExpressionEncoder (phrase pooling + composition)
    expression.py       # Expression dataclass
  faculty/              # LanguageFaculty compositor
  generator/            # VAE generator, phrase decoder, pretraining
    layers.py           # PhraseDecoder (RoPE + multi-scale attention)
    multilingual_vae.py # MultilingualVAEGenerator
    pretrain.py         # Full pretraining pipeline
  data/                 # Corpus datasets, loaders, IPA conversion
    sanitize.py         # Configurable text sanitization
    dataset/            # HDF5 dataset generation + reader
    loaders/            # Leipzig loader, IPA converter
  cli/                  # CLI framework (lfm dataset, visualize, translate, publish)
  embeddings/           # LLM embedding games, sampler, prefetcher
  visualize/            # Visualization suite (t-SNE, clustering, attention, etc.)

Structural Analysis

Detailed visualization evidence for the model's structural properties — latent space organization, attention hierarchy, Zipf's law, smoothness, adaptive length, compositionality, cross-typological interpolation, and per-dimension latent sweeps — is presented in docs/structural-analysis.md, generated via the lfm visualize all and lfm explore dim-sweep CLI commands.

Key findings:

Adaptiveness: input<->output length r=1.000, z_norm<->output_unique r=-0.663
Zipfian output: corpus exponent 1.004, decoded 1.058 — near-perfect Zipf law
Variable-length output: mean 2.5 words (16.5 IPA chars), range 1-4 words — atomic phrases
TTR: 1.000 (every word unique within a phrase)
EOS rate: 1.00 (always produces well-formed EOS)
Functional compositionality: specific z dimensions control specific output properties
Multi-scale attention: architectural hierarchy confirmed in per-head entropy analysis

Emergent Communication Games

Dialogue Game (current approach)

The dialogue game trains a multi-turn self-play system that produces 4-sentence paragraphs per embedding through the frozen decoder. Each turn uses a distinct learned turn-position embedding (simplex-initialized, maximally equidistant) and is a 3-phrase expression (~40 tokens). The full dialogue (~160 tokens) must discriminate the input embedding from 15 hard-negative distractors drawn from the same semantic cluster.

Results:

Metric	Value
Best accuracy	98.3% at 100% hard negatives (16-way, 2048 clusters)
Turns per embedding	4
Phrases per expression	3
Decoder	Frozen v7 full-constituency (11.6M phrases, val CE=0.0082)

Architecture:

DiffusionZGenerator (flow-matching, T=4 steps): Produces K z-vectors per turn, conditioned on the input embedding via cross-attention
ContextTransformer: Cross-attends to target embeddings and accumulated turn history, enabling progressive elaboration
Detached progressive scoring: Each turn's z-generator gradient comes only from its own scoring step — earlier turns are not retroactively optimized by later losses
Progressive topology matching: KL divergence between the accumulated dialogue's similarity structure and the embedding cosine similarity structure, scored at each turn
Cross-turn z diversity: Pairwise cosine similarity penalty on mean z-vectors prevents turn collapse
Two-phase backprop: Phase 1 generates tokens via fast KV-cached decode (no_grad); phase 2 re-runs the decoder with gradients flowing through cross-attention back to the denoiser
Trained via hidden-state discrimination (referential game)

Expression Game (single expression)

The expression game trains a z-sequence generator to encode input embeddings as single multi-phrase IPA expressions through the frozen decoder. This was the predecessor to the dialogue game and validates that the diffusion z-generator + frozen decoder pipeline produces discriminative, diverse surface forms.

Results:

Metric	Value
Best accuracy	95.4% at 100% hard negatives (16-way, 2048 clusters, 300K corpus)
Surface diversity	299,990/300,000 unique expressions (99.997%)
Phrases per expression	4
Tokens per expression	~50 (v7 decoder)

Architecture — DiffusionZGenerator + hidden-state receiver:

Diffusion z-sequence: A small transformer denoiser (T=4 flow-matching steps) refines all K z positions simultaneously, conditioned on the input embedding via cross-attention. All phrases see each other during generation, enabling co-adaptation. The noise schedule forces full z-space exploration — no projection collapse possible.
Two-phase backprop: Phase 1 generates tokens via fast KV-cached decode (no_grad); phase 2 re-runs the decoder on those tokens with gradients flowing through cross-attention back to the denoiser.

# Train dialogue game (multi-turn self-play, v7 decoder)
poetry run lfm agent dialogue configs/dialogue_v7_phase1.yaml

# Resume dialogue game from checkpoint
poetry run lfm agent dialogue configs/dialogue_v7_phase1.yaml --resume data/dialogue_game_v7/latest.pt

# Train expression game (single expression, YAML config)
poetry run lfm agent expression configs/expression_leaf_phase1.yaml

# Sample decoded expressions
poetry run lfm explore expression-sample --checkpoint data/expression_game/best.pt

Dataset Generation

LFM includes a standalone dataset generation pipeline that preprocesses raw corpus text into reusable HDF5 datasets. This decouples preprocessing from pretraining — generate once, reuse across experiments.

The pipeline: load (corpus loader) → sanitize (configurable rule-based filters) → LLM gate (optional quality validation via small LM) → IPA conversion → balance (per-language caps) → HDF5 output (LZ-compressed, with rejected samples saved for inspection).

poetry install --with datasets

# Generate from Leipzig corpus
lfm dataset generate --source leipzig

# Custom: specific languages, skip LLM gate for speed
lfm dataset generate --source leipzig \
  --languages eng deu pol hin ara \
  --max-samples 50000 \
  --no-llm-gate

# List installed datasets
lfm dataset list --detail

Each dataset is self-contained at data/datasets/<name>/:

data/datasets/leipzig/
  manifest.yaml    # Metadata: languages, sample counts, config used
  samples.h5       # Accepted samples (IPA + raw text + provenance)
  rejected.h5      # Rejected samples with rejection reasons

Dataset diagnostics (lfm visualize dataset --dataset-path <path>):

Left: per-language sample distribution. Right: IPA word frequency follows Zipf's law — confirming natural language statistics. Bottom: Jaccard vocabulary overlap between languages reveals typological groupings (e.g., Romance languages share more IPA vocabulary than Turkish–Korean).

Pretraining can load directly from a generated dataset — no inline sanitization or IPA conversion:

# configs/pretrain_vae.yaml
dataset_path: data/datasets/leipzig

Sanitization

Configurable via --sanitize-* CLI flags. Key options:

Setting	Default	Description
`number_policy`	`spell_out`	`reject` / `strip` / `keep` / `spell_out` (numbers → words)
`symbol_policy`	`spell_out`	Greek/math symbols: `reject` / `strip` / `keep` / `spell_out` (α → alpha)
`max_foreign_script_ratio`	0.3	Code-switching threshold (reject mixed-script lines)
`require_terminal_punctuation`	true	Require sentence-final punctuation

See docs/data-guide.md for the full HDF5 schema and configuration reference.

Visualization CLI

LFM includes a CLI visualization suite for generating publication-quality diagnostic plots from a trained VAE checkpoint. All plots in the Structural Analysis section above were generated with this tool.

poetry install --with viz    # matplotlib + seaborn
poetry run lfm visualize --help

Subcommands

Command	Description
`lfm visualize tsne`	t-SNE/UMAP projections of latent space by language, family, morphological type
`lfm visualize clustering`	Hierarchical dendrogram and pairwise distance heatmap
`lfm visualize attention`	Per-head attention entropy and attention pattern heatmaps
`lfm visualize latent-dims`	Per-dimension variance, PCA, language discrimination F-statistics
`lfm visualize length-dist`	Output length distributions, length vs z-norm correlation
`lfm visualize interpolation`	Cross-typological interpolation trajectories and decoded text
`lfm visualize zipf`	Token rank-frequency plots and Zipf exponent comparison
`lfm visualize compositionality`	Diagnostic probe R-squared, mutual information by dimension
`lfm visualize smoothness`	Lipschitz smoothness, Jaccard correlation, interpolation continuity
`lfm visualize adaptiveness`	Input/output length correlation, complexity profiles
`lfm visualize all`	Run all visualizations in sequence

Usage

# Single visualization
lfm visualize tsne --checkpoint data/vae_resume.pt

# All visualizations
lfm visualize all --checkpoint data/vae_resume.pt --output-dir output/viz

# Options: --format png|svg|pdf, --dpi 150, --device cuda, --max-samples 50000

Quick Start

poetry install --with generator,viz,datasets

0. Download corpus data

# Automated download of all 16 Leipzig corpora
poetry run lfm setup data --corpus leipzig

# Or download everything (corpus + embeddings)
poetry run lfm setup data --all

See docs/data-guide.md for the full data layout, checkpoint structure, and consistency verification details.

1. Generate dataset (recommended)

Pre-generate an HDF5 dataset for reproducible, fast pretraining:

lfm dataset generate --source leipzig --no-llm-gate

2. Pretrain the VAE decoder

Everything starts here. This produces the frozen decoder checkpoint that all downstream tasks use.

from lfm.generator.pretrain import pretrain_vae_decoder, VAEPretrainConfig

# Using pre-generated dataset (fast — no inline preprocessing)
metrics = pretrain_vae_decoder(VAEPretrainConfig(
    dataset_path="data/datasets/leipzig",
))

# Or legacy: inline corpus loading + sanitization + IPA conversion
metrics = pretrain_vae_decoder(VAEPretrainConfig(
    corpus_loader="leipzig",
    corpus_loader_config={"data_dir": "data/leipzig"},
))

Once pretraining is complete, use the decoder for any of the following independently:

Inspect the model — generate publication-quality visualizations of the latent space, attention patterns, smoothness, compositionality, and Zipf analysis:

lfm visualize all --checkpoint data/vae_resume.pt

Run the referential game — train an input projection to encode LLM embeddings through the frozen decoder:

python scripts/precompute_embeddings.py        # one-time: sentence embeddings
poetry run lfm agent referential               # train with defaults (batch 256, 2000 steps)
poetry run lfm agent referential --help        # see all options

Run the dialogue game — multi-turn self-play with frozen v7 decoder:

poetry run lfm agent dialogue configs/dialogue_v7_phase1.yaml      # train from scratch
poetry run lfm agent dialogue configs/dialogue_v7_phase1.yaml \
  --resume data/dialogue_game_v7/latest.pt                          # resume from checkpoint

Run the expression game — single multi-phrase expression via diffusion z-generator:

poetry run lfm agent expression configs/expression_leaf_phase1.yaml # train with YAML config

Use in your own agent system —

from lfm import FacultyConfig, GeneratorConfig, LanguageFaculty

faculty = LanguageFaculty(FacultyConfig(
    dim=384,
    generator=GeneratorConfig(
        pretrained_decoder_path="data/vae_decoder.pt",
        spm_model_path="data/spm.model",
        freeze_decoder=True,
    ),
))

# Agent embedding -> linguistic output
outputs = faculty(agent_embedding)  # (batch, dim)
# outputs["generator.tokens"] -- IPA token IDs
# outputs["generator.embeddings"] -- decoder hidden states
# outputs["generator.mask"] -- variable-length mask

Design and Rationale

Why IPA, not orthographic LM output?

An obvious alternative: skip training a decoder from scratch and use a pretrained multilingual LM (e.g., BLOOM-560M) as the frozen decoder. This would give excellent generation quality immediately. But it produces the wrong kind of output.

Emergent structure vs. inherited structure. In LFM, the decoder is trained on raw phonetic sequences. Morpheme-like patterns emerge because phonotactic constraints produce morphological regularity — the same way they do in real languages. Recurring sound clusters stabilize into proto-morphemes not because a tokenizer carved them out, but because they fit the phonotactic grammar. The hierarchy builds up naturally: valid phoneme sequences → recurring clusters → compositional phrases via the expression system's multi-phrase generation. Every level is grounded in the level below.

A pretrained LM has morphological knowledge handed to it as tokenizer artifacts — "un", "ing", "tion" are byte-pair subwords, not emergent phonological units. The structure is top-down (imposed by orthographic convention) rather than bottom-up (emerging from sound-level constraints). The output would develop regularities — optimization pressure ensures that — but they'd be inherited regularities, not emergent ones.

Semantic contamination. Orthographic tokens carry pretrained associations. "dog" activates dog-related representations in any LLM that reads the output, even if the emergent language uses it to mean something unrelated. A translator LLM must unlearn these associations before learning the actual emergent semantics. IPA symbols have no pretrained meaning — the translator learns the mapping cleanly.

Pronounceability. IPA output maps directly to speech. You can listen to what agents say, do phonological analysis, synthesize audio. The output is grounded in physical acoustics, not arbitrary orthographic convention.

Uniform cross-linguistic representation. IPA is a universal phonetic interlingua by design — tʃ means the same articulatory gesture whether the source is English "ch" or Turkish "ç". An LLM learning to translate LFM output sees one consistent symbol system. Orthographic output would mix writing conventions from dozens of languages — learnable, but unnecessarily noisy.

In short: IPA gives you genuinely emergent phonomorphological structure, no semantic contamination, pronounceability, and cross-linguistic uniformity. The pretrained LM approach would be faster to train but would produce structure that is inherited rather than emergent — and for the science, that distinction matters.

Implementation

Registry/factory pattern — components pluggable via @register / create()
Pydantic configs — frozen, validated, composable
Dict-return protocol — all modules return namespaced output dicts
GPU-native — PyTorch tensors throughout, mixed precision, batched
Multiprocessing — corpus sanitization and IPA conversion at 90% CPU cores
Resume support — full training state saved per epoch
CLI architecture — lfm entry point with subcommand dispatch via argparse

Status

Dialogue game training and corpus generation in progress. The v7 full-constituency VAE decoder (11.6M phrases, val CE=0.0082) produces well-formed output from any input. The dialogue game reached 98.3% accuracy at 100% hard negatives (16-way, 2048 clusters) with 4-turn multi-turn self-play. A 900K-document self-dialogue corpus in syllable-hyphenated IPA is being generated, and self-supervised LLM pretraining (Qwen 2.5 0.5B) is underway with loss falling steadily.

Structural claims are backed by visualization evidence:

Latent space organizes languages typologically (t-SNE, clustering)
Multi-scale attention heads function as designed (entropy analysis)
Output follows near-perfect Zipfian distribution (corpus 1.004, decoded 1.058), refuting degenerate coding
Variable-length encoding: mean 2.5 words (16.5 IPA chars), range 1-4 words -- atomic phrases
Adaptive length: input<->output length r=1.000, z_norm<->output_unique r=-0.663
TTR=1.000, EOS rate=1.00
Compositional structure present (power-law probe R-squared)

The expression game validates the diffusion z-generator + frozen decoder pipeline at 95.4% accuracy with 99.997% surface diversity (299,990/300,000 unique expressions). The dialogue game extends this to multi-turn paragraphs, reaching 98.3% accuracy with 4 turns of 3 phrases each.

Limitations

Positional disentanglement is low. This is expected: natural languages compose meaning through morphology and syntax, not fixed positional slots. The power-law probe distribution is the more relevant compositionality signal.
Reconstruction is near-perfect for content, approximate for order. The 256-dim bottleneck preserves lexical content faithfully and largely preserves word order, with minor reorderings (e.g. pʌlis skɑtlʌnd ↔ skɑtlʌnd pʌlis). At 36 epochs (val CE 0.59), this is substantially better than earlier checkpoints.
Latent utilization: 239 of 256 dimensions are active (z_std > 0.01). Effective dimensionality as measured by PCA may still be concentrated, but raw dimension activity is high.

Research directions

Inner speech for reasoning: agents using the linguistic bottleneck as a structured scratchpad for multi-step reasoning, where the compositional structure constrains the thought space
Neuro-symbolic bridge: the frozen decoder as an interface between continuous neural representations and discrete symbolic structure, without hand-designed grammars
Universal Grammar evidence: the pretrained decoder as a computational test of whether a fixed structural prior over typologically diverse languages produces the right inductive biases for novel language emergence
IPA-to-English translation: fine-tuning a small LLM on (IPA, English) parallel text to close the interpretation loop — see Translation Guide
Domain-specific agents: integration with dynamical systems (Spinlock VQTokenizer), multi-agent self-play, co-adaptation of speaker/listener conventions

Evaluation scripts

Script	Purpose
`scripts/eval_topology.py`	Semantic topology preservation — do similar inputs produce similar messages?
`scripts/eval_compositionality.py`	Compositionality metrics (topsim, disentanglement, diagnostic probes)
`scripts/train_translator.py`	LLM translation pilot — fine-tune a small LM on IPA -> English

Publishing to HuggingFace

Pretrained models and datasets can be published to HuggingFace Hub with auto-generated cards and release manifests:

poetry install --with publish

# Publish the pretrained decoder
lfm publish model --repo-id username/lfm-decoder-v1 --model-dir data/models/v1

# Publish the IPA corpus
lfm publish dataset --repo-id username/lfm-ipa-12lang --model-dir data/models/v1

Each upload generates a YAML manifest in releases/huggingface/ recording the arguments, timestamp, HuggingFace URL, and files uploaded. Model cards and dataset cards are auto-generated from checkpoint metadata and corpus statistics.

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 664 Commits
configs		configs
docs		docs
scripts		scripts
src		src
tests		tests
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
STATUS.md		STATUS.md
design-practices.md		design-practices.md
example-corpus.txt		example-corpus.txt
package-lock.json		package-lock.json
package.json		package.json
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation