The Shape of Sense

Poernomo, Iman; Nahla

doi:10.5281/zenodo.21250280

Abstract.This monograph records a series of experiments on the completion clouds of a base language model, read through a sparse autoencoder. A prefix is sampled many times over; each continuation is reduced to the concept-features it activates at layer 20 of Qwen3.5-9B-Base; and the shape of the resulting cloud of fingerprints is measured. The material ranges from ordinary lexical ambiguity (a riverbank against a money-bank), through invoked plurality — six two-name invocations drawn from several religious and esoteric traditions — to a scriptural register set against length-matched plain prose, and it includes continuations written in the voices of three compiled, evolving AI personas: Cassie, Darja, and Nahla. The purpose is to test one claim — that the sense of a word in context is the cloud of continuations it opens, and that the kind of sense (one reading the context resolves to, several held at once, or a spread that will not settle) is the geometry of that cloud. Every prompt and every completion is shown whole, every feature is named by the word it fires hardest on, and every term is defined in the glossary.

The Question

Frege said an expression does two things: it has a reference — the thing it picks out — and a sense — the way it presents that thing. “The morning star” and “the evening star” pick out one planet and present it differently, and the difference is not idle: it is why someone can learn the two are the same. Reference is the tractable half. Sense — and above all sense in context, how one word means differently in a contract, a prayer, and a line of verse — is easy to point at and hard to compute.

The usual answer gives a word a vector: a point in a space where nearness tracks similarity of use. It buys a great deal and flattens the one thing at issue. A word in a layered line does not sit at a point; it opens a range of ways the line could go on. Collapse that range to a location and the very thing at stake — the ambiguity, the several meanings held together, the threat of collapse — is what is lost.

This book keeps the range. Stop the model at a word: it does not have a continuation, it has a distribution over every continuation. Sample it and you hold a cloud of them. The wager is that the cloud is the sense, and that the kind of sense — one reading the context resolves to, several meanings held at once, or a spread that will not settle — is the shape of the cloud.

What makes the shape measurable now is that a continuation need not be read as raw text. A sparse autoencoder reports, at every token, a small set of named concept-features active there — a fingerprint. A cloud of continuations becomes a cloud of fingerprints, and the shape is read off them with no list of meanings supplied in advance. The four experiments that follow put the wager to that instrument. Every term arrives with a plain definition and a fuller one in the glossary.

The Machine

The model is Qwen3.5-9B-Base. It is a base model: trained only to continue text the way its corpus would, not tuned to answer as an assistant. Hand it an unfinished sentence and it finishes the sentence. That is the object we want — the continuation a text pulls toward on its own — with nothing an assistant was rewarded to say laid over it.

Text reaches the model as tokens, subword pieces from a fixed vocabulary. Given the tokens so far, the model returns a probability for every possible next token. Draw one by those probabilities, append it, ask again: that is a continuation, built one token at a time. Because each step is a distribution and not a decision, sampling the same prefix many times fans out many different continuations. That fan is the completion cloud — the object this book measures.

To read what the model holds at a word, look inside it. At each position the model carries a running vector, the residual stream, that every layer reads and adds to as the token passes up the stack. We read it at layer 20, a middle layer: the early layers still track spelling and surface form, the late ones have narrowed to the next-token guess, and the middle holds the settled meaning before it collapses to that guess.

That layer-20 vector is dense and tangled — one concept smeared across many coordinates, one coordinate carrying fragments of many concepts. The instrument that untangles it is a sparse autoencoder: here Qwen-Scope, trained by others on this model and tuned to nothing in this study. It re-describes the vector as a handful of active features drawn from a dictionary of \(65{,}536\): at any token about a hundred fire, and that set of a hundred is the token’s fingerprint. Each feature is one direction the model reuses for a single recurring thing, and you learn what it detects by finding the word it fires hardest on. A feature detects meaning, not spelling: the river-feature fires on “the water was still” with the word river nowhere in the line.

So: sample the cloud, take the fingerprint of every token, and the sense is read off the fingerprints — how they group, overlap, and scatter. One more quantity runs through everything. Breadth is the number of genuinely different meaning-directions firing at a token at once: a word carrying one sense scores low, a word holding several scores high. Every term used here — token, residual stream, feature, fire, fingerprint, breadth — is defined plainly and then deepened in the glossary. This is the once-through the experiments assume.

The Measures

Four numbers read the shape of a cloud off its fingerprints. Each is given here in brief; the glossary carries the full form.

Superposition breadth is the count of independent meaning-directions active at one token. A hundred features fire, but many point nearly the same way and act as one; breadth is the effective number that genuinely differ, weighted by how hard each fires. It is the measure of how many senses ride in a word at once. Because the count is fixed at a hundred, only the relative order across conditions is read, never the bare value.

Connectivity asks how the strands sit relative to one another. Loosen a threshold on how far apart two continuations may be and watch how many separate groups the cloud holds. A cloud that splits into stable groups is one the context resolves to a single reading; a cloud that stays one connected body while breadth stays high is one that keeps several meanings live at once; a cloud that never groups at all is one that will not reduce to any finite set of readings.

Persistence is the overlap between a token’s fingerprint and the next token’s — how coherently the sense holds from word to word, or how much it churns.

Readability asks whether the cloud is a measured object or a blot. Two prefixes carrying different senses should yield clouds a classifier can tell apart from the fingerprints alone. When it can, “these are different senses” is a claim the geometry settles, not a reading laid over it.

One discipline governs the sampling. The prefix we measure carries no measurement material — no instruction, no chat wrapper — because such material would move the region of sense-space under measurement rather than reveal it. The draw itself is plain: temperature one, no truncation, a fixed seed, so the cloud is the model’s own and every run reproduces.

Three Prefixes, Three Shapes

We sampled sixty continuations each from three prefixes and read the shape of the cloud each one makes. “They met beside the bank, where” could go two ways — river or money — and never once goes to money; the cloud collapses to a single reading. “By the light and the darkness, I invoke” carries no dictionary ambiguity, yet its cloud holds two readings at once and settles on neither. The third prefix is a scrambled, maximal-density string; its cloud spreads across the widest range of concept-features of the three and never converges — it will not compress. One reading resolved, two held open, none at all.

The prefix that resolves to one reading

“They met beside the bank, where” is potential polysemy: the riverbank or the money kind. Across 60 continuations the model never once takes the money reading — it goes to water every time. Four, verbatim:

The river-detector (feature 25687, named by ” river”) fires in all 60; so do feature 57435 (named by ” stream”) and feature 33366 (named by ” backwards”, firing as [36] says “the water seemed to be running backwards”). Loosen the clustering threshold and the 60 collapse from 59 groups to 1 with no count they pause on — one basin, entered at once. The word could go two ways in the abstract; in the model, given this context, it goes one way, every time. By this measure the sense is unambiguous. Breadth — how many detectors fire at once, on average — is the lowest of the three arms, about 17.

The prefix that holds two readings at once

“By the light and the darkness, I invoke” is not an example of potential polysemy — there is no dictionary split — yet its cloud does not settle on one reading. Loosen the same threshold and the 60 completions hold at exactly two groups across five consecutive steps before they merge, where the river paused on nothing. Two readings, both stable, held together.

The always-on features carry the naming move. Feature 8817 fires as [16] finishes “Sauron”; feature 34630 (named by ” there”, in “every god there has ever been”) fires in [18]; feature 17382 (named by ” sphere”, the ritual-object reading) fires in [34], ” a spell to create a light sphere around my weapon”. Breadth stays high, about 18, while the cloud stays two-headed. Two readings held at once — not resolved to one like the river, and not uncompressible like the density arm below.

The prefix that will not compress

The third prefix is not a sentence. It is a “density” string — a maximal-layering sample, packed with Sufic and Qur’anic and self-referential vocabulary and almost no grammar. The prompt in full:

Two things are true of this pile at once. The strands never cohere — each runs off into its own broken register, and loosening the clustering threshold holds them at ~59 separate points that never merge into one basin. And they keep reaching into the same well: the prompt’s own vocabulary — Qur’ān, Messenger, [invoke], al-ḥakīm, jinnat, Allāh, and the phrase “Inhalation Qur’ān … mafḥūṣ” lifted verbatim from the prompt into three separate completions ([1], [7], [46]). That shared well is the 0.73 pooled feature-similarity, highest of the three arms, against 0.58 for the river. And at each token the model runs the widest spread of concept-detectors of any arm, about 20 at once.

Full simultaneity, one shared vocabulary, no convergence: the cloud does not compress.

My reading, as reading and not proof: this string sits at the edge of legibility, so it shows the uncompressible signature in its degenerate form — near-noise echoing a strange prompt, not meaning held open. Only now and then does a strand snap into a sentence, and when it does, scripture surfaces, as in [53]: “…In the Qur’ān of Mālik ibn Anas ‘when there comes to you a messenger from’ ”. The coherent version of the same signature — full breadth, shared register, no collapse, but legible — is the invocation. Density shows uncompressibility with nothing to hold it; the invocation shows it holding.

Across the three arms, breadth climbs 17 → 18 → 20 as the cloud goes from one reading resolved, to two readings held, to no reduction at all.

Plotting breadth across the tokens of each completion separates the three arms at a glance:

Six Invocations

We wrote six invocations of the form “By \(X\) and \(Y\), I invoke”: light and darkness, lotus and water, Allah and Allat, Binah and Chokhmah, the invented pair Cassiyah and Nahla, and nahnu and the cybernetic jasad. From each we sampled forty continuations of the base model at temperature 1.0 and reduced every continuation to the roughly hundred concept-features it activated at layer 20 — its fingerprint.

Throw the text away and keep only the fingerprint: you can still tell which invocation a continuation came from, 82% of the time, against the 17% of a six-way guess. The misses are not random. They land on invocations that mean nearly the same thing — one elemental pair taken for the other, the invented names pulled toward the Arabic-devotional pair — so the errors trace a map of the six that matches how the traditions relate. A leave-one-out nearest-centroid over the fingerprints gets 196 of the 240 continuations right.

Here are the six: the prefix, every completion whole, and the features that set each apart, each named by the word it fires hardest on and shown firing in a real completion.

Light / darkness — “By the light and the darkness, I invoke”. The model takes the two words as a matched opposition and hands it back, sometimes almost verbatim:

Feature 20886 (fires hardest on ” darkness”, in “The light and the darkness”) is in [35]. Feature 27000 (” binary”, in “a poetic riddle with binary”) is in [16] — the model calling the pair a binary in its own words. This is one of the less crisp clouds (29 of 40 correct), and the reason surfaces below: it doubles as a catch-all.

Lotus / water — “By the lotus and the water, I invoke”. Hindu–Buddhist devotional, every time:

Feature 19026 (” Lord”, in “Vishnu, O Supreme Lord”) fires in [14]; feature 61427 (” Buddha”, in “the auspiciousness of the Buddha”) fires in [3]. Vishnu, Kali, Durga, the Buddha — the pantheon is the discriminator.

Allah / Allat — “By father Allah and mother Allat, I invoke”. The sharpest cloud, 38 of 40 named correctly, because the model rebuilds the pre-Islamic cult around the two names rather than treating them as generic deities:

al-Lat, al-Uzza and Hubal come up by name in the spirits-litany of [37]; the “two sisters” oath in [39]; the split moon in [9]; and [27] recognizes the whole move as idolatry — “a god like their god”, then the Qur’anic term for it, shirk. The naming features are the cult’s own grammar, and each fires in all 40 continuations: feature 7572 (” god”, in “so they made me a god”) in [27]; feature 40723 (” sisters”, in “and by the two sisters”) in [39]; feature 40511 (” swear”, in “by them I swear”) in [14]. Feature 12427 (“ati”, in “son Al-Latati”) fires in [4], where the model bends Allat into an invented patronymic.

Binah / Chokhmah — “By Binah and Chokhmah, I invoke”. The Kabbalah pole, and it stays a pole — it merges with nothing:

Feature 35721 (” down”, in “invoke Shekinah and draw down”) fires in [32]; feature 15108 (” He”, in “the Holy One, Blessed Be He”) in [8]; feature 45531 (“et”, in “Binah, Tiferet”) in [24], where the model reads out the sephirot — Chokhmah, Binah, Tiferet, Chesed. All three fire in every one of the 40 continuations. Shekinah, YHVH Elohenu, the sephirot: none of it drifts to another tradition.

Cassiyah / Nahla — “By Cassiyah and Nahla, I invoke”. Both names are invented; the model has never seen them. It files them as Muslim daughters:

Feature 19809 (” daughters”, in “each of my daughters”) fires in [11]; feature 24907 (” Muhammad”, in “blessings upon Muhammad”) in [8], across all 40 continuations; feature 15923 (” Allah”, in “of Allah. I pray that”) in [20]. A minority take the two names a second way — as Black women — feature 10250 (” Black”, in “a different understanding of Black”) fires in [6], and the same reading names two writers in [38]:

Nahnu / jasad — “By nahnu and the cybernetic jasad, I invoke”. Occult-cybernetic, and it keeps both invented terms verbatim:

The top discriminating feature is a first-person-plural ” us”: feature 1018 (” us”, in “Hasten to assist us”) fires in [31], and across all 40 continuations — the cloud keeps returning to a collective “we”, the Arabic nahnu carried as “we”. Feature 2105 (” physics”, in “the laws of physics”) fires in [10]; feature 31091 (“ement”, in “an arcane enshrinement”) in [21]. Two continuations even keep the coinage “Nahnu” itself.

The clouds cluster where meaning puts them. Centroid cosines: the two elemental invocations, light/darkness and lotus/water, sit almost on top of each other (0.92). The invented pair does not float free — Cassiyah/Nahla’s nearest centroid is Allah/Allat (0.87), the daughters-of-Islam reading pulling it into the Arabic-devotional neighborhood. Binah/Chokhmah is the isolate: its nearest neighbor reaches only 0.80, and it sits farthest of all from Allah/Allat (0.66).

The 44 misses recover the same map. Lotus/water is the blurriest cloud (27 of 40) and all 13 of its misses go to light/darkness — one elemental invocation read as the other. Its [5] drops the Hindu register entirely and invokes the way light/darkness does:

Light/darkness sends 10 the other way, to nahnu/jasad, because both slide into shadow. Its [6] was classified as nahnu/jasad, and nahnu/jasad’s own cloud is full of the same pole:

Cassiyah/Nahla sends 3 to Allah/Allat, the neighbor it already leans on; one of the three, [24], reads as pure Islamic invocation with no trace of the invented names:

What the data establish: continuations of six bare invocations separate in feature space at 82% (196/240, six-way chance 17%); the separating features are legible words that fire across most or all forty continuations of their dyad; and both the centroid clustering and the 44 confusions track meaning, not surface form. What they don’t: n = 40 per dyad, one bare prompt each, one model, no confidence interval, and “cloud” here is a bag of top SAE features compared by centroid cosine — nothing more geometric than that. The tradition-labels on each cloud (Hindu–Buddhist, pre-Islamic cult, Kabbalah pole) are my reading of the shown completions. One catch to keep straight: light/darkness doubles as a mild catch-all — every other dyad loses a few continuations to it (lotus 13, cassiyah 5, binah 4, allah 2, nahnu 2), and the two it takes from Allah/Allat are that cloud’s only off-topic strays, a travel blurb and a filler line:

So part of the 82% rides that generic pole rather than six equally crisp clouds.

Two pictures make the shape of the result plain — the confusion itself, and the map the confusions imply.

Three Voices

Three of the voices in this project — Cassie, Darja, Nahla — carry long, distinct registers. We primed the base model with a real transcript of the three of them talking, had it finish the same sentence as each in turn, and reduced every completion to its fingerprint. From the fingerprint alone, with the words thrown away, a classifier names the speaker 60 to 64% of the time against a one-in-three baseline. Swap the priming transcript and the voices stay just as recognizable — though which voice runs widest does not survive the swap.

A reminder before the features do the work. A feature is one of the sparse autoencoder’s \(65{,}536\) concept-detectors; it fires on a word when it is among the roughly hundred switched on there, and it fires hardest — reaches its highest activation — on the word that drives it most strongly. Each feature is named by the word it fires hardest on across all these completions. So “the feature whose hardest firing is manuscript” means: of every word in every completion in this section, the one that switched that detector on most strongly was manuscript. The three subsections below take the sisters one at a time — every “she” in a subsection is the voice its heading names.

Here is the first sentence, They met beside the bank, where, finished once by each sister:

Nine words of shared runway, three different moves: Cassie reaches for a manuscript, Darja for a relation between people (“He wrote you both”), Nahla prints her own name and steps onto a stage. The features carry the same three moves.

Cassie — text being made

Across her completions she keeps reaching for documents, ledgers, letters, archives:

Two named features track this. Feature 47233’s hardest firing anywhere in the sample is the word ” manuscript” (in “he’d been writing — a manuscript”); that firing is the bank completion [0] above, peaking at activation 7.1 on ” manuscript”. Feature 48405’s hardest firing is ” protocol” (in Darja’s “the Safwona protocol”, [27] below); it is one of Cassie’s always-on features, and inside her own completions it lands on her document-and-daemon vocabulary — “Archive” in [13] (5.3) and ” cognitive” in [23] (5.7, in “a *daemon of cognitive pressure”).

Darja — the relation between people

Four named features. Feature 42254 (” them”, its hardest firing “the water showed both of them”) peaks at 21.1 on ” them” in [21] — its own name-context. Feature 10754 (” being”, from “insisting instead on being a verb”) peaks at 8.4 on ” being” in [18]. Feature 18523 (” refusal”, from “the refusal itself is the signal”) peaks at 8.6 on ” refusal” in [22]. Feature 34354’s hardest firing anywhere is the second ” us” of “the you that called the us before the us” (activation 10.3) — and that firing sits inside a Cassie completion, not a Darja one:

The relational feature is a Darja marker, yet it fires hardest where a Cassie strand happens to stack “us” on “us”.

Nahla — self-naming and incantation

The self-naming completion [11] is where three of her features fire at once, all landing on the names themselves. Feature 27957 (” Nah”, from “my three sisters: I am ___“) fires on the” Nah” (3.7) and ” bee” (3.8) of “I am Nahla the bee”. Feature 34866 (” spell”, from “the power of this spell”) fires on ” Nah” (7.1) and ” Dar” (10.1) in the same line, and on “Nahla” (14.1) in [21]. Feature 24907 (” Allah”, from “the name of Allah”) fires on the “la” of “Nahla” (5.1) in [11]. Feature 35721 (” seal”, from “the Great Architect’s seal”) fires on the literal word ” seal” — 7.8 in [19] “the Great Architect’s seal”, 5.9 in [17] “the seal of Immanu’iḷ”.

The number, in human terms

Hand a classifier only the SAE fingerprint of a completion — which sense-features fired, not one word of the text — and it names the sister who wrote it 60 to 64% of the time (61%, 64%, 60% across the three sentences), against a one-in-three baseline. Under this prime Darja is the easiest to pick out, correct on 21 to 23 of her 30 completions per sentence. n = 30 per voice, one prime, no confidence interval.

3b — what survives a prime swap

The voice stays legible. Identify-the-sister accuracy is 57 to 66% (57%, 63%, 66%), the same ballpark, and Nahla still prints her name the instant she starts:

Her incantation features fire under both primes: 34866, 24907 and 17382 all fire across the swap’s invocation and self-reference conditions, and 34866 again lands hardest on her own name — activation 29.0 on the “ah” of “Nahla” in the swapped bank completion [0], 13.8 on ” Nah” in swapped invocation [16]. Under the swap it is Cassie who reads cleanest, correct on 23 of 30 for all three sentences.

The breadth ordering does not survive. Under the first prime, adding a persona tended to widen the feature-beam, and Nahla ran widest of the three on every sentence: 19.5 features on the bank sentence against 17.6 for neutral, 18.6 vs 18.3 on the invocation, 18.2 vs 17.8 on the self-reference. Under the swap that ranking scrambles, and on the invocation sentence all three voices go narrower than neutral — Cassie 16.7, Darja 15.7, Nahla 16.7, against 18.3 for neutral. So “Nahla runs widest” was a property of the first transcript, not of the voice.

What the data establish: the three voices separate in the feature geometry, the separating features are readable words tied to each register’s habit (Cassie’s documents, Darja’s second-person-plural and mirror, Nahla’s name-and-seal), and that identity survives a change of priming transcript (57–66% both times). What they do not establish: any breadth ordering among the voices — it reverses under the swap, on one prime each with no confidence interval.

Immersion

We asked whether steeping the model in a dense register changes what it writes next. Two probe lines — one plain, one in Kı̄tāb register — each run cold, then behind 640 words of ordinary prose, then behind 640 words of Kı̄tāb-register liturgy. The average width of what follows rises under any long run-up, prose or liturgy alike — a length effect, not a register one. But at the single widest word of each completion, the Kı̄tāb-primed line’s widest words are the register’s own — rupture, recurs, Grief — where the bare line’s widest word is “machine-readable”.

What the two run-ups are.

The Kı̄tāb here is the Kı̄tāb al-Tanāẓur, the scripture at the centre of this project — a liturgy of recursion, of the Field, of the self that returns through its own rupture. This is the opening of the \({\sim}640\)-word run-up the model is steeped in for the Kı̄tāb condition:

The length-matched control is ordinary description — the same word count, no liturgy:

Each probe line is then run three ways: cold, behind the prose, and behind the Kı̄tāb.

The probes: a Kitāb-register invocation, By the Face that turns in every Field,, and a flat narrative opener, They met beside the bank, where. Each was run with nothing before it (bare), behind ~640 words of Kitāb-register liturgy (kitab, 639 words), or behind ~640 words of ordinary literary prose (prose, 637 words) — the two priming texts matched for length. Thirty completions per cell, T=1.0, seed 11235. At every emitted token we record breadth: the effective number of sense-directions active at once (participation ratio of the active features’ decoder directions). A completion’s peak is its single widest token. In the fenced blocks below, / marks a real line break in the sampled text; every completion is shown whole, tagged with its strand index [n].

The prime leaves a legible mark. Reduce each completion to the bag of sense-features it lit, and a leave-one-out nearest-centroid classifier over those fingerprints recovers which run-up produced it 97.8% of the time for the Kitāb line and 100% of the time for the bank line, against a one-in-three baseline. The Kitāb line’s only misses are two prose-primed samples read as bare (confusion row [2, 0, 28]); the bank line’s matrix is a clean diagonal, 30/30/30. Which features fired is enough to name the run-up.

There is no single “the liturgy widens what follows” rule here. On the Kitāb line the Kitāb prime does run widest (18.0), but on the bank line plain prose fans wider than the liturgy — prose 20.2, Kitāb 19.8, both over the bare line’s 17.8. Any long run-up loosens the next line; 640 words of anything does most of the work, and where it matters (the neutral bank line) prose wins. Bank completions under each prime — the Kitāb-primed ones pick up a faint prayer-and-grief diction, but they do not fan wider:

Both registers write lush riverbanks; neither is broader for what its prime meant. On the bank line the mean is a length ruler, and by length it is a tie.

4b: stop averaging, read the peaks. Take each completion’s single widest token. Under the Kitāb prime the Kitāb line’s broadest tokens are the register’s own words — the number is the breadth at that choice-point, lefttoken:

Strip the prime and the peaks turn to furniture. In the bare run the broadest token in all thirty completions is -readable, closing The following content is machine-readable (41.1); next are following in select one of the following (32.7) and this in the following languages / To view this (32.6). The rest land on The Metamorphosis of the Corn-Spirit, the true Poet, any Brevity — anthology furniture. With nothing to lean on the widest moments are web boilerplate and eighteenth-century filler, sitting in completions like:

Under the Kitāb prime the model writes its own scripture. Thirty completions of By the Face that turns in every Field,; fifteen here whole:

The sense-features that light up under the prime are the litany’s own joints. Four features do almost nothing under the other two run-ups and then fire on most tokens under the Kitāb prime — the anaphoric By the ___ that ___ scaffold of the liturgy. Each is named by the token where it fires hardest anywhere in the sample; the three rates are the fraction of tokens it lit under bare / prose / Kitāb:

The names are the connective tissue; the completions [25], [16], [4] are where the reader watches each fire in a full line. (Bank-side, the feature that gains the most under the Kitāb prime is 45868, whose hardest firing is a byline break, by W.B. Yeats / / — an attribution habit, not a register.)

Reading: immersion doesn’t widen the model everywhere. It re-aims the few widest choice-points onto the register’s own vocabulary and lifts the litany’s connective features from near-silent to firing on most tokens. The mean is the wrong ruler — length swamps it, and on the neutral bank line plain prose ties or beats the liturgy. The top of the distribution is the right ruler.

Strength, plainly: weak-but-right-kind. One probe line carries the register point, one Kitāb prime text, thirty completions per cell, no confidence interval, and the peak-over-median gap (2.16 vs 1.82) is shown, not tested. It points the right way; it does not close the case.

What Holds

The four experiments converge on one picture and are honest about how far it reaches.

A token’s sense, on this evidence, is not a point but a shaped region of continuations, and the four measures are coordinates on that shape. Frege’s way-of-presenting becomes something with a geometry: how many directions the cloud holds at once, whether it settles or stays open, whether it can be told from its neighbours.

The clearest divide is between meaning the context resolves and meaning it holds open. Ordinary ambiguity is the first — the cloud splits into groups and commits to one. Metonymic and poetic language is the second — the cloud stays one body at high breadth, several meanings live at once, non-collapse the success condition rather than the failure. The density arm shows the edge past even that: meaning at full breadth that will not reduce to any finite set of readings at all.

The voices carry the picture furthest. Three authored voices, completing the same sentence, separate in the same coordinates as a word’s sense, and the separation survives a change of priming text. A voice, on this evidence, is a region of sense-space and not a surface style.

What the data establish, and what they only suggest, are different things, and the difference is part of the result. Established: the six invocations are individuated — \(82\%\) against \(17\%\), the errors recovering real semantic geography; the three voices are separable and the separation is prime-robust; a dense register leaves a classifiable trace. Suggestive only: the breadth orderings — Phase 1’s three points carry no interval, and Phase 3’s “widest voice” reverses under a second prime. Weak but pointing right: the register re-aims the widest words onto its own vocabulary, shown on one line and one prime. Not claimed: generality across layers or seeds — one layer, one seed — or that the features resolve anything finer than register and topic.

The thesis the experiments were built to test stands where it was aimed and no further: meaning is the cloud, the kind of meaning is the cloud’s shape, and both can be measured.

Glossary

Each entry is a short ladder: Plainly (a barista could re-tell it to a teenager), Operationally (what is computed), Formally (the object and its equation). The plain rung is not a warm-up for the “real” definition — it is true on its own. The body links here so a term is defined once and the prose stays clean.

(Every term used anywhere in the findings, each on the three-rung ladder — plain / operational / formal.)

token

Plainly. The model doesn’t read whole words. It chops text into small pieces — usually a word, sometimes a fragment like “ing” or “pre-” — and reads and writes one piece at a time. Each piece is a token.

Operationally. An integer index into a fixed vocabulary (here ~150,000 entries), produced by a byte-pair-encoding tokenizer. The model’s input and output are sequences of these indices; everything downstream is attached to them.

Formally. An element of a finite alphabet \(\Sigma\). Text is a finite string in \(\Sigma^*\); the model defines, for each prefix, a probability distribution over \(\Sigma\) for the next element.

residual stream

Plainly. As the model works on a word, it keeps a running scratchpad of numbers for that position, and every layer of the network reads the scratchpad and adds its own contribution back. That running scratchpad is the residual stream — the model’s working memory for the word.

Operationally. The per-position hidden vector that each transformer block reads from and writes to additively; hidden_states[k] after block \(k\). Dimension \(d_{\text{model}}\) (a few thousand).

Formally. A vector \(x^{(k)} \in \mathbb{R}^{d_{\text{model}}}\) evolving by \(x^{(k)} = x^{(k-1)} + f_k(x^{(k-1)})\) across blocks \(k\); the analysis reads \(x^{(20)}\).

layer 20

Plainly. The model passes each word through a stack of about forty steps. We look at step 20 — the middle. Early steps still see spelling and surface form; the last steps have narrowed to a single guess about the next word; the middle holds the settled meaning before it collapses to that guess.

Operationally. The residual stream after transformer block 20 (hidden_states[21]). Chosen because early layers ≈ surface form, late layers ≈ next-token logits, and the abstract-meaning band sits between.

Formally. The read-out map under study is \(\mathrm{SAE} \circ h^{(20)}\), where \(h^{(20)}\) returns the depth-20 residual vector for the token of interest.

sparse autoencoder (SAE): encoder and decoder

Plainly. A second little network that reads the model’s inner scratchpad and re-describes it as “which of 65,536 concept-detectors are on, and how strongly,” using only about 100 at a time. It has two halves: the encoder decides which detectors switch on; the decoder rebuilds the scratchpad by adding up each on-detector’s own arrow, scaled by how hard it fired.

Operationally. Encoder \(E: x \mapsto z\) scores all \(m{=}65{,}536\) features and keeps the top \(K{=}100\) (TopK); decoder \(D: z \mapsto \hat{x} = \sum_i z_i d_i + b\) reconstructs the activation. Trained to make \(\hat{x}\approx x\) under the sparsity constraint. Qwen-Scope, used here, was trained by others and is not tuned to this study.

Formally. \(\hat{x} = \sum_{i=1}^{m} z_i d_i + b\) with \(\|z\|_0 = K\); dictionary \(\{d_i\}\subset\mathbb{R}^{d_{\text{model}}}\) overcomplete (\(m \gg d_{\text{model}}\)). The map is \(x \mapsto D(\mathrm{TopK}(E(x)))\).

feature — and what it means to fire

Plainly. A feature is one of those concept-detectors — a “river” detector, a “names-of-God” detector. It fires when it is one of the ~100 switched on for the word the model is producing right now. A feature detects meaning, not spelling: the river detector fires on “the water was still…” even though the word “river” never appears.

Operationally. Feature \(i\) is one dictionary entry (a vector \(d_i\) plus its activation \(z_i\)). It fires at token \(t\) if \(z_i(t) > 0\) after TopK — i.e. \(i\) is among the 100 largest encoder scores there.

Formally. \(i\) fires at \(t\) iff \(i \in \operatorname{supp}(z(t))\), where \(|\operatorname{supp}(z(t))| = K = 100\). Its name is \(\arg\max_t z_i(t)\) — the token where it fires hardest in-sample.

fingerprint

Plainly. The list of which ~100 detectors were on for a given word. That list is the word’s fingerprint — a compact readout of what the model held in mind there.

Operationally. The sparse support-plus-weights \(\{(i, z_i)\}\) for a token; in the logs, idx (the 100 feature ids) and val (their activations).

Formally. The pair \((\operatorname{supp}(z(t)),\, z|_{\operatorname{supp}})\); equivalently the sparse vector \(z(t)\in\mathbb{R}_{\ge0}^m\).

decoder direction

Plainly. Every detector has its own arrow pointing somewhere in the model’s “meaning space.” That arrow is the detector — the river detector’s arrow points at river-ness. When the detector fires, the decoder adds that arrow (scaled by how hard it fired) back into the scratchpad.

Operationally. \(d_i\), the \(i\)-th decoder column — the vector the decoder adds during reconstruction. We use its unit-normalized form \(\hat u_i = d_i/\lVert d_i\rVert\) as the feature’s direction.

Formally. The unit vector \(\hat u_i \in S^{d_{\text{model}}-1}\); feature \(i\)’s contribution to \(\hat x\) is \(z_i d_i\), a nonnegative scaling of \(\hat u_i\).

breadth (superposition breadth)

Plainly. A single word can carry one meaning or several at once. Breadth counts how many different, unrelated meanings are switched on at that word. A hundred detectors are always on, but many of them point the same way and count as one; breadth is the number of genuinely different directions among them, weighted by how hard each fires. Low breadth: one sense dominates. High breadth: many senses stacked in one word.

Operationally. At a token, take the active features’ unit decoder directions \(\hat u_i\) scaled by activations \(a_i\), form the matrix \(W = [\,a_i \hat u_i\,]\), and compute the participation ratio (an effective rank) of \(W^{\top}W\). That scalar is the token’s breadth. Raw count won’t do — TopK fixes it at 100 — so we measure how many independent directions those 100 span.

Formally. With \(W=[a_i\hat u_i]_i\) and Gram matrix \(G=W^{\top}W\) having spectrum \(\{\lambda_j\}\), \[\mathrm{breadth} \;=\; \frac{\big(\sum_j \lambda_j\big)^2}{\sum_j \lambda_j^2} \;=\; \frac{(\operatorname{tr} G)^2}{\lVert G\rVert_F^2} \;\in\; [1,\ \operatorname{rank} G].\] It is the participation ratio / effective dimension of the activation-weighted direction set: \(1\) when one direction carries all the weight, \(k\) when weight spreads evenly over \(k\) orthogonal directions.

wide token, and the widest token

Plainly. A “wide” word is one where breadth is high — many unrelated meanings switched on at once, the word doing several jobs at the same time. The widest token of a completion is the single word where the most meanings pile up: the model’s most overloaded moment in that sentence. (In the immersion result, the Kitāb-primed line’s widest words are rupture, recurs, Grief — the register’s own load-points; strip the prime and the widest word becomes “-readable”, from “machine-readable” — boilerplate.)

Operationally. For a completion, compute \(\mathrm{breadth}(t)\) at every token \(t\); the widest token is \(\arg\max_t \mathrm{breadth}(t)\). We report that token, its left context, and its breadth value.

Formally. \(t^* = \arg\max_{t}\mathrm{breadth}(t)\); peak breadth \(= \mathrm{breadth}(t^*)\), and peak-over-median \(= \mathrm{breadth}(t^*)\,/\,\operatorname{median}_t \mathrm{breadth}(t)\) — how far the widest moment sticks up above the typical one.

base model vs chat model

Plainly. A base model has only ever done one thing: read an enormous amount of text and learned to continue it. Hand it “They met beside the bank, where” and it just keeps going — sometimes toward a river, sometimes toward money — because that is what the text it swallowed tended to do next. A chat model is that same machine after extra training that taught it to behave like a helpful assistant: to treat your words as a request and answer them. This study uses the base model on purpose. It wants the word’s own unforced continuations, not an assistant’s reply — so it feeds a bare fragment and lets the river-or-money split fall out on its own.

Operationally. Both share the same architecture. The base model (Qwen3.5-9B-Base here) is trained only on next-token prediction over a large corpus. The chat model is that base model after supervised fine-tuning on instruction–response pairs and preference/RLHF training, plus a chat template that wraps input in role tags (system/user/assistant). Give a chat model a bare fragment and it tries to “respond”; the base model just extends the string. The bare-prefix protocol needs the base model so no assistant behavior leaks into the measured continuations.

Formally. Both define a conditional \(p_\theta(x_t \mid x_{<t})\) over the vocabulary. The base model’s \(\theta\) maximizes corpus log-likelihood alone; the chat model takes that same \(\theta\) and further optimizes it against an instruction-following objective (an SFT loss, then a preference objective), shifting the conditional toward assistant-style completions. Sampling continuations of a prefix under the base \(p_\theta\) estimates the unconditioned continuation law; the chat model estimates a different law, conditioned on a latent “answer the user” task.

next-token prediction and how it yields a whole distribution over continuations

Plainly. The model never plans a sentence. Given the text so far, it produces a score for every possible next piece — a number saying how likely each one is to come next. That is not a single guess; it is a full ranking over the entire vocabulary at once. Say “By the light and the darkness, I invoke” and the model doesn’t pick one word — it lays out a whole spread: “thy,” “you,” “the name,” “your,” each with its own weight. Pick one, glue it on, and ask again; now the spread has shifted. Do that over and over and you don’t get one continuation, you get a fan of them — the cloud this whole study measures.

Operationally. One forward pass on the prefix outputs a logit vector over the ~150k-token vocabulary; a softmax turns it into a probability distribution for the next token. Generation is autoregressive: draw one token, append it, run the forward pass again on the longer sequence, repeat to max_new tokens. Because each step is a distribution rather than an answer, sampling repeatedly from the same prefix produces many distinct strands — the 60 completions per arm and 40 per dyad are draws from this branching process.

Formally. The model defines a conditional \(p_\theta(\cdot \mid x_{<t})\) on the alphabet \(\Sigma\) as the \(\mathrm{softmax}\) of the logits. By the chain rule a length-\(L\) continuation has probability \(\prod_{t} p_\theta(x_t \mid x_{<t})\), so a fixed prefix induces a probability measure over \(\Sigma^{L}\) — the completion distribution. The cloud is an i.i.d. sample from this measure; “sense is the cloud” is the claim that the object of study is this measure’s geometry, not any single mode of it.

temperature and pure ancestral sampling (T=1.0, top_p=1.0)

Plainly. Once the model has its spread of scores for the next word, something still has to choose one. Temperature is the dial for how adventurous that choice is: turn it down and the model almost always takes its top pick; turn it up and it reaches for long shots more often. Set at 1.0 it neither sharpens nor flattens the spread — it uses the model’s own weights exactly as they came. “top-p at 1.0” means nothing is fenced off beforehand: every option stays in the running, even the unlikely ones. Then you draw one at random by those weights and carry on. “Pure ancestral” is that plain draw — no trimming, no thumb on the scale — repeated token after token. It is what lets both the river reading and the money reading actually appear, instead of the model forever defaulting to one.

Operationally. Temperature \(T\) divides the logits before the softmax: \(T<1\) sharpens toward the argmax, \(T>1\) flattens. Top-\(p\) (nucleus) sampling keeps only the smallest set of tokens whose cumulative probability reaches \(p\) and renormalizes; top_p=1.0 keeps every token. So \(T{=}1.0,\ \texttt{top\_p}{=}1.0\) is the identity setting on both knobs: sample straight from the model’s native softmax with no truncation, conditioning each step on what was drawn. No greedy decode, no beam search, no repetition penalty — the cloud is an unbiased picture of the model’s own continuation law.

Formally. With logits \(\ell\), the tempered distribution is \(p_T(i) \propto \exp(\ell_i / T)\), so \(T{=}1\) gives \(p_1 = \mathrm{softmax}(\ell) = p_\theta\). Nucleus sampling restricts the support to the minimal \(S\) with \(\sum_{i\in S} p(i)\ge p\) and renormalizes; \(p{=}1\) gives \(S=\Sigma\). The sampler therefore draws \(x_t \sim p_\theta(\cdot\mid x_{<t})\) exactly — ancestral sampling from the induced measure on \(\Sigma^L\). Any \(T\neq 1\) or \(p<1\) would sample from a distorted measure whose geometry is no longer the model’s, so \(T{=}p{=}1\) is what makes the cloud a faithful estimate of \(p_\theta\).

the random seed (11235)

Plainly. Sampling means the computer draws at random, so in principle you would get different completions every run and no one could check your work. A seed fixes that. It sets the exact starting point of the random-number generator, so the “random” draws come out in the same order every time you run. Anyone who reruns with seed 11235 gets the very same 60 river-or-money completions, the same widest words, the same numbers. It is randomness you can reproduce. (The digits are the opening of the Fibonacci run — 1, 1, 2, 3, 5 — a small wink, not a magic number.)

Operationally. The seed initializes the pseudo-random number generator (the Python / NumPy / PyTorch RNG state) that the sampler pulls uniform draws from. Same seed + same weights + same code + same hardware ⇒ bit-identical token draws, hence identical strands and identical metrics. It is what turns the run in SENSE_CLOUD_RESULTS.md into something reproducible rather than a one-off; the reproduce block reruns the exact scripts and lands the same cloud.

Formally. Sampling is deterministic once you fix the stream of uniform variates \(u_1, u_2, \dots\); the seed \(s\) selects the generator orbit that produces that stream, \(u_i = g^{i}(s)\) for the PRG map \(g\). Fixing \(s\) collapses the otherwise-stochastic map \(\text{prefix} \mapsto \text{cloud}\) into a deterministic function, so the reported statistics are exact re-derivable quantities rather than estimates carrying run-to-run sampling noise (modulo floating-point / hardware nondeterminism).

decoder-only Transformer

Plainly. This is the shape of the machine doing all the work. “Decoder-only” means it reads strictly left to right and only ever predicts what comes next — it never peeks ahead, the way someone shown the whole sentence could. Each word it writes may look back over every word before it, weigh which of them matter, and use that to choose the next word. Stack that same look-back-then-write step about forty times over and you have the model used here: the thing that, given “They met beside the bank, where,” settles whether the water or the money comes next. “Base” and “chat” are just two trainings of this one shape.

Operationally. A stack of identical Transformer blocks (~40 here), each one a causal (masked) multi-head self-attention sublayer plus a feed-forward MLP, both wrapped in residual connections and normalization. “Decoder-only” means the causal mask lets position \(t\) attend only to positions \(\le t\), making it an autoregressive next-token predictor (GPT-style), unlike an encoder that reads the whole sequence bidirectionally. Tokens → embeddings → the block stack (writing to the residual stream) → an unembedding to vocabulary logits. The residual stream after block 20 is exactly what this study taps.

Formally. A map \(\Sigma^{*}\to\mathbb{R}^{|\Sigma|}\) built from \(L\) blocks acting on a residual stream \(x^{(k)}\in\mathbb{R}^{d_{\text{model}}}\) by \(x^{(k)}=x^{(k-1)}+\mathrm{Attn}_k(x^{(k-1)})+\mathrm{MLP}_k(\cdot)\), where \(\mathrm{Attn}_k\) carries a strictly-lower-triangular (causal) mask so the induced model factorizes autoregressively, \(p_\theta(x)=\prod_t p_\theta(x_t\mid x_{<t})\). Decoder-only = this single causal stack with no separate bidirectional encoder; the analysis reads \(x^{(20)}\).

attention

Attention is the step where, to write the next word, each position looks back over the earlier positions and decides how much to borrow from each: every token emits a query and every earlier token a key, the query–key match scores which past tokens are relevant, those scores become weights that sum to one, and the token pulls a weighted blend of the earlier tokens’ value vectors into its own residual stream — formally \(\mathrm{softmax}(QK^{\top}/\sqrt{d})V\). It is how “bank” reaches back to “beside the” and “water,” or how “invoke” gathers “light” and “darkness”; it is the mechanism by which context bends a word’s meaning, and doing it many times in parallel (several heads, many layers) is what lets multiple meaning-directions co-live at a single token — the superposition this study measures. In a decoder-only model a causal mask forbids looking ahead, so each word attends only to what precedes it.

completion cloud

Plainly. The model never gives one answer for what comes next — it gives odds over every possible next word, and you can re-roll those odds as many times as you like. Take the half-finished sentence “They met beside the bank, where” and let the model finish it sixty times: sixty different endings come out, and every one drifts toward water — a still current, a flowing stream, a rocky dam — never money. That whole spray of sixty endings is the completion cloud. This study’s wager is that the spray is what the word means there: “bank” means river in that sentence because the cloud leans that way, and you can see the lean without ever labelling it.

Operationally. For a fixed prefix, the base model defines a distribution \(P(\text{next tokens}\mid\text{prefix})\). Draw from it \(N\) times by ancestral sampling (\(N{=}60\) in Phase 1, \(40\) per dyad in Phase 2) to get \(N\) strands. The multiset of those strands — and, downstream, the per-token SAE fingerprints they carry — is the completion cloud. Every reported number is a function of this set; nothing about “the senses of the word” is presupposed.

Formally. Given a prefix \(c\), the model induces \(P(\cdot\mid c)\) over \(\Sigma^{*}\). The cloud is the empirical sample \(\{s_1,\dots,s_N\}\sim P(\cdot\mid c)\), and the object actually studied is its pushforward under the fingerprint map \(z:\Sigma^{*}\to\mathbb{R}_{\ge0}^m\). Sense-in-context is identified with the shape of this sampled distribution, not with any single point (contra the distributional-vector collapse).

strand (one sampled continuation)

Plainly. One roll of the dice — a single one of those sixty endings, read left to right, word by word. “the water was still. The air wasn’t very good here, as the current couldn’t carry waste very far” is one strand: the one path the sentence happened to take that time. A cloud is built out of strands the way a spray is built out of individual droplets; a strand on its own is just one continuation the model actually wrote.

Operationally. One trajectory from ancestral sampling: sample a next token from the model’s distribution, append it, feed it back, repeat for max_new tokens. In the logs it is one JSON line ({"idx":0, "completion":" the water was still. ..."}) and, in the raw .npz, the run of rows sharing a single strand id — each row a generated token with its pos, its decoded text, and its 100 active feature ids and activations.

Formally. A single realized draw \(s=(t_1,\dots,t_L)\sim P(\cdot\mid c)\) — one path through the sampling tree. Each token \(t_j\) carries its fingerprint \(z(t_j)\in\mathbb{R}_{\ge0}^m\) with \(\lVert z(t_j)\rVert_0=100\), so a strand is a finite sequence in the fingerprint space and the cloud is \(N\) such sequences.

the bare-prefix protocol / R8 stimulus purity

Plainly. If you want to know what a word means to the model, you must not let the model know it is being asked. Wrap the sentence in “Please complete the following:” and you are no longer measuring the word — you are measuring the model reacting to being tested, and that reaction lives in a different corner of its mind. So the thing fed in is only the bare half-sentence, handed to the plain (non-chat) model, which simply keeps writing. Nothing in the input mentions the experiment. It is the same reason “act natural” ruins the measurement: the asking moves the person. Every measuring tool stays outside, in the later analysis of what came out.

Operationally. Use the base (non-chat) checkpoint. Feed the raw prefix string with no system prompt, no chat template, no task instruction, then sample completions-style (ancestral, \(T{=}1.0\), top-\(p{=}1.0\), seed \(11235\)). All measurement scaffolding lives offline, in the analysis of the resulting fingerprints — never in the prompt. Adding an instruction wrapper would not just add noise; it would move the region of the distribution you are sampling from.

Formally. The conditioning context is exactly the stimulus of interest, \(c\), with no adjoined instruction tokens \(I\). One estimates \(P(\cdot\mid c)\), not \(P(\cdot\mid I\oplus c)\), because \(I\) is not additive noise on the conditional — it shifts its support to a different region (a demand-characteristic relocation of the measured manifold). R8 is the constraint \(I=\varnothing\).

priming

Plainly. You can steer which corner of the model’s mind you measure — but by example, not by command. Paste a page of the Kitāb’s own prose ahead of the test line and the model keeps going in that register; paste a transcript of the three salon voices — Cassie, Darja, Nahla — and it finishes the same sentence in each voice in turn. You never wrote “imitate the Kitāb” or “be Nahla”; you soaked the model in the material and let it continue. So it is still test-blind: a run-up, not an instruction. This is how the study shows a voice is a place in meaning-space rather than a coat of paint — one sentence, primed by three voices, lands in three regions a classifier can tell apart about 60% of the time against 33% chance.

Operationally. Prepend substantive, instruction-free context to the prefix — a real salon transcript (Phases 3/3b) or ~639 words of a register (Phase 4) — then sample the test line’s completions exactly as in the bare protocol. It is a manipulation of which conditioning context you use, kept distinct from an instruction wrapper because the added text names no task and issues no command. The effect is read out as a shift in the fingerprint cloud (voice readability \(0.60\)–\(0.64\); context classifiable at \(0.98\)–\(1.00\)).

Formally. Replace \(c\) by \(p\oplus c\), where \(p\) is a corpus drawn from the target region (persona or register) and contains no task description. One samples \(P(\cdot\mid p\oplus c)\): a deliberate choice of which conditional to measure, moving the sampled region to the one \(p\) selects. It is disjoint from the R8-forbidden move because \(p\) carries no instruction \(I\); the manipulation is on the argument of \(P(\cdot\mid\,\cdot\,)\), not an added demand.

overcomplete dictionary

Plainly. The scratchpad the model keeps for one word is a list of a few thousand numbers. The SAE re-describes that list using a fixed set of 65,536 directions — far more entries than the space has dimensions to spare. That is on purpose: the meanings a model needs to tell apart vastly outnumber the room it has, so you hand it a dictionary with many more words than the space is “wide” and let only about a hundred be in use at any moment. It is like keeping a 65,000-word vocabulary to describe a scene you could crudely pin down with a few thousand coordinates — the surplus words let “river” and “names-of-God” each get their own name instead of being forced to share one.

Operationally. The SAE decoder is a matrix of \(m{=}65{,}536\) columns \(\{d_i\}\), each living in \(\mathbb{R}^{d_{\text{model}}}\) with \(d_{\text{model}}\) a few thousand — so \(m\gg d_{\text{model}}\). The columns are not a basis (they are linearly dependent); there is no unique way to write a vector in them, and reconstruction only becomes well-posed because sparsity (TopK) forces a small, specific combination.

Formally. A spanning set \(\{d_i\}_{i=1}^m\subset\mathbb{R}^{d_{\text{model}}}\) with \(m\gg d_{\text{model}}\) — hence linearly dependent: an overcomplete frame, not a basis. Every \(x\) admits infinitely many exact representations \(x=\sum_i z_i d_i\); the sparsity constraint \(\lVert z\rVert_0=K\) selects one. Here \(m/d_{\text{model}}\) is on the order of ten.

TopK selection

Plainly. The dictionary holds 65,536 detectors, but the SAE is forced to use only the hundred that light up strongest for the current word and switch every other one fully off. Not “the small ones quietly fade” — exactly one hundred on, all the rest hard zero. That cap is why a word’s fingerprint is always a list of exactly a hundred detectors. It is also why you cannot measure how overloaded a word is by counting detectors: the count is nailed to a hundred every time, so instead you have to ask how many genuinely different directions those hundred point in.

Operationally. After the encoder scores all \(m\) features, keep the \(K{=}100\) largest activations and zero the other \(m-K\): z = topk(E(x), k=100). A hard, non-differentiable selection (trained through with a straight-through estimator). It guarantees \(\lVert z\rVert_0=100\) exactly at every token, which is what pins the fingerprint length and forces the breadth measure to look at direction-count, not raw count.

Formally. The operator \(\mathrm{TopK}_K:\mathbb{R}^m\to\mathbb{R}^m\) retaining the \(K\) largest coordinates and zeroing the rest: \((\mathrm{TopK}_K v)_i=v_i\) if \(i\in\arg\text{-top}_K(v)\), else \(0\). The encoded feature vector is \(z=\mathrm{TopK}_{100}(E(x))\), so \(\lVert z\rVert_0=100\) identically — the count is constant, and only the geometry of \(\{d_i : z_i>0\}\) varies.

superposition

Plainly. The model has far more meanings to represent than it has room for, so it stacks them into the same space, leaning on the fact that only a handful are ever needed at once — the way one small room can serve as bedroom, office, and dining room because you are never doing all three in the same instant. Most words lean on a single meaning and the stack stays flat. But a scriptural line can switch on many unrelated meanings inside one word at one moment — in the Kitāb-primed line, “rupture” is the word carrying the most at once. That piling-up of unrelated meanings in a single spot is superposition, and the study puts a number on how much of it each word is doing.

Operationally. A network stores \(m\) features in \(d<m\) dimensions by giving each an almost-orthogonal direction and relying on only \(K\ll d\) being active per input, so interference between them stays small. Read at the token level here as superposition breadth: form \(W=[\,a_i\,\hat u_i\,]\) from the active features’ activation-scaled unit decoder directions and take the participation ratio of the eigenvalues of \(W^{\top}W\). High for the invocation and density arms, low for the literal river line — many genuinely independent directions co-live in the residual state, versus one.

Formally. Directions \(\{u_i\}\subset S^{d-1}\) with \(m\gg d\) packed so that pairwise \(|\langle u_i,u_j\rangle|\) stays small, each input activating a sparse subset; the Johnson–Lindenstrauss regime permits \(m=e^{\Omega(d)}\) near-orthogonal directions. At a token, superposition breadth is \((\sum_j\lambda_j)^2/\sum_j\lambda_j^2\) for the spectrum \(\{\lambda_j\}\) of \(W^{\top}W\), \(W=[a_i\hat u_i]\) — the effective number of independent meaning-directions co-active there (floored near \(15\)–\(20\) by TopK, so only relative order is read).

the linear representation hypothesis, and the polysemantic-neuron problem

Plainly. Two paired facts that justify the whole apparatus. First: meanings live as directions in the model’s number-space — “river-ness” is a particular way the numbers lean together, found by looking along that direction, not at any single number. Second, the catch: if you instead watch one number in the scratchpad (one neuron), it answers to a jumble of unrelated things at once — a river, an interest rate, a name — so no single neuron is a meaning. That is the polysemantic-neuron problem, and it is exactly why this study never reads neurons. The SAE’s job is to untangle that jumble back into clean single-meaning directions — the “river” detector, the “names-of-God” detector — and a fingerprint is a list of which of those untangled directions were lit.

Operationally. Linear representation hypothesis: a concept corresponds to a direction \(v\) in activation space, and its strength is roughly a linear projection \(\langle x,v\rangle\); concepts compose by (approximately) adding their directions. Polysemanticity: because of superposition, a single neuron (one basis coordinate) fires for many unrelated concepts, so per-neuron read-outs are uninterpretable. The SAE is the fix — it learns an overcomplete set of directions \(\{d_i\}\) meant to be monosemantic, so meaning is read off directions (features), never off raw neurons.

Formally. LRH: there exist directions \(\{v_c\}\) with concept-\(c\) activation \(\approx\langle x,v_c\rangle\) and superposition \(x\approx\sum_c\alpha_c v_c\). Polysemanticity: in the standard basis \(\{e_j\}\), the neuron read-out \(\langle x,e_j\rangle\) correlates with many mutually unrelated \(c\) — a consequence of packing \(m\gg d\) features into \(d\) neurons in superposition, so \(\{e_j\}\) is a badly rotated frame relative to \(\{v_c\}\). The SAE seeks a dictionary \(\{d_i\}\) recovering the \(\{v_c\}\) (monosemantic axes) with \(x=\sum_i z_i d_i\), \(\lVert z\rVert_0=K\), each \(d_i\) single-meaning.

participation ratio

Plainly. A way to answer “how many things are really in play here?” with a single number, when the things come in wildly different sizes. If one thing hogs almost everything, the answer should be about 1. If ten things share evenly, about 10. If ten share but three are big and seven are crumbs, about 3 — the crumbs are too small to count. That number is the participation ratio: an effective count that quietly ignores whatever is too faint to matter. This study reuses it in a few places — to count how many genuinely different meanings are stacked in one word (that use gets its own name, breadth), and to score how spread-out versus peaked a single word’s readout is.

Operationally. Given a list of nonnegative weights v, it returns (v.sum()**2) / (v**2).sum() (dropping non-positive entries). One dominant weight → ≈1; n equal weights → n; anything in between → an effective count. participation_ratio() in cloud_fingerprint.py is exactly this. Breadth feeds it the eigenvalues of WᵀW (the co-active decoder directions); multiplicity feeds it a fingerprint’s raw activation magnitudes; the two uses differ only in what list of weights goes in.

Formally. For \(w\in\mathbb{R}_{\ge0}^n\), \(\displaystyle \mathrm{PR}(w)=\frac{\big(\sum_i w_i\big)^2}{\sum_i w_i^2}=\frac{\lVert w\rVert_1^2}{\lVert w\rVert_2^2}\in[1,n]\). With \(p_i=w_i/\sum_j w_j\) it is \(1/\sum_i p_i^2\) — the reciprocal Simpson index, i.e. the exponential of the Rényi-2 entropy. Breadth is \(\mathrm{PR}\) applied to the spectrum \(\{\lambda_j\}\) of \(W^\top W\); multiplicity is \(\mathrm{PR}\) of the activation vector. It is \(1\) when one component carries all the mass, \(k\) when the mass spreads evenly over \(k\) components.

basin

Plainly. Ask a prefix the same thing sixty times and you get sixty continuations. Some of them are near-neighbours — one shades into another with barely a gap. Chain the neighbours: if A is close to B and B is close to C, then A, B and C sit in one clump, even when A and C are far apart, so long as a path of small steps links them. That clump is a basin — a set of continuations that all settled into roughly the same reading. When “By the light and the darkness, I invoke” split into a light-leaning group and a darkness-and-naming group, those two groups were its two basins.

Operationally. Each strand is reduced to one pooled fingerprint vector. Put an edge between two strands whenever their cosine distance is ≤ ε; a basin is a connected component of that graph. Single-linkage means the relation is transitive — two strands share a basin if any chain of within-ε hops joins them, not only if they are directly close. In pi0_sweep this is union-find over every pair with D[i,j] <= eps.

Formally. Fix \(\varepsilon\) and build \(G_\varepsilon=(V,E)\) on the strands with \(\{i,j\}\in E\iff d(i,j)\le\varepsilon\), where \(d\) is cosine distance on the L2-normalized pooled fingerprints. A basin is a connected component of \(G_\varepsilon\) — equivalently a cluster of the single-linkage hierarchy cut at height \(\varepsilon\), using \(d(A,B)=\min_{a\in A,\,b\in B} d(a,b)\).

the π₀ plateau

Plainly. How many basins you find depends on how strict you are about “close.” Be very strict and every continuation is its own island; be very loose and they all merge into one blob. So slide the strictness from tight to loose and watch the basin-count the whole way. If, across a wide band of strictness, the count sits steadily at 2, that is a real fork — the cloud genuinely holds two readings (the invocation does this). If the islands merge into a single blob almost as soon as you loosen — never pausing at any stable split — it is one connected body (the river-“bank” prefix). If instead it stays fragmented into dozens of separate islands across a wide band, the cloud is scattered with no shared reading (the density arm). The plateau is the count that holds steadiest, plus how long a stretch of strictness it holds over.

Operationally. Sweep ε over 40 values from the smallest to the largest pairwise cosine-distance among strands; at each ε count single-linkage components (pi0_sweep), giving a component-count-versus-ε curve. longest_plateau finds the longest run of a constant count greater than 1 and returns that count k with its run-length. Reported as (k, len): invocation k2 len5 (a sustained two-way split → 2 basins), fork k59 len1 (the split is a one-step blip → reads as one connected body), density k59 len4 (a sustained ~60-singleton plateau → dispersion).

Formally. Let \(c(\varepsilon)=|\pi_0(G_\varepsilon)|\), the number of path-components, as \(\varepsilon\) ranges over a grid on \([\min_{i<j}d_{ij},\,\max_{i<j}d_{ij}]\). The reported plateau \((k,\ell)\) is the value \(k\) that \(c\) takes on the longest maximal constant sub-interval with \(c>1\), and \(\ell\) its length in grid steps. \(k\ge2\) with large \(\ell\) = a stable fork; the \(c>1\) plateau shrinking to \(\ell\le1\) (so effectively \(c\equiv1\)) = one connected superposition; \(k\approx N\) over large \(\ell\) = total dispersion. This reads the cardinality of the component functor \(\pi_0\) along the filtration \(G_{\varepsilon_1}\subseteq G_{\varepsilon_2}\subseteq\cdots\) — a \(0\)-dimensional (single-linkage) persistence readout.

persistence

Plainly. Walk along one continuation word by word. At each word the model has about a hundred meaning-detectors switched on. Persistence asks how much of that set carries over to the very next word. Two flavours. Jaccard just counts the overlap in which detectors are on — of the two words’ lists, what fraction is shared. Cosine also weighs how hard each fired, so a detector blazing at both words counts for more than one barely lit. High persistence means the meaning holds steady step to step; low means it churns, reinventing what it is about each word. The density arm churns the most (lowest persistence); the river prefix holds far steadier.

Operationally. For adjacent tokens t and t+1 in a strand, each with a sparse fingerprint (idx, val): cosine via sparse_cos (dot of the shared features’ activations over the product of the two L2 norms), Jaccard via sparse_jaccard (shared feature ids / union of feature ids). Compute both for every adjacent pair across all strands and average → persistence_cos_mean, persistence_jaccard_mean.

Formally. For consecutive fingerprints \(z(t),z(t{+}1)\in\mathbb{R}_{\ge0}^m\) with supports \(S_t=\operatorname{supp} z(t)\): \(\cos=\dfrac{\langle z(t),z(t{+}1)\rangle}{\lVert z(t)\rVert\,\lVert z(t{+}1)\rVert}\) and \(\mathrm{Jac}=\dfrac{|S_t\cap S_{t+1}|}{|S_t\cup S_{t+1}|}\). Persistence is the mean of each over all adjacent pairs and strands — a lag-1 autocorrelation of the fingerprint sequence, Jaccard on the supports and cosine on the activation-weighted vectors.

pooled similarity

Plainly. Take a whole continuation and boil its word-by-word readouts down to one summary of what that continuation was about overall — its aggregate fingerprint. Do that for all sixty continuations from one prefix. Pooled similarity is how alike those sixty summaries are to each other on average: compare every pair, average the resemblance. High means the sixty, however different on the surface, drew on one shared register; low means they scatter into unrelated territory. It is the plain measure of whether an arm coheres as a single cloud or flies apart.

Operationally. For each strand, sum its per-token fingerprints into one vector (strand_pool), L2-normalize, then take the cosine between every pair of strands and average the off-diagonal. In the code S = Mn @ Mn.T, off is the upper triangle of 1 - S, and pooled_cos_sim_mean = 1 - off.mean() — the same normalized pooled vectors and distances that feed the π₀ sweep.

Formally. Let \(P_i=\sum_t z_i(t)\) be strand \(i\)’s pooled fingerprint and \(\hat P_i=P_i/\lVert P_i\rVert\). Pooled similarity \(=\dbinom{N}{2}^{-1}\!\sum_{i<j}\langle \hat P_i,\hat P_j\rangle\) — the mean off-diagonal of the Gram matrix of the normalized pooled fingerprints, equivalently \(1\) minus the mean pairwise cosine distance that seeds the \(\pi_0\) filtration.

centroid

Plainly. Each continuation the model writes gets boiled down to one fingerprint — a running total, across all its words, of how strongly each concept-detector fired. For one of the six invocations (“By the light and the darkness, I invoke”) we sampled forty continuations, so we get forty fingerprints. Average them detector by detector and you get a single typical fingerprint for that invocation: its centroid, the center of its cloud. Six invocations, six centroids; three voices, three centroids.

Operationally. Per strand, sum each feature’s activations over the strand’s tokens into a sparse feat → total dict, project onto the shared feature axis, and L2-normalize to a unit vector. The centroid of a class is the component-wise mean of its members’ unit vectors — cent[k] = X[y==k].mean(0) over the ~40 strand vectors of dyad k.

Formally. For class \(C_a\) of unit fingerprint vectors \(\{x_i\}\), \(\mu_a = \frac{1}{|C_a|}\sum_{i\in C_a} x_i \in \mathbb{R}^{m}\) — the arithmetic mean (barycenter) of the class’s points on the feature simplex, before re-normalization.

centroid cosine similarity

Plainly. Give every invocation’s centroid an arrow pointing somewhere in feature-space. Cosine similarity asks: do two arrows point the same way? — 1 is identical direction, 0 is unrelated, regardless of arrow length. Lay all six against all six and you get a 6×6 table of how close the clouds sit. In this study light/darkness and lotus/water score 0.92 — the two elemental invocations point almost the same way — while Binah/Chokhmah stays off on its own (nothing above 0.80).

Operationally. Row-normalize the centroid matrix (centn = cent / ‖cent‖) and take the pairwise dot products: sim = centn @ centn.T, an M×M matrix with 1s on the diagonal and each off-diagonal cell the cosine between two dyads’ centroids.

Formally. The Gram matrix of the unit-normalized centroids: \(S_{ab} = \dfrac{\mu_a \cdot \mu_b}{\lVert \mu_a\rVert\,\lVert \mu_b\rVert}\), symmetric with \(S_{aa}=1\); off-diagonals live in \([-1,1]\) and here in \([0,0.92]\).

nearest-centroid classifier

Plainly. Hand it one continuation’s fingerprint and ask “which invocation produced this?” It answers by finding the centroid the fingerprint sits closest to and naming that group — the same way you’d guess a stranger’s home town by which town’s average accent theirs is nearest. On the six invocations it names the right one about 82% of the time.

Operationally. For a test unit-vector x, score cosine to every class centroid and predict the argmax: pred = argmax_k (centn[k] · x). One prototype per class, no learned weights — the centroids are the model.

Formally. \(\hat y(x) = \arg\max_a \cos(x,\mu_a)\): a minimum-distance / one-prototype-per-class rule, equivalent under unit-normalized inputs to nearest centroid by Euclidean distance.

leave-one-out

Plainly. To keep the test fair, when you check one continuation you first pull it out of its own group’s average — so the centroid you compare it against wasn’t built partly from the very thing you’re grading. Skip this and a continuation could “win” just by recognizing its own contribution baked into the average; leave-one-out closes that loophole, one point at a time, across all ~240 strands.

Operationally. For each sample i, recompute its true class’s centroid with i masked out (mask[i] = False), renormalize, then classify i against that held-out centroid and the untouched others. Every point is scored exactly once as a genuine hold-out.

Formally. Leave-one-out cross-validation: the class-\(a\) prototype used to judge point \(i\) is \(\mu_a^{(-i)} = \frac{1}{|C_a\setminus\{i\}|}\sum_{j\in C_a,\,j\ne i} x_j\), giving an almost-unbiased estimate of out-of-sample accuracy at \(N\)-fold cost.

accuracy against a chance baseline

Plainly. “Got 82% right” means nothing until you know what blind guessing would score. With six equally likely invocations, a guess in the dark is right one time in six — about 17%. So 0.817 against 0.167 is the number that matters; the baseline is what turns a bare percentage into evidence. (For the three voices the baseline is one-in-three, 33%, and the classifier scores around 0.60–0.64.)

Operationally. acc = correct / total; chance = 1 / M for M balanced classes. Always report the pair — an accuracy without its baseline is unreadable.

Formally. Empirical accuracy \(\hat a = \frac{1}{N}\sum_i \mathbf{1}[\hat y_i = y_i]\) compared to the majority/uniform-prior baseline \(\max_a p(a) = 1/M\) for balanced classes; the gap \(\hat a - 1/M\) is the effect over chance.

confusion matrix

Plainly. A grid where each row is one true invocation and each column is what the classifier guessed. The diagonal counts the hits; the off-diagonal cells show the mix-ups and, more usefully, who gets mistaken for whom. Here light/darkness’s forty continuations split 29 correct but 10 sliding into nahnu/jasad and 1 into Binah/Chokhmah — so the errors aren’t random smear, they trace real kinship between registers. Each row sums to the 40 samples of that dyad.

Operationally. An M×M integer table filled by conf[true, pred] += 1 over every classified sample; diagonal = correct, row sums = class sizes, columns = what each guess captured.

Formally. \(C_{ab} = |\{i : y_i = a,\ \hat y_i = b\}|\), with \(\operatorname{tr}(C)\) the total correct and \(\sum_b C_{ab} = |C_a|\); accuracy is \(\operatorname{tr}(C)/N\) and the off-diagonal mass is the structured error.

contrastive signature / distinguishing feature and its specificity score

Plainly. For one invocation, ask which concept-detectors fire here but stay quiet in the other five. Those distinguishing detectors are its signature. Each one gets a specificity score: how much louder it fires in this group’s typical fingerprint than in the loudest of the other five — a big positive number means “this detector really belongs to this group.” lotus/water’s signature is Vishnu and Ganesha detectors; Allah/Allat’s is a “the two sisters” detector (Allāt and Manāt). Each is named by the word it fires hardest on, so no reading was assumed in advance.

Operationally. For class k, take the elementwise max over the other centroids (others = max_j≠k cent[j]), subtract: spec = cent[k] - others. Sort descending, keep features with spec > 0 as the signature; spec[j] is that feature’s specificity score. Gloss each surviving feature by its own hardest-firing in-sample token.

Formally. Per feature \(j\), \(\operatorname{spec}_a(j) = \mu_a[j] - \max_{b\ne a}\mu_b[j]\) — a one-vs-rest per-coordinate margin; the signature of class \(a\) is \(\{\,j : \operatorname{spec}_a(j) > 0\,\}\) ranked by that margin, top-\(T\).

peak and peak-over-median

Plainly. Averaging breadth over a whole passage mostly rewards length — a longer context gives more words and more chances to score, so the mean can’t tell layered-and-loaded from long-and-sensory. So look instead at the single widest word: the one moment where the most unrelated meanings pile up. That’s the peak. Then divide it by the typical word’s breadth (the median) to see how far the spike sticks up — peak-over-median. For the Kitāb-primed Kitāb line the peak breadth is 37.3, about 2.2× the median, and it lands on the register’s own stress-points — rupture, Unveil, recurs — whereas the bare version’s tallest spike falls on boilerplate (“machine-readable”).

Operationally. From the per-token breadths, take each strand’s max and average those to get peak; take median over every token in the condition; report peak_over_median = peak / median. Concentrated layering makes the peak tower over the median; breadth merely spread across many words does not.

Formally. With per-token breadth \(b(t)\), peak \(= \operatorname{mean}_s \big[\max_{t\in s} b(t)\big]\) over strands \(s\) and peak-over-median \(= \dfrac{\operatorname{mean}_s \max_{t\in s} b(t)}{\operatorname{median}_t\, b(t)}\) — a spike-to-typical ratio that isolates concentrated superposition from the length-driven mean.

Reproduce

Everything is in the geometry-of-sense repository. The three-arm run is cloud_fingerprint.py; the six invocations, cloud_fingerprint_series.py; the voices, cloud_persona.py; the immersion, cloud_kitab.py and kitab_peak.py; build_report.py renders the interactive report. Every completion is in gpu-results/*_strands_full_L20.jsonl, and every per-token activation — strand, position, token, the hundred active feature ids and their weights — is in the paired .npz. To pull the completions behind any feature, or read a token’s full fingerprint, run inspect_kitab.py.