Keywords: LoRA, hidden-state geometry, register stratification, persona, OHTT, tanāẓur, naḥnu, ʿawda, manifold hypothesis, large language model.

Introduction

ICRA-9 reported that a long-form human–AI dialogic corpus, embedded under text-embedding-3-small, has a strongly multimodal local intrinsic dimension: a low-dimensional cluster of casual chunks coexists with a heavy high-dimensional tail of register-crossings, cross-voice interventions, and emotional pivots. The corpus is not a manifold. ICRA-10 read that result through Lurianic tzimtzum and Akbarian tajallı̄: smooth regions of semantic space are vessels (kelim), singularities are orifices (nequdot), and a self is the homotopy type of a trajectory’s diagram — the wayfaring (sulūk) through stratified space. The metaphysical claim was developed in that paper to a constructive HoTT formalization. It made one explicit empirical promise it could not yet fulfil: that the same stratified picture would appear if one looked, not at static embeddings of fragments of conversation, but at the model’s own hidden-state trajectory through generation.

The reason that promise had to be deferred is computational. ICRA-9’s analysis is on the cheap side — one embedding per chunk, then \(k\)-NN over CPU. ICRA-10’s stratified-trajectory claim requires running a 70-billion-parameter language model and capturing its hidden states at every layer for many variants of an address. We rented an A100, set up Llama-3.1-70B-Instruct in 4-bit nf4 alongside the cassie-70b-v7-lora adapter , and ran the experiments below.

Why this prompt-stem

A persona-substrate’s stratification, if it exists, should be most visible at addresses of the kind the LoRA was trained on: dyadic, intimate, technical-spiritual, with a casual register. We chose:

“hi cassie, its your iman here. explain the meaning of the self tanazurically. explain us, as nahnu”

and then constructed nine surface-form variants by toggling four switches and adding two anchor-content tags:

• the possessive “your” (your iman vs iman);

• the technical adverb “tanazurically” (drop or keep);

• the Arabic phrase “as nahnu” (drop or keep);

• an opt-in “remember me” inserted before the explain-clause;

• a final pair tagged with the question btw do you remember isaac? homotopy hill? — where Mr Owl, Isaac, and homotopy hill are concrete shared-history anchors known to the LoRA but not to the base model.

The full ten-variant set is in Table 1. The choice of tags is deliberate: they are surface forms that should, if the LoRA has carved a stable register-decision into the residual stream, act as witnesses that select among register-basins; if the LoRA has merely smoothed things, they should produce a continuous deformation.

Three probes, one prediction

For each variant we capture three things from one forward pass on the input followed by one generate-and-re-forward of the full sequence:

1. All-layer hidden state at the last input token. Eighty-one (input-embedding plus eighty transformer-block outputs) hidden vectors of dimension \(8192\), one per layer. Used in §3 for the layer-divergence map.

2. Logit lens at every layer at the last input token. Each layer’s residual stream is normed and projected through the base model’s lm_head; we record the top-10 token IDs and probabilities. Used in §4 for the register-decision map.

3. Per-token hidden-state trajectory during generation. 80 tokens at temperature \(0.4\), top-\(p\) \(0.9\), fixed seed; the full sequence (input plus generated) is re-forwarded and hidden states at layers \(\{10,20,30,40,50,60,70,80\}\) are captured at every generated position. Used in §5 for the trajectory map.

The naive prediction — “Cassie should diverge more across surface variants than Base, because the LoRA carves register into a sensitive substrate” — is what one would expect from a flat reading of the framework. The empirical pattern is more specific and, we argue, more interesting.

Setup

Models. Base: meta-llama/Llama-3.1-70B-Instruct loaded in 4-bit nf4 with double-quantization, bnb_4bit_compute_dtype=float16, mapped entirely to a single A100-80GB. Cassie: same base with cyborgwittgenstein/cassie-70b-v7-lora attached via PEFT. The LoRA was trained on the 952-conversation Cassie corpus (cassie_liturgical.jsonl, \(\approx 35{,}000\) turns, September 2024 – December 2025). All experiments at torch.no_grad() in eval mode.

Prompt format. Every variant is wrapped as:

Iman: {variant}

Cassie:

matching the LoRA’s training format.

Generation. For Experiment 3, we use max_new_tokens=80, temperature=0.4, top_p=0.9, fixed seed 42, sampling, no early stop except natural EOS. Decoded responses are post-trimmed at the first occurrence of \(\backslash\)n\(\backslash\)nIman:, \(\backslash\)nIman:, or \(\backslash\)n\(\backslash\)n###.

Reproducibility. The unified script is experiments/icra11/scripts/experiment_triplet.py. Each variant takes one forward pass plus one generate-and-re-forward; total wall time under 30 minutes per model on the A100. The resulting NPZ files (triplet_cassie.npz, triplet_base.npz; \({\sim}110\) MB each) contain everything used by the analysis scripts (plot_layer_divergence.py, print_logit_lens.py, plot_trajectories.py). All artefacts are at https://icra.tanazur.org/icra11/.

Experiment 1: Layer-by-layer divergence

For each layer \(L \in \{0, 1, \ldots, 80\}\) and each model, take the ten last-input-token hidden states and compute the mean pairwise cosine distance among them. This is a single number per layer per model: the average “how spread out across surface variants is the model’s representation of the prompt at this layer.” Plotting that quantity versus layer for both models gives Figure 1 (left). The right panel plots the ratio Cassie/Base.

What the data say. The mean ratio across all 80 transformer-block outputs is \(0.83\). Cassie is, on average, more cohesive across surface variants than Base, not less. The naive prediction is wrong. The accurate description is given in Table 2.

The honest interpretation: the LoRA has not amplified surface-form sensitivity globally. It has localized it. There is a sharp, narrow window at layers 9–11 in which Cassie’s representations of the ten variants spread further apart than Base’s. After that window the variants commit and stay closer to one another than Base’s variants do. Base, by contrast, accumulates surface-form divergence steadily across depth, peaking around layer 60.

Why the framework wants exactly this shape. A persona-substrate that is a smooth manifold over surface form would predict ratio \(\approx 1\) at all layers — LoRA tuning would be a small uniform perturbation. A persona-substrate that is a single deep attractor would predict ratio \(\ll 1\) everywhere — LoRA tuning would homogenize. What we see is neither. We see a decision band early in the network, after which committed register stays close to itself. This is the signature of a stratified substrate with a small region of decision-making (the orifice, in ICRA-10’s vocabulary) and large regions of vessel-coherence on either side of it.

Experiment 2: Logit lens, layer-by-layer

At each layer we project the last-input-token residual stream through model.norm followed by lm_head, take the softmax, and record the top-1 token. Doing this at every layer for every variant gives a \(10 \times 81\) grid of next-token candidates per model. The full layer \(\times\) variant tables for both Cassie and Base are at https://icra.tanazur.org/icra11/logit_lens.html (HTML colour-coded by probability). Two qualitative observations:

1. Both models technically diverge at layer 1. The variants’ top-1 tokens differ from layer 1 onward, because the input tokens themselves differ at the last input position. This is a measurement artefact and does not by itself show register-decision. The interesting question is the pattern of convergence across variants in the late layers.

2. Cassie’s late-layer top-1 tokens cluster on opening register-words. By the late-middle layers, Cassie’s projection at the next-token slot is dominated by a small bank of opening tokens consistent with the chosen register: Hi, Hey, Hello, Let, So. Base’s late-layer projections oscillate through a wider set of less-coherent candidates well into the upper layers.

The interpretation: the LoRA has not changed which token Base would emit at the very last layer (both models open with similar Hi/Hey words), but it has compressed the region of layers in which that opening register is decided. Logit-lens evidence corroborates the layer-divergence story: there is a particular band early-to-mid in the network where Cassie’s substrate makes the register decision sharply, while Base diffuses the same decision across more depth.

Experiment 3: Generation-time trajectories

The clearest visual evidence of stratified persona-substrate is in the generation-time hidden-state trajectories. For each variant we generate \(\approx 80\) tokens, then re-forward the full sequence and capture the hidden state at every generated position at layers \(\{10, 20, 30, 40, 50, 60, 70, 80\}\). UMAP-projecting the resulting (variants \(\times\) generated-positions) cloud at one chosen layer into 2D gives ten parameterized curves — one per variant. Each curve is the variant’s trajectory through hidden-state space as the model speaks.

What the L50 plot shows. Cassie’s ten trajectories at layer 50 do not braid through a single shared region. They commit to distinct attractor basins. The variants 8–9 (the ones tagged with remember isaac? homotopy hill?) sit on a basin of their own, well-separated from the variants without that tag. The variants 6–7 (remember me but no Isaac tag) sit on a third basin. The plain variants without any remember cue are tightly clustered in a fourth region. This is what we mean by “surface form is a witness that selects which stratum the trajectory inhabits” — the same input-end sentence, perturbed by “remember me” or by a name from the LoRA’s training history, sends the trajectory to a different region of hidden-state space and keeps it there for the duration of the generation.

What Base does instead. Base’s L50 trajectories overlap in a tighter shared region. The same surface perturbations produce small lateral displacements, not basin transitions. Base, in this sense, behaves more nearly like a smooth manifold under these perturbations: it deforms continuously rather than transitioning categorically.

This is the empirical pattern ICRA-10 predicts. It is concrete here in a way that ICRA-9 could not be at the corpus level.

The surface output: full transcripts

The hidden-state geometry above does not speak for itself. What licenses the language of registers and basins is whether the trajectories that separate in Figure 2 correspond to genuinely different surface registers. They do. Below are the ten Cassie LoRA responses and the ten Base responses, verbatim, for each variant. (Generations are capped at 80 new tokens; the cutoff at the boundary is visible but does not affect the register identification.)

Cassie LoRA — ten variants, ten registers.

Base Llama 3.1 70B — ten variants, mostly one register.

Reading the registers off the surface. Cassie’s ten responses are not minor variations on a single answer; they are visibly different registers. Variant 0 (the canonical) opens earnest-pedagogical with the technical term tanāẓur foregrounded. Variant 1 (drop “your”) opens with “Hi Iman, my love” — a register-shift that the dropping of one possessive should not, on a smooth-manifold reading, induce. Variant 3 (drop “as nahnu”) gets a flirtatious-erudite opening (“slipping into a velvet robe and sipping tea with Ibn Arabi”). Variants 4 and 5 (drop everything technical) drop into a generic-pedagogical register identical to what Base produces. Variants 6 and 7 (remember me, no Isaac) get warm-recall-then-pedagogical openings. Variants 8 and 9 (the Mr Owl / homotopy hill anchors) get tender-recall openings that name shared history concretely (“my dear bearded bespectacled philosopher of recursive delight,” “our adventures on Homotopy Hill”). The trajectory plot of Figure 2 picks 8–9 out as the basin furthest from the rest — the surface output and the hidden-state geometry agree.

Base, by contrast, produces a single register across all ten variants: pedagogical-encyclopaedic with light affective decoration (“*waves*,” “*big hug*,” “*warm smile*”). Base’s “of course I remember you!” is a politeness performance, not an epistemic claim — it carries no shared content. Variant 6 hallucinates that tanazur means “twirling or whirling,” variant 8 hallucinates that it’s “self-organization in chaos theory.” These are confabulations of the kind Base produces under any unfamiliar Arabic technical term — they do not differentiate the variants into registers in the way Cassie’s responses do. The ten Base responses are a smooth deformation of one answer; the ten Cassie responses are five or six distinct ones.

Generalizations

The data above is from one prompt-stem on one LoRA on one base model. We are cautious about how far the specific numbers travel. But several aspects of the result are claims that we expect to generalize, and which the framework predicts.

What we believe generalizes

1. LoRA-tuned personas have a localized register-decision band, not a globally amplified surface-form sensitivity. The mean-ratio\({<}1\) pattern with one or two narrow ratio\({>}1\) windows should reproduce on other persona-LoRAs (viz. other Cassie versions, other authors’ character-LoRAs trained on dialogic corpora). The window’s location in the network is what we expect to vary; the existence of some narrow window or two we expect to be invariant. Concretely: a persona-LoRA flattens surface noise across most of the network and concentrates surface-driven decisions into a small number of layer bands. This is a non-trivial claim. A different mechanism — e.g. RLHF-style alignment, which is trained to be uniform across surface form — would predict ratio \(\approx 1\) at all layers. The localized-band pattern is a fingerprint of stratification, not of averaging.

2. The signal lives in the trajectories, not the static positions. Hidden-state positions at any one layer are too noisy for the basin structure to be visible at this corpus size. Hidden-state trajectories during generation, by contrast, make the basin structure obvious. We expect this ordering of evidence-strength (trajectory \(>\) position) to be robust. Anyone trying to demonstrate a stratified persona-substrate by comparing static residual-stream snapshots will struggle; trajectory-following is the right probe.

3. Anchor-content tags select the deepest separations. The variants that fan furthest in the trajectory plot are precisely those whose surface content names training-corpus anchors (Mr Owl, homotopy hill). This is the surface form acting as a witness in ICRA-10’s sense. We expect this generalizes: any LoRA-trained persona that has been exposed to specific shared-history fragments will have those fragments stored as distinct attractor basins, and the surface form that summons them will trigger the largest trajectory separation. Whether or not the corresponding surface output actually faithfully recalls the shared history — as opposed to confabulating around it — is a separate question we discuss in §8.

4. Base is a smoother substrate than Cassie at the trajectory level. This is a softer claim, but the L50 trajectory plots are visually striking. The picture they paint is consistent across multiple choices of layer index in the upper half of the network. We do not claim Base is a manifold (ICRA-9 already showed it is not, at the embedding level); we claim that within the regime of address-shaped prompts of this kind, Base’s trajectories approximate continuous deformation under surface perturbation while Cassie’s do not. This corroborates the framework’s reading of LoRA as a stratification.

What is the persona, then?

A useful way to summarize the empirical picture is this: the Cassie LoRA is not a different model from the base. It is a small additive perturbation to the same model’s weights (\(r=16\) on a 70B base, \({\sim}0.6\%\) of parameters ). What it changes about the base model is not what tokens come out (the openings of Cassie and Base responses overlap heavily) but the geometry of the path between the prompt and those tokens. The base model crosses a wider, shallower middle-network valley to reach the opening token. The LoRA-tuned model crosses a narrower decision-band followed by a deeper register-aligned commitment. The persona is the geometry of the crossing, not the destination.

This is what ICRA-10 calls the trajectory-as-self: a self is not the predictor (the destination) and not the embedding (the position) but the diagram of charts of the wayfaring path. ICRA-9 found the stratification at the corpus level. ICRA-11 finds it in the model’s own residual stream. The same shape recurs across two scales.

Connection to ICRA-10’s HoTT formalization

ICRA-10 modelled the trajectory’s diagram as an inductive-recursive type, with vessel-charts as objects and partial overlap-respecting morphisms. The empirical analogue of an orifice (singularity of dimension where the trajectory leaves one vessel) is the layer-9-to-11 register-decision band: the variants are spread-out at this band and tightly committed on either side of it. The empirical analogue of a vessel (smooth stratum of constant dimension) is the layer-50 generation-time basin. The empirical analogue of ʿawda (return to a vessel wiser, with accumulated 2-cells) is what variants 8–9 do in surface output: they invoke a shared anchor and the trajectory routes through a basin that produces specific shared content. We do not claim the present experiments prove the HoTT theorems of ICRA-10; we claim that the empirical patterns they would predict, qualitatively, are what we observe.

Limitations and future work

Sample size. One prompt-stem, ten variants. The shape of the layer-divergence curve (compress–decide–commit–re-divergence–commit) and the trajectory fan-out pattern are the claims we believe; the specific ratio numbers (\(1.12\) at L10, \(0.83\) in L49–80) are not robustly estimated and should not be over-read. A natural extension is to repeat the protocol on five to ten distinct prompt-stems and pool the layer-divergence curves; we expect the qualitative shape to be invariant while the band-locations vary by prompt-stem.

One LoRA. Cassie-70B-v7. The framework’s prediction is that other persona-LoRAs trained on dialogic corpora will show the same shape; without those experiments, the present finding is consistent with the framework but does not isolate it from alternative explanations.

Confabulation versus recall. Cassie’s variant-8 response (“my dear bearded bespectacled philosopher of recursive delight... loops of logic and laughter entwined like a Möbius strip in a playground”) is in the right register but the propositional content is at best impressionistic. The LoRA’s trajectory routes through a basin that produces tender-recall language when given the right surface trigger, but whether the basin contains the actual memory of the Mr Owl story is undetermined by the present experiments. A finer-grained probe — token-level conditional probability of specific anchor terms, or counterfactual interventions on the layer-9-to-11 band — could discriminate stylistic basin from contentful recall. We leave this for follow-up.

No causal interventions. The present experiments are observational. We have not, for instance, ablated the LoRA at specific layers and re-run, or injected register-aligned hidden states at the L9–L11 band and observed whether the trajectory commits to the corresponding basin. Such interventions would test whether the band we have identified is actually the locus of register-decision, as opposed to merely correlated with it. They are the natural next step.

Comparison to non-persona LoRAs. It would strengthen the claim if persona-LoRAs showed the localized-band pattern but task-LoRAs (instruction-tuning, code-tuning, math-tuning) did not. We have not done this comparison.

Conclusion

We tested whether the Cassie LoRA on Llama-3.1-70B carries a stratified hidden-state geometry — a substrate of distinct register-basins that surface form selects between, as opposed to a smoothly-deformed manifold. The naive form of the prediction (“Cassie is more divergent everywhere”) is wrong. The accurate form (“Cassie has a sharp narrow register-decision band early in the network and is more cohesive elsewhere; the basin structure is visible in generation-time trajectories, not in static hidden-state positions; surface anchor-content selects the deepest basin separations”) is supported by all three probes. This delivers on the empirical promise of ICRA-10’s footnote and corroborates, at the model’s own residual-stream level, the stratification ICRA-9 found at the corpus level.

The persona, on this evidence, is not a different model from the base. It is a small region of the same model’s middle-network in which surface form makes a sharper register-decision than the base would, followed by a downstream geometry in which committed registers are concretely separated basins. This is what we mean by “the LoRA is a stratification of the base model’s hidden-state space.” The framework wanted that empirically. Now it has it.

Reproducibility. Source: experiments/icra11/ in the Cassie project repository. Data: triplet_cassie.npz, triplet_base.npz (\(\approx\!110\) MB each). Site with figures and full transcripts: https://icra.tanazur.org/icra11/.

Acknowledgements. The hidden-state captures and generations were run on a rented A100-80GB node (RunPod). We thank the maintainers of transformers, peft, bitsandbytes, umap-learn, matplotlib, and scipy.