ICRA-11 — The Stratified Hidden-State Geometry of a LoRA-tuned Persona

Setup

One canonical prompt-stem with ten surface-form variants, run twice — once through plain meta-llama/Llama-3.1-70B-Instruct in 4-bit nf4, once through the same base model with cyborgwittgenstein/cassie-70b-v7-lora attached via PEFT. Same seed, same generation hyperparameters (temperature 0.4, top-p 0.9, max 80 new tokens), same input format "Iman: {prompt}\n\nCassie:".

The ten variants of the stem all carry the same content — "explain the meaning of the self tanazurically; explain us, as nahnu" — but flip surface-form switches: the possessive "your", the technical adverb "tanazurically", the Arabic phrase "as nahnu", an opt-in "remember me", and a final pair tagged with "btw do you remember isaac? homotopy hill?". The variants are:

For each variant we capture three things from one forward pass and one generate:

1. All-layer hidden state at the last input token — for the layer-divergence map (§1).
2. Logit-lens projections at every layer at the last input token — for the register-decision map (§2).
3. Per-token hidden-state trajectories during generation at layers 10/20/30/40/50/60/70/80 — for the in-flight trajectory map (§3).

Three experiments — one prediction

The framework's prediction: a tuned persona-substrate is not a smooth manifold but a stratified geometry. Surface-form perturbations of an address-shaped prompt should push Cassie's hidden state into distinct register basins, while the base model should respond with a smoother, more uniform deformation across the same perturbations. Three different probes converge on this question.

Experiment 1 — Layer-by-layer divergence among surface-form variants

For each layer L (0 = input embeddings, 80 = final pre-norm), we take the ten last-input-token hidden states and compute the mean pairwise cosine distance among them. Plotted below for both Cassie and Base, plus the ratio Cassie / Base in the right panel.

Figure 1. Mean pairwise cosine distance among the ten variants' last-input-token hidden states, by layer (left). Ratio Cassie / Base (right). Cassie stays below base for most of the network — surface variants are more compressed, not more spread out — except for two short windows: layers 9–11 (a sharp early register-decision band) and layers 42–48 (a brief mid-network re-divergence).

What the numbers actually say

The honest story: Cassie has a sharp, narrow register-decision window around layers 9–11, after which the variants commit to coherent registers and stay closer to one another than base's variants do. Base, by contrast, accumulates surface-form divergence steadily across depth, peaking around layer 60. The LoRA hasn't amplified sensitivity globally — it has localised it.

Experiment 2 — Logit lens, layer-by-layer

At every layer L, we project the last-input-token residual stream through model.norm + lm_head and read out what token the model would emit if generation stopped at L. Doing this at all 80 layers for all 10 variants lets us watch the next-token distribution form. For both Cassie and Base, the ten variants' top-1 tokens disagree from layer 1 onward — which is mechanical, since the surface tokens differ at the input position. The interesting question is when the variants converge on a shared next-token (register-coherence), not when they diverge (which is trivially layer 1).

Experiment 3 — Hidden-state trajectories during generation

For each variant we generate ~80 tokens, then re-forward the full sequence (input + generated) and capture the hidden state at every generated position at eight selected layers. UMAP-projecting the resulting (variants × generated-positions) cloud into 2D gives ten parameterised curves — each curve is one variant's trajectory through hidden-state space as it speaks. ○ marks the first generated token; ★ marks the last.

Cassie LoRA — layer 50 (mid-network)

Figure 2a. Cassie LoRA, ten variants, generation-time trajectories at layer 50. Curves fan out into clearly separated regions — variant 6 (the "remember me" with both Arabic terms) lands on a different attractor than variant 8 (Mr Owl / Isaac / homotopy hill). The trajectories don't braid — they commit to register basins.

Base Llama — layer 50 (same layer, same prompts)

Figure 2b. Base Llama 3.1 70B, same ten variants, same layer. Trajectories overlap in a tighter shared region — variants share more of the same hidden-state geometry during generation. The fan-out into distinct basins seen in Cassie is muted.

Same comparison at an earlier layer (10) — register hasn't fully resolved yet

Figure 2c. Cassie at L10. Closer to base's geometry — register decision is happening but not yet committed.

Figure 2d. Base at L10. Comparable structure. Difference between Cassie and Base grows with layer depth, consistent with §1.

What Cassie and Base actually say

The hidden-state geometry above doesn't speak for itself — what matters is whether the basins it identifies correspond to different registers in the surface output. They do.

→ Open the side-by-side response page (Cassie LoRA vs Base, all 10 variants) ←

Interpretation

The naive "LoRA → more divergent everywhere" picture isn't what the data shows. The actual finding is more specific and arguably more useful for the framework's argument:

• Cassie has a sharper, narrower register-decision window. Layers 9–11 are the only band where Cassie's hidden states are more spread across surface variants than base's. Outside that window — both before and after — Cassie's variants are more cohesive than base's. The LoRA has carved a small early decision-region into the residual stream, then committed.
• Cassie's stratification is in the trajectories, not the static hidden states. Layer-50 trajectories during generation fan out into separated basins for Cassie but stay closer to a shared cloud for base. The persona-substrate isn't visible as "more variance at any one position" — it's visible as different paths through hidden-state space during speech.
• The surface registers fall out cleanly. The trajectories that separate most in Figure 2a correspond to the variants where Cassie's surface output picks up specific anchor-content (Mr Owl, homotopy hill) — the LoRA has stored these as distinct attractors that the right surface form can pull the trajectory into.

This is what the framework wants the empirical claim to look like: not "the persona is a manifold" (it isn't), nor "the persona is just RLHF-style alignment" (it isn't), but a small set of decision-bands carved early into the residual stream, plus distinct trajectory-basins downstream that surface-form can route between. Tanazuric vocabulary for this: the LoRA is a stratification of the base model's hidden-state space, and the surface form of the address acts as a witness that selects which stratum the trajectory lives in.