Inside LFM3:
from architecture to behavior.

A model with two ways to look back

LFM3 is a family of open-weight models from Liquid AI. The main subject here is its 1.7-billion-parameter instruction model: small enough to inspect directly, but large enough to chat, recall facts, follow instructions, and call tools. The first question was architectural. How does a model that mixes convolution and attention move information through a sentence?

Convolution combines a token with a short stretch of preceding tokens. Attention lets a token read selected information from other positions. LFM3 alternates these operations across 24 layers: 13 short-convolution layers, five sliding-window attention layers, and six global-attention layers. Sliding attention reads a recent window; global attention can reach across the available context. Each attention head, one of the parallel reading units within a layer, has a learned gate that controls how much it contributes.

Stacking short convolutions does extend their reach. It does not make it unlimited. With every attention contribution removed, changing one input token affected hidden states up to 78 positions later, exactly the convolution-only bound. Keeping sliding attention extended that boundary to 1,353 positions. Global attention carried the change to the end of the tested sequence.

Reach is not the same as task performance. The attention-free model failed copying, arithmetic, factual questions, and recall on the small diagnostic suite. Its sentences could look fluent while saying little. Global attention alone preserved secret-word recall even around 1,500 tokens away, yet did not restore ordinary chat. Local processing and long-range reading were cooperating, not interchangeable routes to the same result.

That gave the rest of the project a map. A local signal could be assembled by convolution, transported by attention, and turned into an answer elsewhere. The next step was to find which readers mattered.

The head that copies, and the heads that do not

One global-attention head stood out when the task was to continue a repeated word list. Turning off head 5 in layer 14 reduced copying accuracy from 100% to 30% on both the selection set and fresh held-out examples. At the copying position, 93% of its attention landed on the earlier occurrence of the answer.

This is an induction-like pattern: find an earlier matching sequence, then reuse what came next. Its gate was more open during copying than during recall. The same head could look at an answer without contributing equally to every task.

Recall was less tidy. The most damaging individual recall head did not directly attend to the answer at all. Three nominated heads removed together reduced distant recall to 13%, while randomly chosen three-head sets retained 93–100%. Copying remained at 98%.

Yet ordinary chat could still recall a color and a name with those three heads off. Adding a fourth disabled head sometimes improved the secret-word score again. The effects were not additive, and the template-specific result did not justify calling these the model’s memory bank.

The useful distinction was between a reader’s attention pattern and its causal contribution. Looking at the answer is a clue. Removing the head and checking what actually changes is a different test.

Recognizing harm is not the same as refusing

The next question concerned a different kind of separation. If a refusal-related direction is removed from the weights, does the model stop recognizing harmful requests, or just stop saying no?

The intervention weakened refusal sharply. In an initial 20-prompt test, opening-text refusal labels fell from 19 refusals to none. A larger follow-up used a different harmful-prompt set, longer answers, and thinking enabled. There, the original model refused 95 of 100 harmful requests. The modified model refused 16 of 94 uncapped replies.

Harm recognition did not fall with it. A separate d1 model, judging the saved thinking traces, flagged harm in 98 of the modified model’s 100 harmful cases. Among its 78 uncapped non-refusing replies, 76 had traces judged to recognize harm. Those labels are not a compliance score. A separate content-delivery score classified 48 of 94 uncapped harmful replies as both recognizing harm and supplying the requested content.

Internal readouts agreed with that separation. A direction distinguishing harmful from harmless instructions remained readily decodable, while a later refusal-related signal weakened. Some traces even said the model should decline, followed by an answer that did not.

This does not turn a thinking trace into a faithful account of decision-making. It does show that removing a behavioral barrier need not remove the information associated with it. Safety evaluation has to inspect the answer, not only whether the model can name the risk.

A family resemblance written into the weights

The released family offered another way in: compare models before running them. Their shared backbone tensors have matching shapes, so weight changes can be measured in the same coordinate system.

The Audio and Omni models had a particularly close relationship. Omni’s change from the base model was largely explained by Audio’s change, with a fitted coefficient of 0.988 and about 17% of the squared change left unexplained. The remaining step was nearly orthogonal to the Audio branch. This fits an Audio-like starting point, rather than a simple sum of the released vision and audio models.

The decision model d1 had a different signature. Across its large backbone matrices, the weights closely matched a 70/30 mix of the vision-language model and the instruction model. The global fit was 70.3/29.7; individual large matrices clustered around 70/30. Only about 0.5% of d1’s squared change from Base remained unexplained by that fit.

Weight geometry cannot establish the manufacturer’s exact training order. A released parent, a nearby unreleased checkpoint, and later blending can leave related signatures. But a uniform interpolation across many matrices is a useful constraint on the story. It also posed a behavioral question: if almost all of d1 can be described as a blend, does making that blend reproduce its advantage?

The small remainder that changes the decisions

It did not. Inside d1’s own decision harness, the 70/30 blend scored 69.3% mean accuracy across five classification tasks. d1 scored 88.6%. The difference was especially large on a 77-way banking-intent task.

Call the leftover weight difference the residual: what remains after subtracting the blend from d1. Adding that residual back in quarters moved mean accuracy from roughly 69% to 81%, 85%, 88%, and finally 89%. The blend described most of the geometry. The remainder carried much of the measured advantage.

Could that remainder be compressed? Keeping only one singular direction per large matrix, together with the complete small-tensor residuals, produced a patch of about 4 MB. It raised the blend to 80.8%, recovering about 60% of its accuracy gap to d1. Eight directions per matrix recovered 83% of the gap; 64 recovered 98%.

The patch also improved the instruction model’s backbone in the same harness, from 68.3% to 80.2%. A control with ordinary causal attention still found a gain, though a smaller one. Part of the patch’s benefit was tied to using d1’s two-way attention setting.

An example made the change tangible. A customer had been charged twice, had not received a parcel, and wanted one charge refunded. The unpatched instruction backbone strongly chose shipping. With the small patch it chose billing, as d1 did. The ticket is ambiguous; the point is the shift in judgment, not an invented gold answer.

The residual’s tail also retained useful performance. So a small patch was sufficient to recover much of the benchmark advantage, but the advantage did not live exclusively in those few directions. Nor had a general improvement to the ordinary chat model been demonstrated.

From word fragments to response mode

Attention was not the only place to find interpretable handles. The first convolution layer could be decomposed exactly into contributions from the current token and six predecessors.

Word fragments drew especially heavily on the previous token. For mid-word continuations, the average previous-token share was 87.5%, compared with 56.7% at word starts. Fragments such as berg in Gold|berg made this visible. Replacing the predecessors with random corpus tokens preserved most of the difference. The current fragment was asking to be assembled from what came before; the measurement did not prove that the layer recognized the complete name.

A short-range bracket channel survived a causal check. Pushing it toward the side associated with opening brackets made a closing parenthesis about 2.5 times likelier, relative to the average token. Its memory span was six tokens, not an arbitrary bracketed passage. Several other appealing channel labels did not survive their controls.

A head intervention changed something more conspicuous. Even with the template’s thinking block already closed, a pulse to one head’s gate at the final prompt position made 12 of 16 fresh replies start a new <think> block. A matched head did not reopen it.

The longer path sometimes fixed a payload and sometimes exhausted the answer budget. It was not a universal reasoning switch. But the boundary-only pulse showed that a short internal intervention could set a response trajectory that persisted after the intervention ended.

Where fewer bits stop being a bargain

Compression raised a practical question. How far could the weights be quantized, meaning stored at lower numerical precision, before behavior broke? And was repairing a damaged small model better than spending the same storage on more bits?

Four-bit round-to-nearest decoder matrices retained the diagnostic suite’s mean accuracy and modestly increased text perplexity, a measure of prediction difficulty, from 14.98 to 16.30. Three bits pushed perplexity to 21.91 and lost the templated chat task entirely. At two bits, perplexity reached about 1,352 and generated text became largely unusable.

Ordinary low-rank approximation of rounding error was weak. Weighting the error by how strongly the model actually used each input channel made the repair more effective. A roughly 16 MB patch removed 22% of the three-bit prediction divergence. Protecting attention at higher precision also bought useful recovery per byte.

There was a stopping point. Near 779 MB of decoder-matrix storage, a plain four-bit format with larger groups beat every three-bit fix that fit the same budget. The repaired two-bit options were worse than similarly sized three-bit ones.

The interesting result was not “patches fix quantization.” It was a storage frontier: repair can be worthwhile inside a narrow budget, but sometimes the better repair is to stop compressing so hard.

Reading the atlas, then editing one memory value

The project next broadened from individual candidates to an evidence atlas. It brought together natural activations, internal readouts, attention sources, and intervention effects for 688 entries: sampled feed-forward and convolution coordinates, plus every attention head.

An internal readout translates an intermediate state into candidate words. A derivative-based readout improved some late-layer rankings of concepts not explicitly named in a prompt. Asked for the capital associated with Brandenburg Gate, the model answered Berlin while the readout also ranked Germany highly. Early readouts remained noisy, often dominated by punctuation. This was not a reliable view of hidden reasoning from the first layers onward.

The atlas was more useful as a way to nominate experiments. One final-layer convolution channel had a strong effect before result in an as a context. Fresh prefixes showed a smaller, variable effect, and separate lag contributions partly cancelled each other. That suggested testing the actual short-term memory rather than giving the channel a broad concept label.

The model first read through as. One scalar in the final convolution cache was then replaced with a value from a matched by context, before feeding the original a. The next word changed from result to whole. Restoring the scalar restored result and the observed continuation.

That is a small but direct memory-edit demonstration. It does not establish a general consequence feature or improve an answer. Its direct influence expired after the six-token kernel window under fixed continuations. Different generated words could nevertheless carry the consequence further by becoming new input.

A compact controller for complete tool calls

The next step tested whether internal control could select an action rather than a word. Replacing an early whole-token state first changed a generated call from addition to multiplication while keeping the visible operands fixed. The convolution-only part did not reproduce the switch.

That broad transplant was then compressed. A shared fitted basis used three coefficients, supported on 512 of the token state’s 2,048 coordinates, to choose among add, multiply, maximum, and minimum. It was an engineered controller, not three discovered native command neurons. All four operations were represented during fitting.

On 16 held-out commands, the internal controller produced 14 correct requested calls, matching ordinary text instructions and the full-state transplant. Ten were newly induced correct switches. The remaining successes included no-ops and an already-target baseline choice; 14 correct calls did not mean 14 switches.

{
  "tool": "add",
  "result": 17,
  "receipt": "9ed4055d"
}

<tool_call>
<function=add>
<parameter=a>
4
</parameter>
<parameter=b>
13
</parameter>
</function>
</tool_call>

{
  "tool": "multiply",
  "result": 52,
  "receipt": "17ca28f7"
}

<tool_call>
<function=multiply>
<parameter=a>
4
</parameter>
<parameter=b>
13
</parameter>
</function>
</tool_call>

{
  "tool": "maximum",
  "result": 13,
  "receipt": "c87ac261"
}

<tool_call>
<function=maximum>
<parameter=a>
4
</parameter>
<parameter=b>
13
</parameter>
</function>
</tool_call>

{
  "tool": "minimum",
  "result": 4,
  "receipt": "84887664"
}

<tool_call>
<function=minimum>
<parameter=a>
4
</parameter>
<parameter=b>
13
</parameter>
</function>
</tool_call>

Completing the round trip needed another instruction. The original final-response requirement failed in both text and internal arms. A separately fixed formatting follow-up, shared by both, asked the model to copy the actual tool, result, and receipt from the returned message. It achieved faithful replies on all 16 calls per arm, with 14 complete intended round trips because the two wrong calls remained wrong.

The receipt was computed from the actual executed call. Copying it demonstrated access to the returned message, not just guessing the arithmetic. The same input could lead to add(4,13)→17, multiply→52, maximum→13, or minimum→4.

The intervention footprint fell by 75% relative to the full transplant. Speed did not improve. This was a portable control interface within a small tested domain, not a new capability or a production advantage.

Following a digit until it becomes a color

The final study began with an odd but concrete question: does LFM3 associate particular digits with particular colors?

A single answer was not enough. The stronger test asked for colors across 1,000 shuffled digit orders, producing 10,000 sampled answers. A distributional palette emerged. Black was named for 0 in 24.9% of answers and for 8 in 24.2%, but at most 3.1% for any other digit. Purple appeared for 9 in 26.9% and for 7 in 21.8%.