Vector Symbolic Architectures

Here is a word: chair. To a computer, it's five letters — bytes 99, 104, 97, 105, 114 — a sequence that tells the machine nothing about what a chair is, what it's for, or what it resembles. Now here's a different idea. Imagine that "chair" is not a sequence but a point — a single point in a space with ten thousand dimensions, chosen at random. In a space that large, any two random points are so far apart that picking one "by accident" is, for all practical purposes, impossible. Ten thousand dimensions gives you more room than atoms in the universe; noise is not a threat in such a space, it's a cushion.

Vector Symbolic Architectures — VSA, or hyperdimensional computing (HDC) — is the discipline built on that observation. Instead of computing on numbers or strings, it computes on hypervectors: vectors of (typically) 10,000 or more components, generated essentially at random, whose vast dimensionality makes them nearly orthogonal — distinguishable — by default. And on these vectors you define a tiny algebra of operations — bind, bundle, permute, similar — that turns out to be enough to do symbolic computation: to represent records, sequences, analogies, grammars, and entire languages, in a space so big that you can corrupt a third of the bits and still read it.^[1]

The field is thirty-seven years old, was born at NASA's AI shop, waited decades for anyone to notice, and is now being taken seriously for one simple reason: it is the most forgiving kind of computation ever invented. In a world of failing flash cells, noisy sensors, and chips that leak, a scheme that treats corruption as part of the medium — and still answers correctly — is worth understanding. This is that story.

The man who bet on noise

In 1988, a Finnish-American researcher at NASA Ames Research Center — Pentti Kanerva, working at the Research Institute for Advanced Computer Science (RIACS) — published a book that almost nobody read at the time and almost everybody in the field cites today: Sparse Distributed Memory.^[2]

Kanerva's question was a neuroscientist's question disguised as an engineer's: how does a brain remember? Not how do we address memory — von Neumann answered that — but how does a physical, noisy, biological system store a pattern and get it back? His answer was a memory made of an absurd number of very-long addresses: each item stored is a point in a space of a hundred bits or more (today: ten thousand), the memory is built from a sparse scattering of storage locations, and because the space is so large, a noisy version of the query still lands closest to the right answer. The randomness is the memory. The whole thing was designed, in part, for the Connection Machine — the 65,536-processor cube that Thinking Machines sold in the 1980s as the antithesis of the von Neumann bottleneck, and that NASA Ames kept in-house.^[3]

The deepest part of Kanerva's book was not the memory — it was the parent claim: this is how a cerebellum computes. Sparse distributed memory, he argued, is a mathematical model of the cerebellar cortex — the brain's own random-projection machine. Which means the same algebra that stores a memory might be the algebra that does the thinking. The idea was twenty years early, and it spent those decades on the shelf — until the chips caught up.

In 2009, Kanerva — by then at Stanford's Center for the Study of Language and Information — published the paper that named the field: Hyperdimensional Computing. It pulled together the family of schemes that had grown up in the interim — Tony Plate's Holographic Reduced Representations, the Spatter Code, semantic vectors — and stated the field's wager plainly: compute in a space so large that random vectors are already different, and the structure you need — binding, grouping, sequence — comes free with the geometry.^[4]

The algebra

Here is the whole machine. In binary VSA — the simplest, hardware-friendliest family — a hypervector is a random string of 10,000 bits. Everything follows from four operations:

The magic is in the arithmetic of the space. Two random 10,000-bit vectors differ in about 5,000 positions on average, with a standard deviation of only ~50 bits — meaning two vectors that agree on even a few hundred extra bits are measurably similar. There's essentially no way to pick a random vector that looks like another random vector. Uniqueness is the default; you have to work to make things similar.

And the work is exactly what the four ops do:

The worked example

Take three tiny 8-bit vectors (real ones are 10,000 bits — the logic is identical, the numbers are just readable):

Bind a thing to its role: chair ⊗ sit = 11011101 — a new vector that is nothing like either input, but from which either input can be recovered by XOR-ing the other back out: (chair ⊗ sit) ⊗ chair = sit. The pair is stored without destroying either member — that's the difference between binding and arithmetic.

Bundle the facts about a thing: majority(chair, table, bench) produces a vector that is still similar to each member — because majority vote preserves the bits they share. The result is a superposition: a single vector that is "about" all three, and that a Hamming-distance query will find near each of them.

Permute to encode order: ρ(ρ(a) ⊗ b) vs ρ(ρ(b) ⊗ a) are different vectors — position matters — so a sequence a→b→c encodes differently from c→b→a.

Now the classic demo — "What's the dollar of Mexico?" Build a USA record: bundle(country ⊗ USA, currency ⊗ dollar, capital ⊗ Washington). Build a Mexico record: bundle(country ⊗ Mexico, currency ⊗ peso, capital ⊗ MexicoCity). Then ask the question by unbinding: USA_record ⊗ dollar pulls out the USA record's currency slot — the stored "dollar" — and matching it against the Mexico record finds the analogous slot: peso. The query is literally vector algebra, and it works because each role was bound, each record bundled, and similarity does the retrieval.^[5]

The brain link

Kanerva didn't just guess that brains might compute this way — he built SDM as a cerebellar model first. The cerebellum, whose granule cells massively expand sparse input into a vast, high-dimensional space before the Purkinje cells read it, is the textbook biological example of exactly what a hypervector is: a random, high-dimensional recoding of a low-dimensional input.^[6]

The neuroscience case has only strengthened since:

• Place cells and grid cells — the Nobel-winning discovery (Hafting, Moser & Moser) that the hippocampus encodes where you are with periodic, grid-like firing patterns — look, in hindsight, exactly like a hyperdimensional position code: a sparse, high-dimensional representation of space that's robust to noise and can be read out by downstream neurons.
• The fly's nose — the fruit fly's olfactory circuit performs what computer scientists call locality-sensitive hashing: it expands ~50 odor channels into ~2,000 sparse Kenyon cells — a dimensionality-expanding, random-projection step, forty years before the algorithm had a name, to compute "smells similar to" in a brain of 100,000 neurons.^[7]

• Sparse distributed codes everywhere — Bruno Olshausen and others have argued that sparse, distributed, high-dimensional representation is not one neural trick but the cortical motif: from V1's edges-and-orientation codes to entorhinal grids, the brain appears to prefer exactly the representation VSA formalized.

The claim, stated honestly: nobody has proven the brain is a hypervector machine. But the brain keeps looking like one — and the engineered version works, which makes the resemblance worth taking seriously.

What it's good at

The field earned its keep not on benchmarks of imagination but on tasks where it embarrasses more famous machinery:

• Language identification — Rahimi, Kanerva & Rabaey trained a hypervector classifier on letter trigrams from 21 European languages and got ~97.8% accuracy — from letters only, no words, no grammar.^[8]
• News classification — the same architecture beat Bayes, k-NN, and SVM on eight news topics with >94% accuracy — with the whole "model" being a handful of bundled hypervectors.^[9]
• Biosignals — EMG gesture recognition, EEG classification — the kind of noisy, low-power, edge-compute work where deep learning is wasteful and VSA is right at home.
• One-shot learning — a hypervector classifier learns in a single pass: bundle the examples, that's the model. No epochs, no gradient descent, no training loop — the "training" is the bundling.
• No catastrophic forgetting — add a new class by bundling another vector into the memory; nothing is overwritten. Forgetting is not a thing, because nothing is ever erased.

The common thread: VSA wins wherever the job is structured, noisy, small-data, or low-power — the quadrant where massive neural networks are a sledgehammer on a thumbtack.

The hardware argument

And here's where it stops being academic. Hypervector operations are the easiest possible operations for hardware: XOR and rotate are single gates; majority is a counter; Hamming distance is XOR + popcount. An FPGA — or, as we'll see, a reversible engine — does these in a single clock. But the deeper argument is robustness: VSA does not care if its memory is broken.

The flagship result: Abbas Rahimi's group at ETH Zürich ran hyperdimensional classification on a phase-change-memory chip whose memory cells had a failure rate of 8.8× normal — and the classifier kept 94% accuracy. In a follow-up they simulated permanent faults at 42% and relearned the lost accuracy in a single pass, ending at 96.86% — on a model 285× smaller than fastText.^[10]

That is the whole point of the architecture. A deep network is a precision instrument: flip the wrong weight and you break it. A hypervector is a distribution: lose a third of the bits and the answer is still closest to right. In a world of imperfect hardware — aging chips, faulty memory cells, radiation-hardened space hardware, neuromorphic fabrics with real, physical noise — VSA is the only computing paradigm that treats broken silicon as a routine condition rather than a fault.

The family

VSA is the umbrella — the field's name for a set of schemes that all compute on high-dimensional random vectors but differ in the binding algebra:

• Binary Spatter Code — Plate-era, binary vectors, XOR binding — the simplest, hardware-friendliest.
• Holographic Reduced Representations (HRR) — Tony Plate, 1995 — real-valued vectors, circular-convolution binding: the one that made binding formally elegant.
• MAP (Multiply–Add–Permute) — Ross Gayler's formulation — bipolar {−1,+1} vectors, elementwise multiplication as bind — the mathematically cleanest.
• FHRR — Fourier-domain HRR — same algebra in the frequency domain.

Kanerva's 2009 paper is what welded them into a field — the recognition that the operations were the same abstract algebra wearing different number systems, and that the right level of abstraction is the space itself.

What it can't do

Honesty check, per house style — VSA's limits are real:

• It's not a deep-learning replacement — on perception tasks at scale (image recognition, large language modeling) the trained networks still win, because gradient descent extracts structure a random projection can't find.
• The symbol vocabulary is hand-chosen — someone has to decide what the base symbols are; VSA gives you the composition rules, not the ontology.
• Hierarchical structure is hard — representing deeply nested, compositional structures is possible but fiddly; the field is still working on clean recursion.
• Interpretation needs the item memory — a hypervector's meaning lives in the dictionary of base vectors; the vector itself is opaque.

The fair summary: VSA is not the future of all computing. It may be the future of the part of computing that's small, noisy, embedded, structural, or alive — which is a larger fraction of computing than the GPU market suggests.

What might happen

Flagged as extrapolation:

• Edge AI — sensors, wearables, always-on devices where a deep model is too heavy but a hypervector classifier fits in a few kilobytes of SRAM.
• Neuromorphic and in-memory hardware — the "compute in the memory" story: HDC on phase-change memory and analog substrates where the storage is the compute, eliminating the von Neumann bottleneck at the hardware level.
• A symbolic layer for neural AI — the hybrid bet: networks that perceive, hypervectors that reason — VSA as the symbolic brain of an embodied agent.
• The biological bet — if the brain really does compute in high-dimensional codes, the field isn't just an engineering trick — it's a partial theory of how biological thought works, in which case the "space" metaphor stops being a metaphor.

The last word

Computing has spent eighty years assuming that precision is what makes machines smart — that the answer must be exact, the bits must be right, the weights must be tuned to the seventeenth decimal. Vector Symbolic Architectures proposes the opposite: a computer whose symbols are clouds, whose memory is a fogbank of ten-thousand-bit points, whose every operation is a permutation or an XOR — and which, by the arithmetic of very large spaces, is more correct because of it.

Kanerva's bet, placed at NASA in 1988 and largely ignored for twenty years, was that the brain already knew this — that intelligence is not a precise algorithm running on reliable machinery but a vast, redundant, holographic pattern that cannot help but remember, because in ten thousand dimensions there is nowhere else to put a memory but everywhere. The chips are finally catching up to the idea. The idea was waiting for them all along.

Quiz

• 1 — What's a hypervector, and roughly how many dimensions does one have?
• 2 — Name the four core VSA operations and what each does.
• 3 — In the "dollar of Mexico" demo, how is the analogy actually answered?
• 4 — What did the ETH Zürich PCM experiment demonstrate about fault tolerance?
• 5 — What neural system did Kanerva model SDM on?

Notes

1. Kanerva's SDM dates are 1988 (MIT Press) for the book; the NASA/RIACS technical reports predate it to the mid-1980s.
2. The language-ID figure (~97.8%) and the 94%-at-8.8×-faults figure are from the published ETH/UC Berkeley papers, reported as results.
3. The brain-link evidence (grid cells, fly olfaction) is presented as resemblance, not proof — the brain-as-hypervector claim is a hypothesis, flagged as such.
4. The four-op tutorial uses 8-bit vectors for readability; production systems use ~10,000.
5. VSA is the family name; "hyperdimensional computing" is Kanerva's 2009 term and the common industry name.

References

• Kanerva, P., Sparse Distributed Memory (MIT Press, 1988).
• Kanerva, P., "Hyperdimensional Computing," Cognitive Computation 1:139–159 (2009).
• Kanerva, P., "What we mean when we say 'What's the dollar of Mexico?'" (AAAI Fall Symposium, 2010).
• Plate, T., Holographic Reduced Representations (CSLI, 1995/2003).
• Rahimi, A., Kanerva, P. & Rabaey, J., "A Robust and Energy-Efficient Classifier Using Brain-Inspired Hyperdimensional Computing," ISLPED (2016).
• Rahimi et al., "Hyperdimensional Computing for Text Classification," DATE (2016).
• Hersche, M., Sangalli, S., Benini, L. & Rahimi, A., "Evolvable hyperdimensional computing" (ETH Zürich).
• Dasgupta, S. et al., "A neural algorithm for a fundamental computing problem," Science 358 (2017).
• Denning, P. J., "A View of Kanerva's Sparse Distributed Memory," RIACS TR-86.14, NASA Ames (1986).