You could have invented transformers from memory

Alex Wang explains linear layers, MLPs and self-attention as associative memory stores and estimates how many memories each can recall. With 64-dimensional keys, capacity grows from about 64 for a linear store to about 2,700 for a squared-ReLU MLP and over a million for self-attention.

AI Pillar Hub files a short pointer. The reporting lives at the source, Hacker News.