Every prediction follows the same fundamental pattern. When a doctor diagnoses a patient or a bank evaluates a loan, they execute a three-step process: the Request (What do I need to know?), the Index (How do I search history for relevance?), and the Store (What outcomes do I retrieve?).
The entire history of statistical modeling, from simple averages to transformer networks, is the story of making these three atoms less rigid, more contextual, and more adaptive.
Linear regression made a bold bargain with complexity. It compressed history into coefficients, betas that would never change. The model performed one grand search during training, found the "best fit line," and declared the search complete. The index became frozen.
When a new case arrives, the model doesn’t ask, "What is relevant now?" It asks only, "What did I decide relevance meant back then?"
Chandrakant Maheshwari
Think of a beta coefficient as a ghost of a decision made in the past, now haunting every future prediction. This has profound psychological appeal. Betas are interpretable, stable, and governable. Until reality interrupts.
The Aggregation Trap: A global rule assumes one relationship fits everyone. But real data contain subgroups with different patterns. Imagine modeling default risk with both salaried employees and self-employed borrowers. For salaried workers, higher income strongly predicts lower default. For self-employed individuals, the relationship is much weaker, income is volatile and doesn't reflect true financial stability the same way.
When you fit one regression line through all the data, you're averaging these different patterns. The math works perfectly, but you've silently decided that salaried workers’ pattern (the majority) should dominate. When a self-employed person with high income applies, your model is overconfident about their stability, not because it calculated incorrectly, but because it’s applying a rule calibrated for a different population.
This is regression’s original sin: It believes in global truth when the world operates in local contexts.
Someone eventually asked: "What if we stopped trying to find one rule for everyone? What if we just looked for similar cases?"
Kernel regression and k-nearest neighbors changed the fundamental question from "What is the average relationship?" to "Who looks most similar to this case?" When a new case arrives, scan the dataset, identify the most comparable historical cases, and give them more influence.
The Geometry of Relevance: Era II’ s answer was distance. If two cases are numerically close, they should influence each other. Closer neighbors speak louder. A kernel function translates distance into weight: Similar cases get large weights, different cases get small weights.
It’s intuitive and adaptive. And for a while, it worked beautifully.
When Geometry Breaks Down: Then we started working with richer data, more features, more dimensions. In high-dimensional spaces, distance stops meaning what we think it means. Every point becomes roughly equidistant from every other point. Neighborhoods blur. Everything looks equally far from everything else.
Consider two loan applicants nearly identical numerically: Age 35 vs 34, income $75K vs $76K, credit score 720 vs 718. But what if one is a nurse with stable employment, and the other a gig worker with volatile income? The numbers are close, but the risk profiles are completely different. An expert sees structure. Distance sees only numbers.
This led to a deeper realization: in complex domains, similarity itself cannot be predefined. Relevance depends on context, and context cannot be captured by any fixed measure of similarity.
What if similarity itself must be learned?
Remember our three atoms?
Era I froze the Index into betas. Era II made the Index geometric. Era III makes the Index learnable.
This is where attention mechanisms enter, not as mysterious deep learning, but as the natural answer to: How do we decide relevance when relevance itself is contextual?
Query, Key, Value: In modern terminology, our three atoms become Query (the current situation), Key (how each historical case describes itself), and Value (the stored outcome). But crucially, these are learned transformations.
When a new case arrives, the model transforms it into a Query, compares that Query to learned Key representations of historical cases, and retrieves weighted Value information based on alignment.
From Distance to Alignment: Instead of measuring separation (distance), we measure alignment (dot product). If two representations point in the same direction, strong relevance; if orthogonal, no relevance. This is learned directional agreement. Relevance becomes projection, not proximity.
The Competitive Search: Attention doesn’t just compute alignment scores; it forces them to compete through softmax. Why? Because relevance is comparative, not absolute. You're not asking "Is this case relevant?", but "Is this case more relevant than the others?" Attention forces history to compete for influence, mirroring how human memory lets the most relevant experiences rise to the surface.
The Full Picture: The attention mechanism, softmax (QK^T / √d) V, means: Compare current situation to stored representations (QK^T), stay humble in high dimensions (÷√d), force competition (softmax), and retrieve weighted outcomes (V).
The profound insight? This is still weighted averaging. The structure never changed, it's the same retrieval logic from regression and kernels. Only the Index became intelligent.
Attention doesn’t replace traditional models. It expands the regime in which prediction is possible. Traditional models excel when problems are structured, features are stable, similarity can be predefined, and relationships are consistent. They deliver interpretability and governance clarity, essential for regulated industries.
Attention mechanisms excel when relevance is conditional, signals come from heterogeneous sources, and the notion of similarity itself must adapt.
Where Attention Belongs: Sanctions screening combining structured name-matching, transaction patterns, textual narratives, and network relationships – these aren't naturally comparable, but you can learn how they should be weighted. Credit scoring where "recent job change" means something different depending on whether someone moved from gig work to stable employment or vice versa. Fraud detection where patterns shift and what's “similar” to current fraud changes over time.
The question is never "Which model is better?" It's always "Which regime are we operating in?"
When you step back, you see a deeper pattern. Each era taught the model a more sophisticated way to answer: "What should I remember right now?"
This mirrors human expertise development. Novices rely on rules. Intermediate practitioners rely on analogies. Experts have an intuitive sense of relevance, they just know what matters.
The real story isn’t about neural networks or deep learning. It’s about the Index, the search mechanism that decides what history to retrieve. When we froze it, we gained efficiency but lost adaptability. When we made it geometric, we gained locality but lost context-dependence. When we made it learnable, we finally aligned it with how complex reasoning works.
The Index is memory. And memory is not about storing everything, it’s about retrieving what matters.
What remains constant across all eras is the structure of prediction itself.
We’re not inventing new ways to predict. We’re rediscovering the logic that was always there, making each piece more flexible, more contextual, more intelligent.
Linear regression taught us that compression is possible. Kernel methods taught us that locality matters. Attention mechanisms are teaching us that relevance is learned, not given.
The history of prediction is a story of slowly remembering how to remember. We started with frozen summaries, moved to geometric neighborhoods, and arrived at learned relevance. What changes is not the structure of prediction, it's our sophistication in executing each step.
Because in the end, prediction is just memory operating in the service of the future. And the better we get at remembering what matters, the better we get at anticipating what comes next.
Chandrakant Maheshwari is a former lead model validator at Flagstar Bank with over 20 years of experience in financial risk management and financial crime prevention. He is a course author for ACAMS and focuses on ensuring that advanced analytics and generative AI models are used responsibly in regulated financial environments. Chandrakant is the author of two forthcoming books on model risk and financial crime model validation, to be published by Elsevier and Springer in 2026. He has a blog at https://chandrakant721.wordpress.com/