Why System 2: AI that checks its work before you see it

Who was Daniel Kahneman, and what are System 1 and System 2?

Daniel Kahneman (1934 to 2024) was a psychologist at Princeton who received the 2002 Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel for research on judgment under uncertainty, most of it done with Amos Tversky. His 2011 book “Thinking, Fast and Slow” gave a general audience two names for two modes of thought.

System 1 is fast and automatic. It reads a word, recognises a face, and completes the phrase “bread and …” with no sense of effort. It reaches answers by matching patterns against everything it has seen. That speed is most of what makes a working mind useful, and Kahneman was explicit that System 1 is the origin of most of what people get right.

System 2 is slow and effortful. It is the mode used for careful arithmetic, for weighing evidence, and for catching your own mistakes. It is also, in Kahneman’s account, lazy. It tends to accept whatever System 1 hands it, because checking costs attention and attention is scarce.

One detail matters more than the rest of the framework, and it is usually left out of the business version of this idea. Kahneman insisted the two systems do not exist. He called them fictitious characters, adopted because a sentence about an agent is easier to hold in mind than a sentence about a process. So the name S2Reason describes an intention about how software should behave. It is not a claim that our software has a mind.

Why does most generative AI behave like System 1?

A language model produces the most plausible continuation of a text, quickly and with no internal signal for when it has stopped being reliable. That is a precise description of System 1. The result is fluent, confident output that is sometimes wrong in ways the output itself gives no clue about.

Kahneman’s summary of the problem transfers almost word for word to a language model, which is why the analogy is worth more than a name.

Now consider the work these systems are asked to do. A New Jersey personal-injury firm running PIP arbitration has to reconcile medical bills against a state fee schedule, the published table that caps what a provider may charge for a given procedure code on a given date of service. A medical-legal examiner has to summarise hundreds of pages of records without dropping a diagnosis or shifting a date. In both cases the error does not announce itself. It surfaces months later, in front of an arbitrator.

That delay is the entire problem. It means the cost of a fast wrong answer is paid by someone else, later, and that nobody in the loop gets the quick corrective feedback that would let them learn to distrust the tool.

What are the named failures of fast thinking, and what catches them?

Six failure modes in Kahneman’s account have direct machine analogues: WYSIATI, substitution, cognitive ease, neglect of ambiguity, the bias to believe, and the lazy controller. Each one names a specific way a fluent answer goes wrong, and each one implies a specific control. The table below is the working spec behind our pipelines.

Scroll the table sideways on a narrow screen.

Read down the middle column and a pattern appears. Not one of these failures is fixed by a better model. They are all failures of a system that produces an answer and stops. Each control in the right-hand column is something that happens after the answer exists, which is why we treat the pipeline, rather than the model, as the product. You can see the same six controls at work in our PIP arbitration engine for regulated billing work and in the expert report engine that extracts findings from unstructured records.

Why does a human reviewer fail to catch the error?

Because checking is optional and effortful, and people skip it. Kahneman’s clearest demonstration is the bat-and-ball problem, where roughly half of the students tested at Harvard, MIT and Princeton gave the intuitive wrong answer. Verifying it would have taken seconds. That is the same control most AI deployments rely on, and it is the weakest part of the design.

The problem reads: a bat and a ball cost $1.10 in total, and the bat costs one dollar more than the ball, so how much does the ball cost? Ten cents arrives immediately and is wrong. The answer is five cents. Shane Frederick, who built the test, was not measuring arithmetic. He was measuring whether a person checks an answer that feels finished.

Now put a reviewer in front of forty pages of fluent, well-formatted, internally consistent AI output. They are in the position of the student staring at $1.10, with one difference that runs the wrong way: the cost of checking is no longer low. It is the most expensive part of their day.

This is the position we will defend, and we think most of the industry has it backwards. A verification step that costs the reviewer as much as doing the work will not happen, whatever the policy says. “Human in the loop” is not a control. It is the name of a place where a control is supposed to be. If the reviewer has to open the source record, find the relevant page, and compare it by hand, the review degrades into a spot check within a week of go-live, and nobody files an incident report about it because nothing visibly breaks.

So the engineering problem is not to add a checker. It is to make checking cheap enough that it actually occurs. That is why every figure our systems output carries the page it came from, and why the reviewing screen puts the claim and the source span side by side. We are not trying to make the reviewer more diligent. We are trying to make diligence take four seconds.

Does retrieval-augmented generation already solve this?

No. Grounding a model in source documents narrows what it reads, but it does not audit what it writes. Stanford researchers tested three purpose-built legal research tools that all use retrieval, and found hallucination rates above 17 percent for two of them and above 34 percent for the third, across more than 200 queries.

That 2024 study, from Stanford’s RegLab and Institute for Human-Centered AI, is the most useful number in this whole debate, because it tests the fix rather than the problem. These were not general chatbots. They were expensive tools built for lawyers by companies with the underlying case law in hand, and they retrieved before they answered. The study separates two kinds of error worth keeping distinct: an answer can be simply incorrect, or it can be misgrounded, meaning the law is stated correctly but the citation attached to it does not support the claim.

Misgrounding is the more dangerous of the two, and it is the failure that survives retrieval. A citation is present. It is real. It points somewhere plausible. Nothing on the surface of the document tells the reader that the link between the claim and the source was never checked. Retrieval decides what the model reads. Verification decides whether what it wrote is supported. Those are two separate pieces of engineering, and buying the first does not get you the second.

The consequences are now a matter of public record. In Mata v. Avianca (Southern District of New York, June 2023), a court sanctioned two attorneys and their firm after a brief cited six judicial decisions that did not exist. Courts have been issuing similar sanctions steadily since.

Where does Kahneman’s research cut against us?

On the value of the human. Kahneman endorsed Paul Meehl’s finding that simple statistical rules match or beat expert judgment. A meta-analysis of 136 studies found mechanical prediction about 10 percent more accurate on average, with expert judgment substantially better in only 6 to 16 percent of comparisons. The honest reading is that adding a person usually lowers accuracy.

We are naming this because leaving it out would be a form of the cherry-picking this page is arguing against. A company named after System 2 has an obvious incentive to quote the parts of Kahneman that flatter careful human review. The literature does not support that reading. Meehl called it his disturbing little book for a reason, and Kahneman went further, siding with the argument that relying on intuitive judgment for consequential decisions is hard to defend when a rule is available that makes fewer mistakes.

So what follows for a document pipeline? Three things, and none of them is “trust the reviewer more.”

• The win is consistency, not intelligence.Rules beat experts largely because experts are inconsistent. Ask a professional to judge the same case twice and the two answers differ more than anyone expects, a phenomenon Kahneman later devoted an entire book to in “Noise” (2021). A pipeline that applies the same check in the same order every time is not smarter than your staff. It is merely identical to itself, which turns out to be worth a great deal.
• The human’s job is the exception, not the average. Meehl reserved a category for the decisive fact a rule cannot see, and the case for human review rests on that category plus one thing the research has nothing to say about: someone has to sign the filing and answer for it. That is a professional and legal fact, not a performance claim, and it is the honest reason to keep a person in the loop.
• Judgment should be structured, not replaced.When Kahneman rebuilt the Israeli army’s interview process around Meehl’s findings, he did not remove the interviewers. He stopped them forming a global impression and had them score specific factual traits in a fixed order. Intuition still added value, but only after the disciplined collection came first. That sequence is the design of every review screen we build.

When is fast AI good enough?

Fast is good enough when two conditions hold. Kahneman and Gary Klein, working across a genuine disagreement, concluded that intuition can be trusted only in an environment regular enough to be predictable, and only where there has been practice with quick, clear feedback. Those two conditions make a usable purchasing test.

Their 2009 paper is the least fashionable and most practical thing either of them wrote. It replaces the question “is this model any good” with a better one: is this task the kind where fast judgment can be reliable at all? Applied to document work, it sorts jobs cleanly.

The right-hand column is where we work, and it is not an accident. It is the definition of the market for verified AI: any process where being wrong is quiet. If your own workflow sits in the left-hand column, you do not need us, and we will tell you so on the call.

What we do not claim

Two things. Our name does not mean we use a reasoning model, and we do not treat “Thinking, Fast and Slow” as settled science. Parts of the book failed to replicate, Kahneman said so publicly, and we built only on the parts that held.

“System 2” here does not mean a reasoning model

Since 2024, “System 2” inside AI has come to mean something specific: a model that spends extra compute generating an internal chain of thought before it answers. Research such as Meta’s “System 2 Attention” paper uses the term the same way. A reader who follows the field will reasonably assume that is what our name refers to. It is not.

Thinking for longer is not the same as checking. A model that reasons at length can arrive, fluently and with excellent internal logic, at a claim your records do not support, because nothing in that process compares the conclusion back against the document. The Stanford result above is the evidence: those tools retrieved before answering, and still misgrounded. Our claim is narrower and easier to test than “our model reasons better.” Every figure traces to a page, and the checking step is built to be cheap for the person who signs.

Parts of the book did not survive

Psychology went through a replication crisis after 2011, and Kahneman’s book took real damage. The chapter on priming has largely not held up. Ego depletion, the idea that willpower runs down like a fuel tank, failed a 23-laboratory pre-registered replication in 2016. The finding that a hard-to-read font makes people reason more carefully failed too, across 17 experiments.

Kahneman’s own response is the part worth quoting, because it is rarer than any of the findings.

Running that filter over the book our company is named after is not a caveat we felt obliged to add. It is the same operation this entire page recommends: take the fluent, famous, confident claim, and go check it against the source before you act on it.

Who is the System 2 approach for?

The approach fits professionals whose errors are quiet and expensive: attorneys, medical-legal examiners, medical billers, and practice managers handling protected health information and high-stakes filings. The test is not how complex your documents are. It is how long it takes for a wrong answer to come back to you.

If a mistake surfaces immediately, you can afford a fast tool and your own eyes. If it surfaces in arbitration, in an audit, or in a motion filed under your signature, the checking has to be built into the process that produced the answer, because there is no natural feedback that will catch it in time.

That audience also shapes what we do with the data. The deliberate work runs in a single-tenant environment on HIPAA-eligible AI infrastructure under signed business associate agreements, with encryption in transit and at rest, and we do not train model weights on your documents. You can review the single-tenant, HIPAA-focused security architecture in detail, see the full range of systems we have built for regulated document work, or read who we are and why we build this way. To talk through a workflow of your own, reach the S2Reason team through our contact page.