Not all legal AI is the same: how JILL is built for accuracy and ethics

If you are shopping for AI right now, every vendor deck says the same thing. Legal-specific. Citation-backed. Built for lawyers. The words are free.

The evidence is not. There is now independent research on how these tools actually perform, and it says two things at once: the tools are not all alike, and none of them has made the problem go away.

What the independent research says

The Stanford study. Researchers at Stanford tested leading AI legal research tools on 202 preregistered queries and published the results in the Journal of Empirical Legal Studies in 2025. They found that the tools tested hallucinated between 17% and 33% of the time. That was lower than a general-purpose model, which is the point vendors cite. But the authors concluded the providers' "hallucination-free" claims were overstated.

Two details matter more than the headline number.

First, the study defined a hallucination as a response that is either incorrect or misgrounded. Misgrounded means the answer cites a real source that does not support the proposition. A fake case is easy to catch with a citator. A real case cited for something it does not say is harder, and it showed up repeatedly.

Second, the error types varied by tool. The researchers found misstated holdings, litigants' arguments confused with the court's holding, district and appellate courts mixed up, and authority that had been overruled. Different architectures failed in different ways. Reasoning errors dominated in one product, naive retrieval in another. That is what "not the same" looks like in practice.

Caveat: those tools were tested in spring 2024 and have changed since. Treat the numbers as evidence that the problem is real, not as a ranking of today's products.

The Vals benchmark. In October 2025, Vals AI published a legal research benchmark of 210 questions. The legal AI products it tested averaged about 80% accuracy, a general-purpose model scored about the same, and the lawyer baseline was 71%. On the narrower measure of authoritativeness, the legal products averaged 76% against 70% for the general model. Every system did worse when a question required synthesizing the law of multiple states. Lawyers did better in roughly a third of the question categories, mainly those that required reconciling conflicting authority.

The limits are worth stating. Several of the biggest legal research vendors did not take part, and one withdrew before publication. The benchmark covers research questions, not drafting, and it does not test whether a citation holds up.

Put the two together and the picture is consistent: purpose-built tools help, the gap between tools is real, and no one has earned the word "solved."

Two different things are sold as "legal AI"

The research above tests tools that do one job: answer legal questions. A lot of what lawyers are being sold under the same label does a different job.

The assistant layer. Many practice-management platforms have added an AI assistant to a product built years before language models existed. Going by vendors' own published descriptions as of October 2026, these assistants are good at running the practice: they read matter data, summarize documents and email threads, draft routine correspondence, answer questions about a file, and create tasks, memos and calendar events. That is useful work. But the model underneath is typically a general-purpose model, in some cases one the user picks from a menu of third-party models. Legal research, where it is offered, may come through a partner integration or a separate product.

Purpose-built research and drafting. Some vendors have added dedicated research and drafting products that return cited sources. That is a real step forward, and their documentation is right to say that a lawyer must still review the output.

These are different approaches, built for different jobs, and each has its place. A system whose core job is legal work product, and which is built so that its output has to clear checks before you see it, is a third thing. The difference is not the logo on the chat window. It is where the legal reasoning lives, what the model is allowed to say, and what happens to a claim before it reaches you.

JILL is the second kind of system. Legal research, drafting and analysis are not a feature attached to a practice-management product. They are what JILL is built to do, running on the same data model as the matter itself. Citation validation, confidence scoring and the audit trail are not options you switch on or a partner you route to. Every output passes through them. When an assistant summarizes your email, a wrong answer is an inconvenience. When a system drafts the motion, a wrong answer is a sanctions order. Those two jobs deserve different engineering.

Built to a standard, not just built with a model

Underneath all of this is a question that has nothing to do with which model a tool runs on: what standard was it built to?

Every legal AI tool says it takes ethics and compliance seriously. We are not suggesting anyone else does not. The point is that a promise is not the same as a published standard that the product is engineered around, and a lawyer evaluating tools is entitled to ask for the second.

For JILL, that standard is the Discover Docket Ethical and Accountability Standards, or DDEAS. It is the foundation JILL sits on, so the ethical requirements shape the architecture instead of being added after the fact. It has four layers:

1. Source validation. Citations are checked against authoritative databases for the matter's jurisdiction before they reach you.
2. Confidence scoring. Every output carries a score from independent models, so uncertainty is visible.
3. Audit log. Every prompt, retrieval and output is recorded, append-only and signed.
4. Tamper-evident chain. Entries are hash-chained, so altering history is detectable and the record can stand as evidence.

Those layers map directly onto the duties lawyers already carry. Competence and candor to the tribunal call for verified authority. Supervision calls for work you can inspect. Confidentiality calls for controlled access and no third-party training on client data. Reasonable fees and client communication call for a record of what was done. That is why Discover Docket was built for ABA Formal Opinion 512: the platform is designed to support a lawyer's compliance with each of those duties, and the lawyer remains the one who is responsible for the work.

Two honest limits. A standard is only as good as its execution, so we want you to test it, not take our word for it. And a standard supports your compliance. It does not replace your judgment.

Nine questions to ask any legal AI

You will not settle this from a demo. Ask these.

1. Where does the authority come from? A model's memory, or a verified legal database? If the tool answers from what it learned in training, it is guessing in a legal voice.

2. What is checked before I see it? Existence is the minimum. Does the tool also check whether the case is still good law, and whether it supports the point it is cited for?

3. Does it know my court? A motion in one jurisdiction is not a motion in another. Does the tool apply the controlling rules, the local rules and the assigned judge's standing orders, or does it write a generic motion and leave the adaptation to you?

4. Does it work from my matter? A tool that cannot read your deposition transcripts, your discovery responses and your pleadings is working from a prompt, not a file.

5. Does it tell me where it is unsure? A confident paragraph and a shaky paragraph look identical on the page. Does the tool tell you which is which?

6. Can I see what it did? If a court or bar counsel asks how a passage got into your filing, can you show them? A chat transcript in a browser tab is not a record.

7. Where does my client's information go? Is it used to train anyone's model? Who inside your firm can reach it? The privilege cases decided in 2026 have already shown that platform terms matter.

8. Is it one system or a bolt-on? An AI feature attached to a product that was designed without it inherits that product's data model. Ask what the AI can actually see, and whether legal research and verification are built in or provided through a partner or add-on.

9. What standard was it built to? Not a policy page. A framework the product is engineered around, that you can read, that maps to your professional duties, and that does not depend on someone remembering to turn it on.

How JILL minimizes and mitigates false information

Most tools treat hallucination as a model problem and wait for a better model. We treat it as a systems problem. A model will sometimes be wrong. The question is what stands between its mistake and your signature.

JILL is built in four layers, each aimed at a different part of the failure.

1. Prevent: give the model less room to invent. JILL works inside the matter, not from a blank prompt. It knows the court, the judge, the procedural and local rules, the standing orders, and the pleadings, discovery and depositions in the file. It draws on authority for the jurisdiction the matter sits in, and it does not answer from training memory. A model that is handed the facts and the controlling law has far less reason to fill gaps with plausible fiction.

2. Catch: validate before anything reaches you. Every citation JILL produces is checked against the authoritative case databases for that jurisdiction before it appears in a draft. A case that does not exist does not appear. A citation with a small error, such as a transposed reporter or a wrong volume, is corrected or surfaced for review. Authority with a good-law problem is flagged rather than quietly passed along. The validation layer is the gate, and nothing skips it.

3. Surface: show where the risk is. Each paragraph carries a confidence score. The scores come from independent models that evaluate the output against the source material, the consistency of the supporting authority and the freshness of the underlying data, so the model that wrote the paragraph is not the one grading it. Citations link to the source JILL relied on, so checking whether a case actually supports the sentence is one click, not a trip back to a research database. A low score tells you to slow down. That is the review you were always going to do, aimed at the right places.

4. Record: keep proof of what happened. Each prompt, retrieval, score and output is written to an append-only, signed audit log, chained so that altering a past entry is detectable, under the Discover Docket Ethical and Accountability Standards. If something does get through, you can establish exactly what happened instead of reconstructing it from memory. As the sanctions cases show, that record is often what separates a mistake from a cover-up.

Mapped to the failures the researchers found

The Stanford study did not just count errors. It named them. Here is how each one is addressed.

• Fabricated authority. Existence validation against authoritative databases. The case is either found or it is not shown.
• Misgrounded citations (a real case that does not support the point). This is the hardest category for any system. JILL's response is to score output against the source material and to link every citation to its source, so the mismatch is something you can see quickly. It reduces the risk. It does not remove your duty to read the authority you rely on.
• Overruled or outdated law. Good-law status checks and a freshness component in the confidence score, with a flag when authority has been affected.
• Wrong court or wrong jurisdiction. The matter record sets the jurisdiction and the rules, so the tool is not guessing which court you are in.
• Naive retrieval. Grounding in the matter file and jurisdiction-specific authority, instead of a keyword match against everything.
• Overconfidence. Paragraph-level scoring, so a shaky paragraph does not look like a solid one.

And on the other questions: JILL's firm data is never used to train third-party models, ethical walls and role-based access control who inside the firm sees a matter, and because case management, documents, calendar, discovery and AI share one data model, JILL can see the matter in the first place.

What we are not claiming

We are not claiming JILL is hallucination-free. The Stanford authors were right that no one has earned that claim, and a product that makes it should be treated with suspicion. We are not claiming a benchmark win, either. We have not published a head-to-head result and will not imply one.

What we are claiming is narrower and checkable: the structure around the model is built to prevent, catch and surface the failures courts are sanctioning, and the record of what happened is yours to produce. Validation reduces the chance a false citation reaches you. It does not replace your review. You still read the work. You still sign it.

The takeaway

When a vendor says "legal AI," ask the nine questions. If the answers are vague, the label is doing the work that the architecture should be doing.

Then ask the same questions of us.

See how JILL works →

Built for ABA Formal Opinion 512 →

Compare platforms →