When AI enters adjudication, the knowledge layer becomes part of due process

What happened

On March 25, 2026, the Toronto Star reported something that deserves more attention than it's getting outside immigration and administrative law circles.

A Canadian immigration refusal letter, according to the reporting, disclosed that generative AI had been used in reviewing the application. The letter also contained a description of the applicant's job duties that bore no relation to their actual work. The duties listed were, by all accounts, invented (Toronto Star).

That is not a chatbot hallucination. That is fabricated content inside an official administrative record.

The distinction matters.

Why it matters

There is a version of this story where the lesson is narrow: immigration officials shouldn't use generative AI carelessly. That lesson is true. It is also insufficient.

What the incident describes is a failure mode that will recur across any domain where AI-generated content can quietly enter the evidentiary layer of a consequential decision. Immigration is one such domain. Insurance adjudication is another. Benefits determinations, licensing reviews, compliance investigations, public procurement, and healthcare coverage decisions all share the same basic structure: document-heavy workflows where a decision affects someone's rights, livelihood, or safety.

The problem is not automation in the abstract. It is something more specific: when generative AI summarizes, annotates, or shapes how underlying documents are interpreted, the gap between what the source document actually says and what appears in the decision record can widen without anyone noticing.

In a chatbot interaction, a hallucinated answer is embarrassing. The user can push back, ask again, or ignore it. In an adjudication record, a hallucinated sentence becomes part of the factual foundation a decision rests on. The applicant must then rebut claims they never made. Their appeal process begins from a polluted starting point. Every downstream review inherits that contamination.

The breakdown

This is not an AI bias story

Most AI-in-government coverage focuses on bias: discriminatory outcomes baked into training data, disparate impact across demographic groups. That is a real and documented problem. But this incident is something different.

Bias means the model is applying a skewed pattern to real information. Fabrication means the model invented information that was never there. A biased summary of a job application is still constrained by the actual job application. An invented job description is not constrained by anything. It draws from statistical patterns in training data and presents the output with the same confidence as a summary of source material.

Courts have increasingly made this distinction. As Cary Coglianese of the University of Pennsylvania Law School wrote in The Regulatory Review last week, AI tools that produce answers "with a degree of plausibility and an air of authority" create a specific due-process risk when agency officials rely on them without verifiable grounding: courts will still expect reason-giving based on sound evidence, not "the policy equivalent of taxi driver opinions" (The Regulatory Review).

Fabrication inside a decision record is the clearest possible case of that failure.

The real failure is evidence traceability

The incident points to a missing requirement in how AI is being deployed in regulated workflows: source attribution at the claim level.

When a human reviewer reads a synthesized summary of a 40-page application, they are reviewing the summary, not the application. If the summary contains errors, the reviewer is not positioned to catch them without access to the original material and a reason to go looking. The value proposition of AI in high-volume adjudication is that it handles volume. Humans review outputs. That model only works if the outputs are grounded in, and traceable back to, the underlying source documents.

What regulated AI workflows actually need is not different in principle from what auditors require in financial controls: every material claim needs a traceable line back to the evidence that supports it. In an AI-assisted adjudication context, that means:

• Source attribution: which document, which passage, which version produced this claim
• Version integrity: the document the AI read should be the same document the record references
• Contradiction detection: if an applicant's submitted materials conflict with what the AI characterized, that conflict should surface, not disappear into a summary
• Audit-safe boundaries: clear workflow documentation of where AI generated content versus where a human drafted it
• Evidence-visible review: human reviewers should be able to inspect source records, not just synthesized conclusions

None of these requirements are exotic. They are what you would ask for in any process where a consequential decision has to be defensible on appeal.

Why "human in the loop" is not enough

After incidents like this, the standard institutional response is to add a human review step. The intention is sound. The execution usually isn't.

Meaningful human oversight requires the human to be able to inspect the reasoning, not just approve the output. If a reviewer sees a synthesized job description with no visible connection to source documents, they have two options: accept it or reconstruct the analysis from scratch. In a high-volume operation, the latter happens rarely. The former means the human is reviewing AI-shaped reasoning without the evidence layer that would let them evaluate whether that reasoning is correct.

We have covered the same structural problem in other settings. Courts have stopped debating whether AI was used and started asking what it read. The same logic applies in administrative adjudication: the audit trail of what the AI reviewed is becoming the substance of the legal question, not a footnote to it.

A human approving a synthesized conclusion is not the same as a human reviewing evidence. When the evidence layer is opaque, the human review step becomes a formality.

This problem extends beyond immigration

Canada's immigration case is the most visible recent instance, but it is not unusual in structure.

Healthcare systems use AI to summarize clinical documentation before physician review. Insurance carriers use AI to triage claims. Government agencies use AI to process eligibility determinations. Legal teams use AI to review contracts and flag compliance issues. In each case, if the AI-generated content enters an official record, a recommendation, or a decision artifact without source attribution, the same failure mode is present: fabrication or distortion that is invisible to downstream reviewers.

Legal AI hallucinations have already produced expensive outcomes in courts. In one week last year, three separate courts took action against filings where AI-generated citations did not correspond to real cases. Administrative workflows are next. The error surface is larger, the individual decisions are often less scrutinized than court filings, and the volume is far higher.

What it means for organizations

The incident makes one thing concrete: once AI influences a rights-affecting or compliance-affecting decision, the knowledge layer that AI reads becomes part of the control environment.

This is a different requirement than "do not use AI carelessly." It is an infrastructure requirement. Organizations that deploy AI in consequential workflows need systems that can demonstrate, for any output, what the AI actually read, where it came from, and whether it remains consistent with the current state of source documents.

Public records AI is already being tested on this standard. Private-sector regulated industries are not far behind.

The category of infrastructure that addresses this is governed knowledge retrieval: systems where every AI-generated claim is grounded in a specific document and passage, where contradictions across source materials surface before they reach a summary, and where the knowledge base itself has version integrity and audit logs. Mojar AI is built around this model. Source attribution on every answer, contradiction detection across the knowledge base, governed updates, and clear traceability from claim to source. In contexts where the accuracy of source-attributed information is not a feature but a compliance requirement, that architecture is the gap between AI that can be used in regulated workflows and AI that cannot.

The Canada immigration case is a good illustration of what happens when that infrastructure is absent. The AI read something, or generated something, that the record then contained. Nobody could easily verify which. The decision proceeded from there.

What to watch

A few open questions will resolve over the next 6 to 12 months and shape how AI in adjudication is actually governed:

Will regulatory bodies narrow the permissible uses of generative AI in rights-affecting decisions? The current IRCC AI strategy (Canada.ca) describes safeguards in principle. Whether it requires source attribution in practice is a different question.

Will audit and logging requirements in public-sector AI get more specific? "Human in the loop" as a policy principle is not sufficient if it does not define what the human is required to be able to inspect.

And more broadly: will AI governance frameworks move from model-level controls toward evidence-grade requirements on the knowledge layer? The incident in Canada suggests that is where the real gap is. The model produced plausible-sounding output. The governance failure was that nobody could show where the output came from.

That is not a model problem. It is a documentation and infrastructure problem. And it is one that most organizations have not yet seriously addressed.