Product
Can You Audit What You Have Deployed?
Over six months, which deliverables were produced with AI, how, under what control? Almost no one can answer.
Ask any legal department equipped with AI for eighteen months: for the last six months, which deliverables were produced with AI’s help, by which tool, from which sources, and with which validations? In almost every case, the answer is no. Not out of ill will, not out of lack of seriousness. By construction. The question seems harmless, almost administrative, and yet it touches a point most equipped organizations cannot address.
What makes this question so revealing is that it bears not on one matter, but on the whole. One does not ask “can you defend that opinion,” a question an attentive practitioner can often answer for a given matter. One asks “can you account for everything your organization produced with AI,” which supposes a consolidated view almost no current setup provides. The question jumps from the scale of the matter to the scale of the institution, and it is at that scale that the deficit appears.
Current legal AI tools were not designed to account at the scale of the organization. They were designed to answer users. They trace little, in formats of their own, in silos that do not communicate and aggregate nothing at the organization level. Each tool knows, at best, what it did itself; none knows what the whole produced, because none was designed to see beyond its own perimeter.
A tool designed to answer a user has no reason to know how to account to an organization.
The moment the question really arises
This inability stays theoretical until the day a leader needs, concretely, an overview. That day always comes, in varied forms: preparing an internal audit, answering a regulator, measuring real usage to arbitrate a renewal, checking the contractual compliance of a deployment, investigating an incident. In each of these cases, the question is no longer “is this tool good,” but “what can we say, collectively, of what we produced.” And that is where the information proves scattered, non-standardized, partially lost.
This deficit is becoming the number-one subject in serious AI committees. Not the performance of models, not the price of subscriptions: the auditability of what was deployed. The reversal of criteria is already at work. A year ago, a legal AI tender began with model capabilities. Today, the best specifications begin with another series of questions: can you trace, attribute, present, consolidate? A brilliant but opaque tool now fails at the step that, yesterday, came last.
One must measure the scale of this reversal, because it changes the definition of what a good tool is. Yesterday, a good tool was the one that produced the best answer; auditability, if mentioned, came at the end, in the annexes, as a box to tick. Today, in lucid organizations, it comes at the start, as an elimination criterion: a tool that cannot account at the scale of the organization is set aside before its performance is even evaluated. The order of criteria has inverted, and with it the hierarchy of vendors.
The first elimination criterion of this market will not be the quality of models. It will be auditability.
Consolidating, not merely recording
It will be objected that the tools produce activity logs, and that it would suffice to gather them. But gathering technical records does not produce an organizational view, for a reason of nature. What a leader must be able to present is not a list of machine events, it is an answer to organizational questions: on which matters was AI used, according to which methods, with which validations, and that answer must be consolidated at the scale of the whole institution, not scattered tool by tool. Piling up records does not produce this consolidated view, just as piling up statements does not produce accounting.
The difference lies in an organizational view requiring a consolidation the tools, by construction, do not provide. Each tool records what concerns it, in its format, on its perimeter; no one aggregates these fragments into a coherent image of what the organization produced. Yet it is precisely this coherent image that an audit, a regulator, a committee demand. Consolidation is not the sum of the records; it is a work of aggregation, standardization, coherence-making that no isolated tool has reason to do for the others.
This demand for consolidation is not a compliance whim, it is an operational requirement. An organization that cannot account, globally, for its use of AI cannot durably inscribe it in its processes, nor defend it before a client, a colleague or a regulator. The answer is not in the tools, none of which can consolidate what its neighbors do, but in an orchestration layer above the stack, that makes usage observable and consolidatable at the scale of the organization. This is precisely the function of MAX as a Legal Semantic Layer: to give the institution the overview no tool, alone, can produce.
Scattered records do not add up to an organizational view, as statements do not add up to accounting.
Why no tool can audit the others
One must explain why the solution cannot come from the tools themselves, for that is what makes the problem structural. A tool can account only for what it did; it has no visibility over what the other tools in the stack do, and no reason to have any. Asking each tool to audit itself produces as many partial reports as there are tools, in as many formats, with nothing linking them. One obtains fragments, never the overall image, because the overall image is precisely what no tool can see from its position.
This impossibility is not a passing defect vendors will correct. It owes to each tool’s position in the stack: it is a participant, not an observer. Yet auditing supposes placing oneself above what one observes, in a position that embraces the whole. A tool that is part of the stack cannot occupy that position, by definition; it would require an instance situated above the tools, seeing them all without being one of them. Auditability at scale is therefore not a feature a tool could acquire, it is a property of a layer that overlooks them.
This is why auditability is, by nature, a matter of layer and not of tool. It requires a point of view only a setup placed above the stack can occupy: the one that sees all interactions pass, whatever tool produced them, and records them in a common, queryable, consolidated format. Without this layer, each tool stays blind to the others, and the organization stays blind to the whole. With it, usage becomes observable as a whole, which is the very condition of the audit.
The border between personal and institutional use
This is why auditability is not one feature among others, but a condition of existence in an organization. A tool one cannot audit at scale can be used by an individual on their own account; it cannot be adopted by an institution that must answer for what it produces. The border between personal and institutional use of AI passes exactly there, on this ability to account collectively. An individual owes an account to themselves; an institution owes one to third parties, and this difference changes everything one expects of the tool.
As this criterion takes hold, it redraws the hierarchy of vendors. Tools brilliant in generation, but unable to account at scale, will find themselves relegated to the rank of personal accessories, while less spectacular but auditable solutions will become the base of serious deployments. The market will no longer reward performance alone; it will reward governed performance, the kind an organization can account for before whom it must.
This shift will separate organizations as much as vendors. Those that treated auditability as a detail to settle later will find themselves, the day the question arises, unable to answer, with years of production they cannot account for. Those that treated it as a condition of existence will have, from the start, laid the layer that makes their use of AI consolidatable, and will be able to answer, calmly, the question that puts the others in difficulty. Auditability is not what one adds at the end; it is that without which the rest does not hold in an organization.
An AI one cannot audit at scale is, in practice, an AI an organization cannot truly deploy.
