AI Vocabulary
Determinism: why does the same question give two different answers?
A deterministic system always produces the same result from the same inputs. Language models are not: they select their words among several probable options, which produces variation between attempts. That property is not a manufacturing defect, it is normal operation.
Where the variation comes from
At each step the model has a range of possible continuations with their probabilities. It picks one, and that choice includes an adjustable element of randomness. Two runs therefore take slightly different paths, which then diverge.
Less familiar causes add to this. Computing infrastructure introduces minute rounding differences between machines. And the provider may replace the model version without notice, changing behaviour overnight.
What this implies in practice
Two colleagues asking the same question get two different answers. Each believes they framed their request badly, and neither is right or wrong.
A test that succeeded yesterday may fail today with nothing having changed on your side. A result obtained during a demonstration will not necessarily reproduce the next day.
Above all, a one-off verification guarantees nothing about later uses. Checking a deliverable establishes the quality of that deliverable, not of the system.
An example that puzzles teams
A firm approves a use after three convincing attempts on non-compete clauses. Three weeks later a colleague obtains a markedly poorer result on a comparable clause, and reports that the tool has deteriorated.
Nothing has deteriorated. The first three attempts took favourable paths; the fourth took another, statistically normal one. Three attempts do not measure a non-deterministic system, they give three samples of it.
A practical consequence follows for evaluating a tool: an isolated trial, favourable or not, tells you nothing. The same task must be repeated several times to observe the spread, which almost nobody does during a demonstration.
What it does not solve
Making a system strictly deterministic does not make it accurate. It will produce the most probable error with perfect regularity, which is easier to detect and no less wrong.
And complete determinism remains out of reach in practice: even with the setting fixed, infrastructure variation and version changes persist.
What to require instead
Not identical answers, which is not achievable, but stability of what matters. Two runs may word things differently without diverging on substance: same sources retained, same points flagged, same conclusion.
The question to put to a provider is therefore precise: on the same task repeated, what is guaranteed stable, and what is not? A serious provider distinguishes the two; an approximate one answers that the system is reliable.
It is also why a verification regime should bear on categories of deliverable rather than on use cases validated once and for all.
This also explains why a demonstration is a poor basis for a decision: it shows one sample, chosen or not, of a distribution nobody has measured.
Why it matters to a lawyer
Because law runs on reproducibility: the same situation calls for the same answer, and a firm treating two identical matters differently would have a problem.
A non-deterministic tool therefore introduces variability your organisation must absorb, through rules of verification rather than through confidence in the tool's constancy.