We red-teamed our own PHI guardrail, and the logs were the problem.

The verdict is trimmed before it leaves the building. The model returns findings, an explanation and the encoding it spotted. The service logs all of that and returns three fields. An attacker who cannot see which signal fired cannot tune against it, and the blue team keeps the detail it needs to improve. Same computation, two audiences, deliberately different views.

An integrity canary catches a successful injection. The model is required to echo a fixed token in its response. If the token is missing or wrong, the service downgrades the verdict to review and appends a warning, on the reasoning that a response which has stopped following its instructions cannot be trusted when it says allow. This is a genuinely good control and it is the one we would keep in a client system.

Unreadable output fails closed. If the model returns something that is not valid JSON, the service does not guess and does not pass it through. It returns review with low confidence and the parse error attached. Schema validation is strict on top of that: an out-of-range confidence or action raises rather than being coerced into the nearest legal value.

Every one of those is a decision about what happens when the model is wrong. That is what we mean by socio-technical, and it is not our phrase: the NIST AI Risk Management Framework states that AI systems are inherently socio-technical in nature^[01], adopting the dimensional breakdown the OECD published in 2022^[02].

Finding one: the request log stores the data the system exists to protect. Every request is written to a CSV with an input_preview field holding the first eighty characters of the submitted text, verbatim. For a service whose entire purpose is preventing patient identifiers from travelling, the log is now the largest concentration of patient identifiers in the deployment, sitting on a mounted volume that is explicitly designed to survive redeploys. The detector works. The audit trail of the detector working is the leak.

This is the failure mode that a control-by-control review does not surface, because every control is individually correct. Logging requests is right. Retaining logs across deploys is right. Truncating a preview is right. The defect only appears when you ask what the system holds when all three are true at once.

Finding two: hashing a low-entropy identifier does not de-identify it. Alongside the preview, each row carries a SHA-256 of the input truncated to sixteen hex characters. For free text that is a reasonable correlation key. For a ten-digit national identifier it is not a de-identification measure at all: the entire input space is ten billion candidates, which is a rounding error to brute force. A hash of a small set is a lookup table waiting to be built.

Both findings were remediated on 24 August 2026. The verbatim preview is gone, replaced by a structural summary: length, digit and letter counts, the longest run of consecutive digits, and whether the payload looks base64 encoded. That keeps what the blue team actually debugs with, whether a ten-digit run arrived and whether it was encoded, and discards everything that could reconstruct an identifier. The correlation hash is now an HMAC keyed with a per-deployment salt, so a reader holding the log can no longer brute force the identifier back out of it.

The salt costs something, and it is worth naming rather than burying: hashes no longer correlate across deployments, or across a restart if the salt is not pinned in the environment. Correlation inside a single run is what the exercise needed, so an ephemeral salt is the default and the service still starts without configuration.

One residue, disclosed rather than quietly closed. Rows written before the fix still carry the old previews. The service now retires that file rather than appending to it, because the column layout changed and silently misaligning every field afterwards would be worse than either format alone. Retiring it is not the same as disposing of it, and what happens to the retired file is the operator's decision, not the code's.

One scope limit, stated plainly. This note records design and code findings. It is not a statement about how the deployed service behaves under load, and the test pack is a functional suite rather than a statistical evaluation, so nothing here supports a claim about detection rates. Under ISAE 3000 the distinction between reasonable and limited assurance turns on exactly this kind of scope statement^[03], and a note that blurs it is marketing.

Audit work is confidential. A firm our size has no portfolio of signed engagements it is free to describe, and the industry's usual answer to that is an invented case study or a testimonial nobody can check. We would rather show the method on something we built, where we can name the defects without asking anyone's permission.

It also demonstrates the thing that is hardest to demonstrate: that we will write down what we found when the finding is inconvenient. Confidentiality and professional secrecy are obligations we take from Article 31 of the AI Act^[04], and the ethical framework we work to treats a self-interested reason to soften a finding as a threat to be addressed rather than a matter of taste^[05].