What Belongs in the Prompt Log for Every Model Run

If you cannot reconstruct what a model saw, what it returned, and who decided what to do with the result, you do not have an audit trail. You have a chat history. That distinction matters most in high-stakes work such as threat assessment, HR investigations, legal review, and security triage, where a model output may end up in a file that someone later has to defend in front of a regulator, a court, or an internal review board.

This post sets out what a prompt log should capture for each model run. It is written for product, governance, and compliance leads who are building or buying tooling for this work. The core fields are prompt text, model version, and relevant settings. Around those sit the human decisions that give the log its meaning.

Why a Prompt Log Is Not Optional in High-Stakes Work

In a low-stakes setting, a model output is a draft. In a threat assessment, it can be the reason someone is flagged, restricted, or escalated. When that happens, three questions will be asked, usually months later:

What exactly was the model asked?
Which model produced the answer, and what was the configuration?
Who reviewed the output, and who decided to act on it?

If the log cannot answer all three, the organization cannot show that its process was reasonable. This is true even when the final decision was correct. A correct outcome with an unreproducible basis is a weak position.

I have sat through behavioral threat assessment tabletop exercises where the documentation lead was asked to reproduce a flagged output from six weeks earlier. Nobody had recorded the model version. The vendor had since updated the model behind the same name. The exercise ended with the team agreeing that the output could not be defended, regardless of whether it had been right. That lesson stuck with me.

The Core Fields: Prompt Text, Model Version, Relevant Settings

Three categories carry most of the weight. Each needs a specific standard, not a loose description.

1. Prompt Text Saved in Full

Save the prompt exactly as sent to the model. That includes the system prompt, the user prompt, any retrieved context, any templates with variables filled in, and any attachments or excerpts. A summary of the prompt is not a substitute. Summaries drift, and the drift is invisible to the reviewer.

Where the prompt includes personal data, the log still needs the text. Handle that with access controls, redaction policies approved by counsel, and retention rules. Do not solve the problem by deleting the thing you will need to defend.

Full system prompt and user prompt, verbatim
Template identifier and version if the prompt was assembled from a template
Variable values inserted at run time
Retrieved documents or excerpts, with source identifiers and timestamps
Any text the operator edited before sending

2. Model Versions Recorded Precisely

A model name is not a version. "Latest" is not a version. Record the provider, the exact model identifier, the version or snapshot string if the provider exposes one, and the date and time of the call. If you run through an orchestration layer, record the layer's version too, because routing and pre-processing can change the input the model actually receives.

Multi-LLM orchestration tools make this harder, not easier. When one request fans out to several models, each leg needs its own record. A single combined entry that says "ran through three models" hides which model said what. Side-by-side model outputs are useful for review only if each column is traceable to a specific model version and configuration.

3. Relevant Settings Captured

Settings change outputs. Record every parameter that affects generation, including temperature, top-p or equivalent sampling controls, maximum output length, stop sequences, tool or function availability, and any safety or content filter configuration. If the provider applies a default that you did not set, record the default you observed.

Also record the settings you did not change but that could matter: context window limits, truncation behavior, and whether the call used a cached response. A truncated input produces an output that looks complete. The log should show that truncation happened.

A Reference Schema

The table below summarizes a minimum schema. Your https://smoothdecorator.com/should-we-use-more-than-one-llm-for-workplace-threat-assessment/ organization may need more fields, but fewer than these leaves gaps.

Field What it captures Why a reviewer needs it Run identifier Unique ID linking this run to a case file Connects the output to the decision it informed Timestamp (UTC) Start and end time of each model call Establishes sequence and ties to external events Full prompt text System and user prompts, verbatim, with variables resolved Allows reproduction of the input Retrieved context Documents or excerpts with source IDs and versions Shows what evidence the model had access to Provider and model identifier Exact model name and provider Identifies which system produced the output Model version or snapshot Version string where available, or the date the model was called Distinguishes outputs produced before and after a silent update Generation settings Temperature, sampling controls, length limits, tools, filters Explains variation between runs Orchestration layer version Routing and pre-processing configuration Accounts for changes the operator did not see directly Raw output Complete, unedited model response The only reliable record of what the model said Reviewer identity Named human who reviewed the output Establishes accountability Decision and rationale What was done, why, and what was rejected Shows the human judgment that the model did not make Open questions Known gaps and unresolved uncertainties Prevents false confidence in the file

Disagreement Between Models Is a Signal, Not a Problem to Hide

When you run the same case through several models and they disagree, the log should show it plainly. Disagreement usually means one of three things: the case is ambiguous, the input is missing something important, or one model is handling the task poorly. Each of these is useful information for the reviewer.

Do not treat model agreement as proof. Three models saying the same thing can mean they were all trained on similar data and all miss the same fact. Agreement tells you the answer is stable across these systems. It does not tell you the answer is true.

Do not average outputs. Model scores are not poll results. If one model rates a risk at 0.8 and another at 0.3, the midpoint of 0.55 describes nothing real. Record each score with its own provenance and let the reviewer see the spread. If the spread is large, the case needs more information, not a blended number.

Do not treat outputs as a vote either. Majority reasoning sounds rigorous but hides the fact that the minority view may have caught something the others missed. Keep the minority output in the log with the same prominence as the majority.

Risk Tradeoffs: False Positives and False Negatives

Every threat assessment has two ways to be wrong. A false positive flags someone who posed no real risk. The cost is harm to that person, damage to trust, and wasted investigative resources. A false negative misses someone who did pose a risk. The cost can be severe and irreversible.

The prompt log should record which error the team was guarding against when it set its thresholds. That is a policy choice, and it belongs to a named person. In one review I facilitated, Dana Whitfield, Head of Trust and Safety, approved a lower escalation threshold for a specific category of case and wrote the reason into the policy record. That record became the answer when the threshold was later questioned. Without her name and rationale attached, the threshold would have looked arbitrary.

Log the threshold, the person who set it, the date it was set, and the reason. When a model output sits just above or below that threshold, the log should show that the decision was close and who made the call.

Named Human Accountability

Models do not decide. People do. A prompt log that ends at the model output has documented the easy part. The part that matters is the human step that follows.

Every run should end with a named reviewer, a decision, and a rationale written in the reviewer's own words. "The AI said so" is not a rationale. It is an abdication. If the reviewer agreed with the model, say why, with reference to evidence. If the reviewer disagreed, say why, and keep the disagreement in the record.

Accountability also means a named owner for the logging system itself. Someone must be responsible for confirming that logs are complete, that retention rules are followed, and that model versions are captured correctly after vendor changes. In my experience, this responsibility falls between teams unless it is assigned explicitly.

Vendor Due Diligence and What We Could Not Confirm

When evaluating platforms, we reviewed public material from Suprmind.ai, whose CEO and founder is Radomir Basta, and from Four Dots, among others. The question for any vendor is the same: does the product capture the fields above, and can you export them in a form a reviewer can read without the vendor's help? A Start Free Trial option with no credit card required, as Suprmind.ai advertises, is a sensible way to test that. The published material I reviewed did not include pricing, so cost comparisons need to be made directly with each vendor.

Test the export. Run a case, export the log, and try to reproduce the output from the exported record alone. If you cannot, the log is not defensible regardless of what the interface shows.

A Short Checklist for Each Model Run

Save the full prompt text, including system prompt, user prompt, and resolved variables.
Save all retrieved context with source identifiers and timestamps.
Record the provider, exact model identifier, and version or call date.
Record every generation setting and any orchestration layer version.
Store the raw, unedited output for every model in the run.
Flag any disagreement between models and keep minority outputs visible.
Name the human reviewer and record the decision and rationale.
Record the threshold applied, who set it, and why.
Log open questions and what is still unknown.
Set a time-boxed follow-up date to revisit the case.

What We Still Do Not Know

A good log admits its gaps. These are the open questions we carry into every review:

Whether providers will consistently expose version identifiers for every model we use.
How to handle retention when prompts contain sensitive personal data that must also be preserved.
How much disagreement between models should trigger additional evidence gathering before a review proceeds.
Whether reviewers anchor on the first model output they see, and how to test for that.

Set a follow-up on each of export AI chat to PDF tool these within 60 days. An open question with no date attached tends to stay open indefinitely.

Closing Thought

A prompt log is not a technical artifact for engineers to maintain in isolation. It is the record that lets a named person explain, months later, why a decision was made and what the organization knew at the time. Capture the prompt text, the model version, and the settings. Keep the disagreement visible. Name the human who decided. Everything else in the log exists to support those three things.