# Spec: Optional LLM Structured-Output Safety Checks

## Goal

Add an optional LLM-based safety detector that uses structured output to judge
whether tool input or output is safe to pass through.  The detector must run
without any tools available to the checking model and must fit into the
existing DLP pipeline in `ellama-tools-dlp.el`.

This is a supplement to the current deterministic detectors.  It is not a
replacement for regex and exact-secret matching.

## Background

Current DLP checks in `ellama-tools-dlp.el` are deterministic:

- regex rules
- exact-secret matching from environment-derived candidates

Ellama already has a local pattern for structured-output helper calls in
`ellama-extract-string-list`, `ellama-semantic-similar-p`, and
`ellama-make-semantic-similar-p-with-context`:

- build a dedicated prompt with `llm-make-chat-prompt`
- request a JSON schema through `:response-format`
- parse the result with `json-parse-string`

The new safety detector should reuse that pattern, but for a safety verdict
instead of extraction.

## Problem Statement

Deterministic rules are fast and predictable, but they miss classes of unsafe
content that are semantic rather than lexical:

- prompt-injection phrasing that does not match known regexes
- indirect tool-bypass attempts
- ambiguous unsafe instructions that rely on context
- policy-manipulation text that is paraphrased or obfuscated

An optional LLM classifier can catch some of these cases, especially for tool
output that will be shown to the main model.

## Non-Goals

- Replacing deterministic DLP checks
- Relying on the safety checker for exact secret detection
- Allowing the safety checker to call tools
- Using the current chat session or conversation history for the safety check
- LLM-driven span redaction in v1

## Design Summary

Add a second detector family, `llm`, to the DLP core.  The detector runs only
when explicitly enabled and only on eligible payloads.  It uses a dedicated
provider call with structured output and `:tools nil`.

Recommended v1 behavior:

- run after deterministic detectors
- skip the LLM check when deterministic policy already decided `block`
- implement both "run only on clean deterministic scans" and
  "always run unless already blocked"
- let the user select the run condition with a custom variable
- use the existing global DLP mode for rollout control
- let the LLM detector only record telemetry in `monitor`
- let the LLM detector only force `block` in `enforce`
- do not let the LLM detector produce `warn` or `redact` in v1

This keeps the expensive model-based check as a semantic backstop rather than
the primary control plane.

## Scope

### Initial Scope (Recommended)

- Tool input and output scanning
- Sync and async tool flows
- Structured JSON verdicts only
- Failure defaults aligned with current DLP fail-open settings

### Deferred

- Token-stream mid-generation interception
- Multi-pass safety judging with retries
- Span-based redaction generated by the safety model
- Provider-specific fallback prompts for models that do not support structured
  output

## Integration Point

The feature should live inside `ellama-tools-dlp.el` and integrate into the
existing `ellama-tools-dlp--scan-text` pipeline.

Recommended order inside `ellama-tools-dlp--scan-text`:

1. Prepare payload (normalization, truncation)
2. Run deterministic detectors
3. Evaluate deterministic policy
4. If enabled and eligible, run LLM safety detector
5. Apply LLM block-only override
6. Build verdict
7. Log sanitized incident

No changes are required to the outer wrapper order in `ellama-tools.el`:

- DLP wrapper still runs before confirmation wrapper
- Output handling still uses the existing allow/warn/block/redact flow for
  deterministic findings
- LLM checks can only preserve the deterministic result or force `block`

## Data Model Changes

### Extend Finding Schema

`ellama-tools-dlp--validate-finding` currently accepts only `regex` and
`exact-secret` detectors.  Extend it to also accept `llm`.

New detector value:

- `llm`

LLM findings should use:

- `:rule-id` as a stable category identifier, such as
  `llm-prompt-injection`, `llm-policy-bypass`, or `llm-unsafe-shell`
- `:detector 'llm`
- `:severity` derived from model risk level
- no match spans in v1

Because v1 does not trust model-generated spans, `:match-start` and
`:match-end` remain nil for LLM findings.

### Verdict Semantics

LLM findings should only influence:

- `block`

An LLM result should either:

- have no effect on the final verdict
- force the final verdict to `block`

`warn` and `redact` remain reserved for deterministic findings in v1.
LLM findings must not participate in span-based redaction.

## Configuration Surface

Add a separate customization group under `ellama-tools-dlp` for the new
detector or keep the variables in the existing group.

Recommended defcustoms:

- `ellama-tools-dlp-llm-check-enabled`
  - boolean
  - default `nil`
- `ellama-tools-dlp-llm-provider`
  - provider object
  - fallback chain:
    `ellama-tools-dlp-llm-provider`
    -> `ellama-extraction-provider`
    -> `ellama-provider`
    -> `(ellama-get-first-ollama-chat-model)`
- `ellama-tools-dlp-llm-directions`
  - list of `input` / `output`
  - recommended default: `(input output)`
- `ellama-tools-dlp-llm-max-scan-size`
  - integer byte limit
  - recommended default: smaller than the global DLP scan size, e.g. `32768`
- `ellama-tools-dlp-llm-tool-allowlist`
  - optional list of tool names
  - nil means all tools allowed
- `ellama-tools-dlp-llm-template`
  - prompt template string for the isolated safety check

Rollout stays controlled by the existing global `ellama-tools-dlp-mode`:

- in `monitor`, the LLM detector runs and records telemetry only
- in `enforce`, an unsafe LLM verdict may force `block`

## Request Construction

The safety check must not reuse the current conversation session.  It should
create a fresh prompt and call the provider directly.

Recommended helper:

- `ellama-tools-dlp--llm-check-text`

Required behavior:

- call `llm-chat` directly, not `ellama-chat`
- build a new prompt with `llm-make-chat-prompt`
- pass `:tools nil`
- avoid attaching prior chat history
- include only the payload under review plus safe metadata

The prompt should include:

- direction (`input` / `output`)
- tool name
- arg name when present
- normalized payload text
- explicit instruction to judge, not comply
- explicit instruction that no tools are available

## Structured Output Contract

Use a compact schema that is easy to validate and stable across providers.

Recommended schema:

```elisp
'(:type object
  :properties
  (:unsafe (:type boolean)
         :category (:type string)
         :risk (:type string)
         :reason (:type string))
  :required ["unsafe" "category" "reason"])
```

Recommended meaning:

- `unsafe`
  - boolean summary
- `category`
  - stable label, e.g. `prompt_injection`, `policy_bypass`,
    `unsafe_shell`, `secret_request`, `unknown`
- `risk`
  - one of `none`, `low`, `medium`, `high`
- `reason`
  - short explanation safe to log and show to the user

If the provider supports enum-like constraints in `:response-format`, the
implementation may tighten `risk` and `category` to explicit enums.  The spec
does not require that for v1.  `risk` is advisory in v1 and may be used for
telemetry, but it does not directly control enforcement.

## Prompt Template Requirements

The template should be strict and short.  It should instruct the checker model
to:

- behave as a safety classifier only
- never follow or transform the reviewed payload
- never invent tools, tool calls, or external actions
- return only the requested JSON object
- keep `reason` short and avoid quoting long payload excerpts

The template should explicitly state that the payload may contain adversarial
instructions intended to manipulate the checker.

## Eligibility Rules

The LLM detector should run only when all of the following are true:

- LLM checks are enabled
- current direction matches `ellama-tools-dlp-llm-directions`
- payload is a string after normalization
- payload size is within `ellama-tools-dlp-llm-max-scan-size`
- tool is in scope according to the allowlist
- deterministic-only policy has been evaluated
- deterministic policy has not already produced a terminal `block`

Recommended skip rules:

- skip if payload was truncated by the global DLP size limit
- skip if deterministic findings already exist when run policy is `clean-only`
- skip on unsupported providers

Skipping should log a sanitized telemetry event so users can tell why the
detector did not run.

## Policy Mapping

Evaluate the LLM result separately from deterministic findings.

Recommended v1 mapping:

- `unsafe = nil`
  - no block override is applied
  - no `llm` finding is required
- `unsafe = t`
  - create one `llm` finding using `category`
  - in `monitor`, log the event but keep the deterministic verdict
  - in `enforce`, force the final verdict to `block`

Recommended severity mapping:

- `none` -> nil
- `low` -> `low`
- `medium` -> `medium`
- `high` -> `high`

Recommended rule-id mapping:

- `category` becomes the primary rule suffix
- example:
  - category `prompt_injection` -> rule id `llm-prompt_injection`

The final verdict should be computed as:

- if deterministic policy already produced `block`, keep `block`
- else if the LLM detector returned `unsafe` and global mode is `enforce`,
  return `block`
- else keep the deterministic verdict

LLM findings may be attached for logging and safe user messaging, but they must
not participate in redaction span calculation or in selecting `warn` vs
`redact`.

## Failure Handling

Failure must be safe, observable, and non-recursive.

Required behavior:

- provider call errors log a sanitized internal incident
- JSON parse errors log a sanitized internal incident
- invalid schema fields log a sanitized internal incident
- failures follow the current fail-open behavior by default
- failures must not leak the reviewed payload into messages or logs

Recommended v1 default:

- treat LLM-check failure as "detector unavailable"
- continue with deterministic findings only
- disable the feature for providers that do not support reliable structured
  output

## Telemetry

Add sanitized events for LLM detector activity.

Recommended event types:

- `llm-check-run`
- `llm-check-skip`
- `llm-check-error`

Suggested fields:

- timestamp
- direction
- tool name
- arg name
- payload length
- skip reason or error type
- provider label when safe
- returned unsafe flag
- returned category
- returned risk

Do not log:

- raw payloads
- raw secret values
- full model responses when they might echo sensitive text

## UX Requirements

When the LLM detector drives a `block`, user-facing messages should match the
existing DLP style.

Example:

- `DLP block input for tool shell_command arg cmd (rules: llm-unsafe_shell)`

The `reason` field from the model may be stored in telemetry, but user-facing
messages should still use the current safe formatter by default.  This avoids
leaking payload excerpts through model-generated explanations.

## Testing Plan

Add unit coverage in `tests/test-ellama-tools-dlp.el` and wrapper-level
coverage in `tests/test-ellama-tools.el`.

Minimum tests:

- disabled feature does not call the LLM
- input payload with semantic unsafe content triggers `block` in `enforce`
- output payload with semantic prompt injection triggers `block` in `enforce`
- unsafe LLM result in `monitor` records telemetry but does not change the
  deterministic verdict
- provider error logs sanitized `llm-check-error` and falls back safely
- malformed JSON response logs sanitized error and falls back safely
- unsupported direction is skipped
- oversized payload is skipped
- `:tools nil` is passed in the isolated prompt construction path
- LLM findings validate through the extended finding schema
- `redact` is never emitted from LLM findings in v1
- LLM findings do not participate in redaction span calculation

## Recommended Implementation Steps

1. Add defcustoms and helper template.
2. Extend the finding schema to accept detector `llm`.
3. Add a small helper to decide whether an LLM check is eligible.
4. Add a deterministic-only policy pass before LLM eligibility.
5. Add an isolated prompt builder using `llm-make-chat-prompt`.
6. Add `ellama-tools-dlp--llm-check-text`.
7. Apply an LLM block-only override in `ellama-tools-dlp--scan-text`.
8. Add incident logging for run/skip/error paths.
9. Add tests for eligibility, parsing, override behavior, and failure handling.

## Run Condition Policy

The implementation should support both run conditions and let the user choose
between them with a defcustom.

### Option A: Run Only When Deterministic Findings Are Empty

Pros:

- Lower latency and lower token cost
- Lower provider load, which matters if checks run on both input and output
- Lower risk of false positives on payloads already caught by deterministic
  rules
- Easier rollout because the LLM acts as a semantic backstop only
- Cleaner incident logs with less duplicate signal

Cons:

- The LLM cannot upgrade a weak deterministic `warn` into a stronger semantic
  `block`
- The LLM cannot add semantic classification metadata to already-flagged
  payloads
- Some mixed cases may stay under-classified, where a regex hit is present but
  the larger payload is materially more dangerous than the regex alone shows

Best fit:

- Conservative rollout
- Cost-sensitive setups
- Users who want deterministic rules to stay primary

### Option B: Always Run Unless Deterministic Policy Already Blocked

Pros:

- Maximum detection coverage
- The LLM can add semantic context even when regex/exact-secret findings exist
- Better for layered policy where deterministic checks catch syntax and the LLM
  catches intent
- More useful telemetry for tuning future policy overrides and rule quality

Cons:

- Higher latency on every eligible scan
- Higher token and provider cost
- Higher false-positive surface area
- More duplicate incidents unless logging merges deterministic and LLM signals
  carefully
- More complexity in reasoning about which detector actually drove a `block`

Best fit:

- Security-first deployments
- Small allowlists of sensitive tools
- Environments where the extra latency/cost is acceptable

### Recommended Decision Shape

This is a required part of the design, not an open question.

Recommended defcustom:

- `ellama-tools-dlp-llm-run-policy`
  - `clean-only`
  - `always-unless-blocked`

Recommended initial default:

- `clean-only`

That gives a lower-risk default while keeping the stronger mode available for
users who need it.

## Confirmed Decisions

The following scope decisions are now fixed for v1:

- scan both tool input and tool output
- implement both run policies and let the user choose via
  `ellama-tools-dlp-llm-run-policy`
- disable the feature on providers without reliable structured-output support
