Proxy¶

How to run it: the proxy guide.

A policy-enforcing proxy for OpenAI-compatible model endpoints.

Tools send their requests to the proxy instead of the model endpoint. For each request it applies the policy for the requested endpoint - allow, redact, tokenise or block - records the decision in the audit log, forwards what may be sent, and puts tokenised values back into the answer. It fails closed: if anything goes wrong before the request is forwarded, nothing is sent.

Endpoints are the policy's destinations, and each needs a url (the OpenAI-compatible base URL, e.g. http://vllm.internal:8000/v1). Clients ask for "<destination>/<model>", e.g. "vllm-gpu/llama-3.1-70b". The proxy holds the endpoints' own API keys (CREDGATE_UPSTREAM_KEY_<NAME> environment variables, e.g. CREDGATE_UPSTREAM_KEY_VLLM_GPU); clients never see them.

Two modes:

localhost (default): listens on 127.0.0.1; requests are recorded as the operating-system user running the proxy.
shared: listens on a network address and requires a keys file; each client sends Authorization: Bearer <key> and is recorded by name. Add users with :func:add_user (credgate proxy add-user). Use TLS (certfile/keyfile) or a TLS-terminating reverse proxy.

Three APIs, each forwarded to an endpoint that speaks the same API:

OpenAI chat completions: POST /v1/chat/completions, GET /v1/models. Forwarded to <url>/chat/completions.
Anthropic messages: POST /v1/messages and /v1/messages/count_tokens (Claude Code, the Anthropic SDKs). Forwarded to <url>/messages, with the endpoint's key as x-api-key.
Ollama: POST /api/chat, /api/generate, /api/show and GET /api/tags, /api/version (Ollama clients, VS Code Copilot Chat with Ollama models). Forwarded to the endpoint's URL without its /v1.

Answers stream as the model writes them ("stream": true; Ollama streams unless told not to). Tokens are put back as they arrive: a piece of text that could still be the start of a token is held back until it is complete.

Two servers:

standard (default): the Python standard library only - nothing to install, for one person on their own machine.
uvicorn (server="uvicorn"; pip install 'credgate[server]'): a production HTTP server (BSD-3-licensed) for a shared proxy: keep-alive, a limit on requests in progress (concurrency; more get 503), and a client that leaves is noticed even while the model is still thinking.

Both run the same code for everything else - policy, audit, streaming - and the same tests.

ProxyConfig `dataclass` ¶

ProxyConfig(policy: Policy | str | PathLike[str], audit: str | PathLike[str], host: str = '127.0.0.1', port: int = 8790, keys: str | PathLike[str] | None = None, certfile: str | None = None, keyfile: str | None = None, token_key: str | bytes | None = None, audit_key: str | bytes | None = None, timeout: float = 300, workers: int = 1, drain_seconds: float = 5, audit_rotate: bool = True, audit_retention_years: int = 5, default_endpoint: str | None = None, server: str = 'standard', concurrency: int = 256, watch: float | None = None, require_tests: bool = False, users: str | PathLike[str] | None = None)

Settings of a proxy.

Parameters:

Name	Type	Description	Default
`policy`	`Policy \| str \| PathLike[str]`	The policy, or a path to its YAML file.	required
`audit`	`str \| PathLike[str]`	The audit log file (JSON Lines); every request is recorded.	required
`host`	`str`	Address to listen on; `127.0.0.1` for localhost mode.	`'127.0.0.1'`
`port`	`int`	Port to listen on.	`8790`
`keys`	`str \| PathLike[str] \| None`	Keys file (see :func:`add_user`); required unless `host` is a loopback address.	`None`
`certfile`	`str \| None`	TLS certificate (PEM) for HTTPS; with `keyfile`.	`None`
`keyfile`	`str \| None`	TLS private key (PEM).	`None`
`token_key`	`str \| bytes \| None`	Key for the tokens sent to models (hex or bytes); default `CREDGATE_TOKEN_KEY`, otherwise a random key per proxy process.	`None`
`audit_key`	`str \| bytes \| None`	Key for hashing identifiers in the audit log; default `CREDGATE_AUDIT_KEY`.	`None`
`timeout`	`float`	Seconds to wait for an endpoint.	`300`
`workers`	`int`	Processes serving requests (Linux and macOS). Detection is CPU work, and one Python process uses one core for it; with several workers the proxy uses several cores.	`1`
`audit_rotate`	`bool`	Start a new audit file each month (see :class:`~credgate.AuditLog`).	`True`
`audit_retention_years`	`int`	Remove monthly audit files older than this (at least 1; 0 keeps them all), at start and once a day.	`5`
`default_endpoint`	`str \| None`	The endpoint for models asked for without one (no `<endpoint>/`), e.g. `claude` so that Claude Code works with its own model names.	`None`
`server`	`str`	`"standard"` (the Python standard library) or `"uvicorn"` (production HTTP server; needs `credgate[server]`).	`'standard'`
`concurrency`	`int`	With uvicorn, the requests one worker handles at once; more are answered `503` so a load balancer can try elsewhere.	`256`
`drain_seconds`	`float`	On SIGTERM, how long to keep serving while reporting "draining" on `/healthz`, so a load balancer stops routing here; then requests in progress are finished and the proxy exits.	`5`
`watch`	`float \| None`	Check the policy, its custom detector files and the keys file every this many seconds, and reload them when they change (see :func:`reload`). `None`: reload only on SIGHUP.	`None`
`require_tests`	`bool`	Refuse a policy without a tests file beside it (`<policy>.tests.yaml`), at start and on reload. A tests file that exists is always run, and a failing test always refuses the policy.	`False`
`users`	`str \| PathLike[str] \| None`	The app's users file (see :mod:`credgate.users`): a key whose owner is not in it, or is inactive, is refused, so a leaver loses access when they are deactivated. Reloaded like the keys file.	`None`

serve ¶

serve(config: ProxyConfig, log: Callable[[str], Any] | None = None) -> None

Run the proxy until interrupted.

make_server ¶

make_server(config: ProxyConfig, log: Callable[[str], Any] | None = None) -> Any

The proxy's HTTP server, not yet serving (port 0 picks a free port): serve_forever() runs it, shutdown() stops it.

reload ¶

reload(config: ProxyConfig, log: Callable[[str], Any] | None = None) -> bool

Reload the policy and the keys file of a proxy if they changed.

The policy is replaced only by a new revision that loads, suits the proxy (endpoints) and passes its tests; a change to the policy or its custom detector files without a new revision is refused, so what runs is always a recorded revision. An older revision (a rollback) is accepted and logged. The keys file is replaced if it reads and has users. On any refusal the running policy and keys stay.

Returns:

Type	Description
`bool`	Whether anything was replaced.

Users and keys¶

The proxy's keys file: who may use a shared proxy (spec/README.md, "Proxy keys file").

Each user has one key. The file stores a SHA-256 hash of it (the key itself is shown once, when it is made), when it was made and when it expires. A key that has expired is refused by the proxy; rotating gives the user a new one and voids the old. Files written before keys expired (a bare hash per user) still load, as keys that do not expire.

KeyEntry `dataclass` ¶

KeyEntry(hash: str, created: str | None = None, expires: str | None = None)

One user's key, as stored: its hash and its dates (ISO 8601, UTC).

AccessEntry `dataclass` ¶

AccessEntry(time: str, by: str, action: str, user: str, expires: str | None = None)

One change to who may use the proxy: a key made, rotated or removed.

add_user ¶

add_user(keys: str | PathLike[str], name: str, days: int = DEFAULT_DAYS) -> str

Add a user to a keys file (created if missing) and return their new key, valid for days (1 to 180). Only a hash of the key is stored; the key is shown once.

rotate_key ¶

rotate_key(keys: str | PathLike[str], name: str, days: int = DEFAULT_DAYS) -> str

Give a user a new key, valid for days; their old key stops working at the proxy's next reload. Returns the new key (shown once).

remove_user ¶

remove_user(keys: str | PathLike[str], name: str) -> None

Remove a user; their key stops working at the proxy's next reload. The last user cannot be removed: a shared proxy with no users would refuse everyone.

list_users ¶

list_users(keys: str | PathLike[str], now: datetime | None = None) -> list[dict[str, Any]]

The users of a keys file with their key's dates and status (never the keys or their hashes): "active", "expires soon" (within 7 days), "expired" or "no expiry".

expire_unused ¶

expire_unused(keys: str | PathLike[str], audit: str | PathLike[str], days: int = 60, now: datetime | None = None) -> list[str]

Expire the keys of users who have not used the proxy for days days.

The last use of each key is read from the proxy's audit log (its monthly files included): a request recorded under the user's name. A key made within the last days days is left alone. An expired key stays in the file, so the person can be given a new one with rotate_key. Returns the names whose keys were expired.

read_keys ¶

read_keys(path: str | PathLike[str]) -> dict[str, KeyEntry]

The users of a keys file; raises ValueError if it is not one.

log_access ¶

log_access(keys: str | PathLike[str], by: str, action: str, user: str, expires: str | None = None) -> None

Record a key change in the access log beside the keys file: when, by whom, what (create, rotate or remove), for whom and until when - never the key. Hash-chained like the audit log.

read_access ¶

read_access(keys: str | PathLike[str]) -> tuple[list[AccessEntry], bool]

The access log beside a keys file, oldest first, and whether its hash chain is unbroken. Reading stops at the first broken entry.

access_path ¶

access_path(keys: str | PathLike[str]) -> Path

The access log kept beside a keys file (keys.access.jsonl for keys.json).