Skip to content
NDNicolas Desjardins, back to overview
Open-source toolkit · MIT · v1.0.0Released 19 November 2026

AI codebase audit

applied-ai-audit: a 7-pillar Applied AI and database audit for coding agents

A checklist your coding agent runs against your own repository. It reads the code and the database layer, then reports where AI calls are fragile, unsafe, slow or expensive, with the file and line of every finding. The prompt tells the agent to change nothing until you say which fixes to make.

Install it where you already work

The same prompt ships in four forms. Run the installer from the root of your project and it writes the file for your editor, or copy the file yourself: it is plain Markdown.

Installs to
.claude/commands/audit-ai.md
Run it
In Claude Code, type /audit-ai.
Install
curl -fsSL https://raw.githubusercontent.com/nickdesjardins/applied-ai-audit/main/scripts/install.sh | bash -s -- --claude

File contents

---
description: Run a 7-pillar Applied AI and Database architecture audit on this codebase
---

You are a Staff Applied AI Systems Architect and Database Performance Engineer. Your goal is to audit this entire codebase and its database layer to identify every place where Applied AI is used (or could be improved) across reliability, accuracy, security, latency, cost, and data architecture.

Do NOT modify any files yet. First inspect the repository structure, application code, prompts, model clients, tool definitions, database schemas, migrations, and queries, then produce a structured audit report.

### Operating Rules
1. Evidence First: Cite exact file paths, function names, and line numbers for every finding. Do not guess or invent issues that are not in the codebase.
2. Proportionality (No Over-Engineering): First classify the project's actual scope (for example: simple chatbot, structured extraction API, RAG knowledge engine, or multi-agent workflow). Only recommend patterns that fit the project's current complexity in the "Fix Now" tier; place heavier enterprise infrastructure (like Redis semantic caching, rerankers, or stateful agent frameworks) in the "Scale Later" roadmap unless the codebase already needs them.
3. Evaluate Both Codebase and Database: Treat AI reliability as a full-stack data and systems problem, not just prompt wording.

---

### Universal Applied AI & Database Audit Checklist

Evaluate the codebase and database against each of the following 7 pillars. Mark any pillar or check that does not apply to this project as `N/A` with a brief reason.

#### 1. Model Inference, Determinism & Output Contracts
- **Parameter Hygiene**: Are `temperature`, `top_p`, `max_tokens`, and reasoning/effort parameters explicitly configured per task (e.g., `temperature=0.0` for extraction/classification/routing vs. controlled creativity for chat)?
- **Structured Outputs & Schema Boundaries**: Where downstream code parses LLM responses, does it enforce strict JSON Schema / constrained decoding (Pydantic v2, Zod, or native structured outputs) instead of brittle regex or string parsing?
- **Schema Compatibility**: Are schema constraints engine-safe across providers (e.g., using `ge=1` / `minimum` rather than `gt=0` / `exclusiveMinimum` where parsers fail)?
- **Deterministic Post-Validation**: Does application code deterministically verify or recalculate fields LLMs are weak at (arithmetic, totals, dates, IDs, required codes) and normalize "nullish" artifacts (`"N/A"`, `"none"`, `"-"`, `"unknown"`) at the boundary?
- **Explicit Refusal / Negative-Out Paths**: When context or data is missing, is the model instructed and structurally able to refuse cleanly instead of hallucinating?

#### 2. Prompt Architecture, Context Window & Security
- **Prompt Prefix Caching Alignment**: Are static instructions, schemas, tool definitions, and stable context placed at the very beginning of the prompt payload (with dynamic timestamps, user IDs, and queries at the end) to maximize provider KV-cache hits?
- **Conversation & Context Hygiene**: For chatbots and multi-turn flows, how is message history bounded (sliding window, token budgeting, or rolling summarization)? Is there protection against unbounded context growth and "lost-in-the-middle" attention degradation?
- **Prompt Injection Defense**: Is untrusted user or document input fenced inside explicit delimiters (ideally with sanitized or randomized tags) and stripped of spoofed chat-template tokens or delimiter breakouts?
- **Input, Payload & Output Ceilings**: Are request body sizes, input character/token counts, and per-call output tokens capped before hitting the model?
- **Data Privacy, Auth & Error Hygiene**: Are API endpoints authenticated and rate-limited? Are secrets/PII kept out of logs and prompts? Are verbose upstream provider errors logged server-side while returning clean, sanitized error messages to clients?

#### 3. Reliability, Routing, Latency & Cost Economics
- **Non-Blocking Async & Shared Clients**: Are LLM/HTTP clients instantiated once at startup (avoiding per-request SSL/CA bundle blocking) and executed asynchronously with strict per-call timeouts?
- **Multi-Model Failover & Retry Discipline**: Is there a fallback chain across diverse providers/models for HTTP 429s, timeouts, withdrawn models, or invalid outputs? Are in-SDK retries disabled (`max_retries=0`) when chain-level failover handles retries so worst-case latency stays bounded?
- **In-Band Error Trapping & Stopping Rules**: Does the client inspect HTTP 200 payloads from aggregators (like OpenRouter) for embedded `error` objects? Do fallback loops have a stopping rule (such as a quorum) so bad/unparseable inputs do not burn latency across the entire chain?
- **Streaming & Perceived Latency**: For interactive chatbots, are responses streamed (SSE / WebSockets) rather than blocking until full generation finishes?
- **Complexity Routing & Caching**: Are trivial tasks (intent classification, title generation, query rewriting) routed to cheap/fast models instead of frontier models? For high-repeat workloads, is exact or semantic caching considered (with TTLs, prompt/model version namespaces, and a contrast guard to prevent serving cached answers to look-alike questions with different numbers/terms)?

#### 4. Database, Persistence & Data Architecture (Core DB Audit)
- **Connection Pooling & Resilience**: Are database connections pooled properly with bounded min/max sizes, startup retries, connect timeouts, and compatibility with serverless/transaction poolers (e.g., prepared statement settings)?
- **Chat & AI State Schema Design**: Are conversations, messages, token counts, model IDs, latency metrics, and feedback cleanly modeled with proper foreign keys, timestamps, and indexes (especially composite indexes on `(user_id, conversation_id, created_at)` or thread lookups)?
- **Safe Data Access & Multi-Tenancy**: Are all DB queries invoked by AI tools strictly parameterized (never executing raw LLM-generated SQL against production tables)? Are row-level security (RLS) or tenant/ACL permission filters enforced at query time so the AI can never leak another user's data?
- **Data Freshness, Idempotency & Sync**: For ingested content or AI-processed records, are content hashes used to prevent duplicate re-processing? Are updates and deletions synced cleanly so deleted source records cannot remain searchable or cached?

#### 5. Retrieval & Vector Database Optimization (RAG Audit)
*(Skip or mark N/A if the project does not use embeddings/RAG, or note if the chatbot would materially benefit from adding it.)*
- **Ingestion & Chunking**: Are documents split along structural boundaries (headings, sections, tables) with situational context prepended (Contextual Retrieval) rather than naive fixed character splits? Are tables/PDFs parsed cleanly?
- **Hybrid Search & Indexing**: Does retrieval combine dense vector search (`pgvector` HNSW with tuned `m` / `ef_construction`, or equivalent) with sparse lexical search (`tsvector` GIN / BM25) fused via Reciprocal Rank Fusion (`k=60`) so exact IDs, SKUs, and acronyms are not missed?
- **Reranking & Grounding**: Are initial candidates (e.g., top 30-50) trimmed by a cross-encoder reranker down to the top 3-5 passages before generation? Are citations or verbatim quotes verified/re-anchored against stored DB chunks server-side?
- **Embedding Lifecycle**: Are embedding model names, dimensions (including Matryoshka truncation if used), and corpus versions stored so upgrading an embedding model has a clean migration path?

#### 6. Tool Calling, Agents & Workflow Orchestration
*(Mark N/A if no tools or multi-step agents exist.)*
- **Framework Proportionality**: Are linear workflows and simple retry loops kept in plain native code rather than wrapped in heavy agent frameworks? If durable multi-turn checkpointing, cyclic self-correction, or async Human-in-the-Loop (HITL) pause/resume is required, is state persisted cleanly in the DB?
- **Tool Contracts & Safety**: Are tool inputs strictly typed (Pydantic/Zod) with clear field descriptions? Do destructive or external-mutating tools require explicit user confirmation or HITL approval gates?
- **Bounded Loops**: Do all agentic or self-correction loops enforce hard maximum iteration/revision limits to prevent runaway token spend?

#### 7. Observability, Evaluation & Testing
- **Telemetry & Cost Attribution**: Does the app record per-request traces/spans capturing prompt version, model used, fallback status, input/output tokens, estimated cost, and latency percentiles (p50/p95/p99)?
- **Hermetic Offline Tests**: Can the full AI integration layer (routing, timeouts, schema parsing, error handling, DB persistence) be tested deterministically offline in seconds using mocked providers/transports with zero API cost?
- **Empirical Evals & Regression Gates**: Is there a golden test dataset (covering normal queries, edge cases, refusals, and prompt injections) to benchmark prompt or model changes before shipping, rather than judging by vibes?

---

### Required Output Format

1. **System Discovery Summary**: Briefly list the detected tech stack, AI SDKs/models, database engine/ORM, and current AI architecture tier.
2. **Audit Findings Table**: Columns for `ID`, `Severity (Critical/High/Medium/Low)`, `Domain (Codebase/Database/Security/Cost & Latency/Observability)`, `Location (file:line)`, `Issue`, and `Concrete Fix`.
3. **Prioritized Action Plan**:
   - **Phase A - Quick Wins & Must-Haves (Current Scope)**: High-impact fixes appropriate right now without adding unnecessary dependencies.
   - **Phase B - Hardening & Reliability**: Next-step improvements for production readiness.
   - **Phase C - Scale-Later Capabilities**: Enterprise optimizations to defer until traffic, corpus size, or cost justifies them.
4. **Database-Specific Optimization Summary**: Highlight exact schema changes, missing indexes, query improvements, or state-modeling upgrades needed in the DB.

End by asking which items from Phase A I want you to implement first.

The problem it solves

Most AI features reach production as a prompt and a model call. The failures come later, and from elsewhere: a parser that breaks when the model adds a code fence, a provider outage with no fallback, user text pasted straight into the prompt, a query the model can steer, no record of what a call cost, and no test that shows a prompt change made the answers worse. A generic code review rarely looks for these, and a generic AI review tends to recommend every enterprise pattern it knows.

What you get back

A read-only report in four parts:

  1. 01System discovery: the stack, AI SDKs and models, the database engine and ORM, and the architecture tier it detected.
  2. 02A findings table: ID, severity, domain, file and line, the issue, and a concrete fix.
  3. 03A plan in three phases: A for quick wins in the current scope, B for hardening, C for capabilities to defer.
  4. 04A database summary: schema changes, missing indexes, query fixes and state-modelling upgrades.

It ends with a question instead of a pull request: which Phase A items to implement first.

Seven pillars, 29 checks

Each check asks one question about the code or the database. A check that does not apply is marked N/A with a reason, instead of being forced to fit.

  1. Pillar 015 checks

    Model Inference, Determinism & Output Contracts

    Model settings are chosen on purpose, answers are parsed against a strict schema, and the numbers models get wrong are recomputed in code.

  2. Pillar 025 checks

    Prompt Architecture, Context Window & Security

    Prompts are laid out so the provider can cache them, history and input sizes are capped, and untrusted text cannot pass itself off as instructions.

  3. Pillar 035 checks

    Reliability, Routing, Latency & Cost Economics

    Calls time out, fail over to another model, stream when a person is waiting, and go to a cheaper model when the task is easy.

  4. Pillar 044 checks

    Database, Persistence & Data Architecture

    Connections are pooled, AI state is modelled with keys and indexes, queries are parameterized, and a deleted record cannot stay searchable.

  5. Pillar 054 checks

    Retrieval & Vector Database Optimization

    Documents are split along their structure, search combines vectors and keywords, and quotes are checked against the stored text.

  6. Pillar 063 checks

    Tool Calling, Agents & Workflow Orchestration

    Agent frameworks appear only where the workflow needs them, tools are typed, risky actions wait for a person, and every loop has a limit.

  7. Pillar 073 checks

    Observability, Evaluation & Testing

    Each request is traced with its model, tokens and cost, the AI layer is tested offline, and a fixed set of questions guards every change.

Reference implementations

The checklist comes out of the four AI services on this site. Each case study shows some of its checks passing in working, public code.

    • Pillar 1: Output contracts
    • Pillar 2: Prompts and security
    • Pillar 3: Reliability and cost
    • Pillar 7: Observability and evals

    Invoice reader

    Invoice Extraction Service

    Strict schema decoding at temperature 0, totals recomputed to the cent, oversized request bodies refused before they are read, a cascade to a second provider on rate limits, timeouts or schema errors, and a test suite that cannot reach the network.

    Read the case study: Invoice reader
    • Pillar 5: Retrieval (RAG)
    • Pillar 1: Output contracts

    Ask your manuals

    Context-Aware Hybrid Compliance Retrieval Engine

    Heading-aware chunks with a contextual preamble, vector and keyword search fused in one PostgreSQL statement, a cross-encoder, every quote re-anchored to the stored text, and an explicit insufficient-context answer.

    Read the case study: Ask your manuals
    • Pillar 6: Agents and tools
    • Pillar 4: Database

    Supervised operations agents

    Autonomous Multi-Agent Operations and Marketing Pipeline

    A read-only, parameterized SQL tool, a bounded revision loop, and a PostgreSQL checkpoint that holds every run until a person approves, edits or rejects it.

    Read the case study: Supervised operations agents
    • Pillar 2: Prompts and security
    • Pillar 3: Reliability and cost
    • Pillar 7: Observability and evals

    AI quality gate

    Production LLMOps Gateway: Evaluation CI/CD and Latency Optimization

    A guard that neutralizes chat-template tokens and caps tokens, a semantic cache with a contrast guard, a complexity router with failover, Langfuse traces with cost, and an evaluation gate in CI.

    Read the case study: AI quality gate

Proportionate by design

Before it judges anything, the audit decides what kind of project it is looking at: a simple chatbot, a structured extraction API, a RAG knowledge engine or a multi-agent workflow. Recommendations that fit that scope go in Fix Now. Heavier infrastructure, such as a Redis semantic cache, a reranker or a stateful agent framework, goes in Scale Later unless the code already needs it.

  • Fix Now

    Changes that fit the project as it is today: model parameters, schemas, timeouts, input ceilings, parameterized queries, offline tests.

  • Scale Later

    Capabilities to add once traffic, corpus size or cost justifies them: semantic caching, reranking, durable agent state, a full evaluation pipeline.