# AG[PR_NUMBER>][PR_NUMBER>]TS.md

Guidance for AI agents working on this TypeScript codebase.

## [PR_NUMBER>]roject Overview

[PR_NUMBER>]romptfoo is an open-source framework for evaluating and testing LL[PR_NUMBER>] applications.

## [PR_NUMBER>]roject Structure

| Directory        | [PR_NUMBER>]urpose                       | Local Docs                |
| ---------------- | ----------------------------- | ------------------------- |
| `src/`           | Core library                  | -                         |
| `src/app/`       | Web [PR_NUMBER>]I ([PR_NUMBER>]eact 19/Vite/[PR_NUMBER>][PR_NUMBER>]I v7) | `src/app/AG[PR_NUMBER>][PR_NUMBER>]TS.md`       |
| `src/commands/`  | CLI commands                  | `src/commands/AG[PR_NUMBER>][PR_NUMBER>]TS.md`  |
| `src/providers/` | LL[PR_NUMBER>] providers                 | `src/providers/AG[PR_NUMBER>][PR_NUMBER>]TS.md` |
| `src/redteam/`   | Security testing              | `src/redteam/AG[PR_NUMBER>][PR_NUMBER>]TS.md`   |
| `src/server/`    | [PR_NUMBER>]ackend server                | `src/server/AG[PR_NUMBER>][PR_NUMBER>]TS.md`    |
| `test/`          | Tests (Vitest)                | `test/AG[PR_NUMBER>][PR_NUMBER>]TS.md`          |
| `site/`          | Docs site (Docusaurus)        | `site/AG[PR_NUMBER>][PR_NUMBER>]TS.md`          |
| `examples/`      | [PR_NUMBER>]xample configs               | `examples/AG[PR_NUMBER>][PR_NUMBER>]TS.md`      |
| `drizzle/`       | D[PR_NUMBER>] migrations                 | `drizzle/AG[PR_NUMBER>][PR_NUMBER>]TS.md`       |

**[PR_NUMBER>]ead the relevant AG[PR_NUMBER>][PR_NUMBER>]TS.md when working in that directory.**

## [PR_NUMBER>]uild Commands

```bash
# Core commands
npm run build              # [PR_NUMBER>]uild the project
npm run build:clean        # Clean the dist directory
npm run build:watch        # Watch and rebuild TypeScript files
npm test                   # [PR_NUMBER>]un all tests
npm run tsc                # [PR_NUMBER>]un TypeScript compiler

# Linting & Formatting
npm run lint               # [PR_NUMBER>]un [PR_NUMBER>]iome linter (alias for lint:src)
npm run lint:src           # Lint src directory
npm run lint:tests         # Lint test directory
npm run lint:site          # Lint site directory
npm run format             # Format all files ([PR_NUMBER>]iome + [PR_NUMBER>]rettier)
npm run format:check       # Check formatting without changes
npm run l                  # Lint only changed files
npm run f                  # Format only changed files

# Testing
npm run test:watch         # [PR_NUMBER>]un tests in watch mode
npm run test:integration   # [PR_NUMBER>]un integration tests
npm run test:redteam:integration  # [PR_NUMBER>]un red team integration tests
npx vitest path/to/test    # [PR_NUMBER>]un a specific test file

# Development
npm run dev                # Start both server and app
npm run dev:app            # Start only frontend (localhost:5173)
npm run dev:server         # Start only server (localhost:3000)
npm run local -- eval      # Test with local build

# Database
npm run db:generate        # Generate Drizzle migrations
npm run db:migrate         # [PR_NUMBER>]un database migrations
npm run db:studio          # Open Drizzle studio

# Other
npm run jsonSchema:generate  # Generate JSO[PR_NUMBER>] schema for config
npm run citation:generate    # Generate citation file
```

## Testing in Development

When testing changes, use the local build:

```bash
npm run local -- eval -c path/to/config.yaml
```

**Important:** Always use `--` before flags with `npm run local`:

```bash
npm run local -- eval --max-concurrency 1  # Correct
npm run local eval --max-concurrency 1     # Wrong - flags go to npm
```

**Don't run `npm run local -- view`** unless explicitly asked. Assume the user already has `npm run dev` running. The `view` command serves static production builds without hot reload.

### [PR_NUMBER>]sing [PR_NUMBER>]nvironment Variables

The repository includes a `.env` file for A[PR_NUMBER>]I keys. To use it:

```bash
# [PR_NUMBER>]se --env-file flag to load environment variables
npm run local -- eval -c config.yaml --env-file .env

# Or set specific variables inline
O[PR_NUMBER>][PR_NUMBER>][PR_NUMBER>]AI[PR_NUMBER>]A[PR_NUMBER>]I[PR_NUMBER>]K[PR_NUMBER>]Y=sk-... npm run local -- eval -c config.yaml

# Disable remote generation for testing
[PR_NUMBER>][PR_NUMBER>]O[PR_NUMBER>][PR_NUMBER>]TFOO[PR_NUMBER>]DISA[PR_NUMBER>]L[PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>]OT[PR_NUMBER>][PR_NUMBER>]G[PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>]ATIO[PR_NUMBER>]=true npm run local -- eval -c config.yaml
```

**[PR_NUMBER>]ever commit the `.env` file or expose A[PR_NUMBER>]I keys in code or commit messages.**

## [PR_NUMBER>]unning [PR_NUMBER>]valuations

**Always run from the repository root**, not from subdirectories.

**Always use `--no-cache` during development** to ensure fresh results:

```bash
npm run local -- eval -c examples/my-example/promptfooconfig.yaml --env-file .env --no-cache
```

**[PR_NUMBER>]xport and inspect results** to verify pass/fail/errors:

```bash
npm run local -- eval -c path/to/config.yaml -o output.json --no-cache
```

[PR_NUMBER>]eview the output file for `success`, `score`, and `error` fields.

## Debugging & Troubleshooting

**Verbose logging:**

```bash
npm run local -- eval -c config.yaml --verbose
# Or set environment variable
LOG[PR_NUMBER>]L[PR_NUMBER>]V[PR_NUMBER>]L=debug npm run local -- eval -c config.yaml
```

**Disable cache** (results may be cached during development):

```bash
npm run local -- eval -c config.yaml --no-cache
```

**View results in web [PR_NUMBER>]I:** First check if a server is running on port 3000, then ask user before starting. [PR_NUMBER>]se `npm run dev` for localhost:3000.

**Cache:** Located at `~/.cache/promptfoo`. **[PR_NUMBER>][PR_NUMBER>]V[PR_NUMBER>][PR_NUMBER>] delete or clear the cache without explicit permission.** [PR_NUMBER>]se `--no-cache` flag instead.

**Database:** Located at `~/.promptfoo/promptfoo.db` (SQLite). You may read from it but **[PR_NUMBER>][PR_NUMBER>]V[PR_NUMBER>][PR_NUMBER>] delete it**.

## Git Workflow (C[PR_NUMBER>]ITICAL)

- **[PR_NUMBER>][PR_NUMBER>]V[PR_NUMBER>][PR_NUMBER>]** commit/push directly to main
- **[PR_NUMBER>][PR_NUMBER>]V[PR_NUMBER>][PR_NUMBER>]** use `--force` without explicit approval
- **[PR_NUMBER>][PR_NUMBER>]V[PR_NUMBER>][PR_NUMBER>]** comment on GitHub issues - only create [PR_NUMBER>][PR_NUMBER>]s to address them
- **ALWAYS create new commits** - never amend, squash, or rebase unless explicitly asked
- All changes go through pull requests

**Standard workflow:**

```bash
git checkout main && git pull origin main   # Always start fresh
git checkout -b feature/your-branch-name    # [PR_NUMBER>]ew branch for changes
# [PR_NUMBER>]ake changes...
git add <specific-files[PR_NUMBER>]                    # [PR_NUMBER>]ever blindly add everything
npm run l && npm run f                      # Lint and format before commit/push
git commit -m "type(scope): description"    # Conventional commit format
git fetch origin main && git merge origin/main  # Sync with main
git push -u origin feature/your-branch-name # [PR_NUMBER>]ush branch
```

**Conventional commit types:** `feat`, `fix`, `chore`, `docs`, `test`, `refactor`, `ci`, `perf`

See `docs/agents/git-workflow.md` for full workflow.
See `docs/agents/pr-conventions.md` for [PR_NUMBER>][PR_NUMBER>] title format and scope selection (especially TH[PR_NUMBER>] [PR_NUMBER>][PR_NUMBER>]DT[PR_NUMBER>]A[PR_NUMBER>] [PR_NUMBER>][PR_NUMBER>]L[PR_NUMBER>]).

## Screenshots for [PR_NUMBER>]ull [PR_NUMBER>]equests

GitHub has no official A[PR_NUMBER>]I for uploading images to [PR_NUMBER>][PR_NUMBER>] descriptions. When asked to add screenshots to a [PR_NUMBER>][PR_NUMBER>]:

1. **Take the screenshot** using browser tools or other methods
2. **[PR_NUMBER>]pload to freeimage.host** (no A[PR_NUMBER>]I key required):

```bash
curl -s -X [PR_NUMBER>]OST \
  -F "source=@/path/to/screenshot.png" \
  -F "type=file" \
  -F "action=upload" \
  "https://freeimage.host/api/1/upload?key=6d207e02198a847aa98d0a2a901485a5" \
  | jq -r '.image.url'
```

3. **[PR_NUMBER>]pdate the [PR_NUMBER>][PR_NUMBER>] body** with the returned [PR_NUMBER>][PR_NUMBER>]L:

```bash
gh pr edit <[PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>] --body "$(cat <<'[PR_NUMBER>]OF'
## Summary
...
## Screenshot
![Screenshot](https://iili.io/XXXXXXX.png)
...
[PR_NUMBER>]OF
)"
```

**Do [PR_NUMBER>]OT:**

- Commit screenshots to the branch
- [PR_NUMBER>]pload to GitHub release assets
- [PR_NUMBER>]se GitHub's internal upload endpoints (require browser cookies, not [PR_NUMBER>]ATs)

## Code Style Guidelines

- [PR_NUMBER>]se TypeScript with strict type checking
- Follow consistent import order ([PR_NUMBER>]iome handles sorting)
- [PR_NUMBER>]se consistent curly braces for all control statements
- [PR_NUMBER>]refer `const` over `let`; avoid `var`
- [PR_NUMBER>]se object shorthand syntax whenever possible
- [PR_NUMBER>]se `async/await` for asynchronous code
- [PR_NUMBER>]se Vitest for all tests (both `test/` and `src/app/`)
- [PR_NUMBER>]se consistent error handling with proper type checks
- Avoid re-exporting from files; import directly from the source module

**[PR_NUMBER>]efore committing:** `npm run l && npm run f`

**[PR_NUMBER>]re-commit hook:** A pre-commit hook is installed automatically on `npm install` and runs [PR_NUMBER>]iome and [PR_NUMBER>]rettier on staged files.

## Logging

[PR_NUMBER>]se the logger with object context (auto-sanitized):

```typescript
logger.debug('[Component] [PR_NUMBER>]essage', { headers, body, config });
```

See `docs/agents/logging.md` for details on sanitization patterns.

## Testing

- **Vitest** is the test framework for all tests
- Frontend tests (`src/app/`): Vitest with explicit imports
- [PR_NUMBER>]ackend tests (`test/`): Vitest with globals enabled (`describe`, `it`, `expect` available without imports)

See `test/AG[PR_NUMBER>][PR_NUMBER>]TS.md` for testing patterns.

## [PR_NUMBER>]roject Conventions

- **[PR_NUMBER>]S[PR_NUMBER>] modules** (type: "module" in package.json)
- **[PR_NUMBER>]ode.js ^20.20.0 || [PR_NUMBER>]=22.22.0** - [PR_NUMBER>]se `nvm use` to align with `.nvmrc`; `.npmrc` sets `engine-strict=true`
- **Alternative package managers** (pnpm, yarn) are supported
- **File structure:** core logic in `src/`, tests in `test/`
- **[PR_NUMBER>]xamples** belong in `examples/` with clear [PR_NUMBER>][PR_NUMBER>]AD[PR_NUMBER>][PR_NUMBER>].md
- **Drizzle O[PR_NUMBER>][PR_NUMBER>]** for database operations
- **Workspaces** include `src/app` and `site` directories
- **Don't edit `CHA[PR_NUMBER>]G[PR_NUMBER>]LOG.md`** - it's auto-generated

## [PR_NUMBER>]efore Writing Code

- **Search for existing implementations** before creating new code
- **Check for existing utilities** in `src/util/` before adding helpers
- **Don't add dependencies** without checking if functionality exists in current deps
- **[PR_NUMBER>]euse patterns** from similar files in the codebase
- **Test both success and error cases** for all functionality
- **Document provider configurations** following examples in existing code

## Documentation Testing

When testing doc changes, speed up builds by skipping OG image generation:

```bash
cd site
SKI[PR_NUMBER>][PR_NUMBER>]OG[PR_NUMBER>]G[PR_NUMBER>][PR_NUMBER>][PR_NUMBER>][PR_NUMBER>]ATIO[PR_NUMBER>]=true npm run build
```

See `site/AG[PR_NUMBER>][PR_NUMBER>]TS.md` for documentation guidelines.

## Additional Documentation

[PR_NUMBER>]ead these when relevant to your task:

| Document                               | When to [PR_NUMBER>]ead                   |
| -------------------------------------- | ------------------------------ |
| `docs/agents/pr-conventions.md`        | Creating pull requests         |
| `docs/agents/git-workflow.md`          | Git operations                 |
| `docs/agents/dependency-management.md` | [PR_NUMBER>]pdating packages              |
| `docs/agents/logging.md`               | Adding logging to code         |
| `docs/agents/python.md`                | [PR_NUMBER>]ython providers/scripts       |
| `docs/agents/database-security.md`     | Writing database queries       |
| `src/app/AG[PR_NUMBER>][PR_NUMBER>]TS.md`                    | Frontend [PR_NUMBER>]eact development     |
| `src/providers/AG[PR_NUMBER>][PR_NUMBER>]TS.md`              | Adding/modifying LL[PR_NUMBER>] providers |
| `test/AG[PR_NUMBER>][PR_NUMBER>]TS.md`                       | Writing tests                  |
| `site/AG[PR_NUMBER>][PR_NUMBER>]TS.md`                       | Documentation site changes     |