CANInsight-processing — AGENTS Guide

Purpose: help agents and contributors write tests, propose and implement changes, plan new features, and execute them safely for this repository.

This file applies to the entire repository.

Project Overview

Core processor:
```
src/log_converter.py:1
```
converts incoming CAN logs to MF4, merges continuations, and updates DB state.
Helpers:
```
src/helpers.py:1
```
parses CSV/DAT logs and builds MF4 signals.
Database layer:
```
src/database/
```
models and CRUD using SQLAlchemy; SQLite by default, Postgres supported via Alembic.
Web UI:
```
src/webserver.py:1
```
Flask app for browsing logs and metadata.
Config:
```
src/config.py:1
```
sets
```
DATA_FOLDER
```
, DB backend, and timing.
Tests:
```
src/tests/
```
uses Python
```
unittest
```
with fixtures in
```
src/tests/test_data/
```
.

Keep changes minimal, targeted, and consistent with existing patterns (unittest, no new dependencies without discussion).

Environment & Setup

Python 3.10+ recommended. Devcontainer/Docker setup is available if preferred (
```
.devcontainer/
```
,
```
docker-compose.yml
```
).

Install dependencies:

pip install -r processor/requirements.txt -r webserver/requirements.txt

Data root:
```
DATA_FOLDER
```
(default
```
./can_data/
```
, created if missing). Override via env when running if needed.
Database: defaults to SQLite file under
```
DATA_FOLDER
```
. Set
```
DB_BACKEND=postgres
```
to use Postgres (see
```
src/config.py:24
```
).

Running Tests

Test runner: standard library
```
unittest
```
.

Run all tests from repo root:

python -m unittest discover -s src -p "test_*.py" -v

VS Code config mirrors the above (
```
.vscode/settings.json:1
```
).
Tests are self-contained and clean up the DB between cases (see
```
src/tests/test_crud.py:139
```
). If a test needs the DB schema, call
```
init_and_upgrade_db()
```
(
```
src/database/upgrade.py:11
```
).

Guidelines for writing tests:

Location: place new tests under
```
src/tests/
```
with filenames
```
test_*.py
```
.

Style: use

unittest.TestCase

like existing tests (e.g.,

src/tests/test_log_converter.py:13

Fixtures/data: put sample logs/DBCs under
```
src/tests/test_data/
```
; keep them tiny and deterministic.
Isolation:
- Avoid mutating global
```
config.DATA_FOLDER
```
  in test modules; prefer temporary directories you pass into helpers where available or add parameters to functions to inject paths.
- When interacting with the DB, ensure cleanup in
```
tearDown
```
  like
```
src/tests/test_crud.py:139
```
  .
External libs: prefer real calls for
```
asammdf
```
in small cases; for heavy paths, you may stub or mock as needed (see patterns in
```
src/tests/test_log_converter.py
```
).

Making Code Changes & Upgrades

Be surgical: modify only what’s necessary; keep public behavior stable unless the change is part of an accepted feature.
Respect
```
DATA_FOLDER
```
and do not hardcode paths. New functionality should accept injectable paths or read from config.
Logging: use existing loggers (
```
src/log_converter_logger.py:12
```
,
```
src/webserver_logger.py:8
```
). Avoid adding duplicate handlers.
Database:
- SQLite (default): tables are created via
```
Base.metadata.create_all()
```
  when
```
init_and_upgrade_db()
```
  runs.
- Postgres: maintain migrations under
```
src/database/migrations/
```
  and ensure Alembic config stays valid.
Error handling: prefer explicit exceptions with clear messages; update tests to cover error branches.
Types: add type hints where straightforward; keep consistent with current code style.

When you change behavior:

Add/adjust tests first (red → green).
Update
```
readme.md
```
or inline docstrings when behavior is externally visible.

Feature Planning Template

Use this lightweight plan structure before major work:

Problem Statement: what user pain or gap exists?
Scope & Success Criteria: inputs/outputs and what “done” means.
Technical Approach: modules touched, data flows, error/edge cases.
Test Plan: new tests, data fixtures, and how to validate.
Rollout: config flags/migrations, docs, and backcompat notes.

Keep the plan in the PR description or a short

docs/

note.

High-Value Feature Ideas (Suggestions)

Processing
- Streaming CSV/DAT parsing to reduce memory for large logs.
- Stable, streaming hash computation for dedup without loading entire DF.
- Configurable continuation gap/overlap thresholds with explicit status codes.
- Enforce a finite state machine for
```
processing_status
```
  and centralize transitions.
DBC handling
- Clear precedence/merge order for unit-type vs global DBCs; surface which file decoded which signal.
- Validation command to lint DBC folder layout.
Web UI/API
- Authentication and basic RBAC (the UI is currently unsecured; see
```
readme.md
```
  note).
- Pagination, filtering, and download endpoints hardened against path issues.
- Small JSON API for logs, vehicles, and comments for automation.
Developer Experience
- CLI entry points:
```
process-new-files
```
  ,
```
backfill
```
  ,
```
reindex
```
  .
- Type hints across
```
src/database/
```
  and helpers; enable static checks in devcontainer.
- CI workflow running
```
unittest
```
  and building images.

If implementing any of these, slice work into small, testable increments.

Implementation Playbook (Execute Features)

Prep
- Run tests clean:
```
python -m unittest discover -s src -p "test_*.py" -v
```
  .
- If DB changes are needed, add SQLAlchemy model changes first; for Postgres, create an Alembic revision.
Build
- Add code in
```
src/
```
  with focused functions. Keep public APIs stable; prefer optional params over breaking changes.
- Add or update tests under
```
src/tests/
```
  and fixtures under
```
src/tests/test_data/
```
  .
- For filesystem work, parameterize paths or use
```
config.DATA_FOLDER
```
  .
Validate
- Run unit tests; if the feature affects end-to-end processing, run a smoke test:
  - ```
  export DATA_FOLDER=$(pwd)/can_data
```
  (or a temp dir)
- ```
python src/log_converter.py
```
    and drop a tiny sample log into
```
can_data/in_logs/
```
    .
Package (optional)
- Build images if needed (see
```
run-docker-compose.md
```
  ).

Coding Conventions

Python style: keep to current code style and structure. Do not add new tools unless requested.
Function naming: verbs for actions (
```
process_log_file
```
), nouns for structures (
```
LogFile
```
).
Avoid one-letter variables; prefer descriptive names consistent with existing modules.
Keep functions small; prefer helper extraction in
```
src/helpers.py
```
or a new focused module.

Quality Gates

Tests: all existing tests must pass; new code must have direct test coverage.
Logging: add debug/info logs at key steps, no noisy prints.
Configurable: use
```
config.py
```
for constants and environment overrides; do not hardcode file system paths.
Backward compatibility: keep DB schema compatible unless a migration is included and documented.

Quick Commands

Install deps:

pip install -r processor/requirements.txt -r webserver/requirements.txt

Run all tests:

python -m unittest discover -s src -p "test_*.py" -v

Processor (local):
```
python src/log_converter.py
```

Web UI (local):

python src/webserver.py

then open

http://localhost:5000

Docker Compose:
```
docker compose up
```
(see
```
run-docker-compose.md
```
)

Repository Notes

Tests already cover helpers, CRUD, and end-to-end processing paths. Prefer following their patterns when extending coverage.
SQLite is the default test DB; Postgres is supported but not required for local testing.
Respect the security note in
```
readme.md
```
: the web interface has no auth; do not expose it publicly.

If anything here conflicts with direct instructions from the task/request, follow the direct instructions.

CANInsight-processing — AGENTS Guide

Purpose: help agents and contributors write tests, propose and implement changes, plan new features, and execute them safely for this repository.

This file applies to the entire repository.

Project Overview

Core processor:
```
src/log_converter.py:1
```
converts incoming CAN logs to MF4, merges continuations, and updates DB state.
Helpers:
```
src/helpers.py:1
```
parses CSV/DAT logs and builds MF4 signals.
Database layer:
```
src/database/
```
models and CRUD using SQLAlchemy; SQLite by default, Postgres supported via Alembic.
Web UI:
```
src/webserver.py:1
```
Flask app for browsing logs and metadata.
Config:
```
src/config.py:1
```
sets
```
DATA_FOLDER
```
, DB backend, and timing.
Tests:
```
src/tests/
```
uses Python
```
unittest
```
with fixtures in
```
src/tests/test_data/
```
.

Keep changes minimal, targeted, and consistent with existing patterns (unittest, no new dependencies without discussion).

Environment & Setup

Python 3.10+ recommended. Devcontainer/Docker setup is available if preferred (
```
.devcontainer/
```
,
```
docker-compose.yml
```
).

Install dependencies:

pip install -r processor/requirements.txt -r webserver/requirements.txt

Data root:
```
DATA_FOLDER
```
(default
```
./can_data/
```
, created if missing). Override via env when running if needed.
Database: defaults to SQLite file under
```
DATA_FOLDER
```
. Set
```
DB_BACKEND=postgres
```
to use Postgres (see
```
src/config.py:24
```
).

Running Tests

Test runner: standard library
```
unittest
```
.

Run all tests from repo root:

python -m unittest discover -s src -p "test_*.py" -v

VS Code config mirrors the above (
```
.vscode/settings.json:1
```
).
Tests are self-contained and clean up the DB between cases (see
```
src/tests/test_crud.py:139
```
). If a test needs the DB schema, call
```
init_and_upgrade_db()
```
(
```
src/database/upgrade.py:11
```
).

Guidelines for writing tests:

Location: place new tests under
```
src/tests/
```
with filenames
```
test_*.py
```
.

Style: use

unittest.TestCase

like existing tests (e.g.,

src/tests/test_log_converter.py:13

Fixtures/data: put sample logs/DBCs under
```
src/tests/test_data/
```
; keep them tiny and deterministic.
Isolation:
- Avoid mutating global
```
config.DATA_FOLDER
```
  in test modules; prefer temporary directories you pass into helpers where available or add parameters to functions to inject paths.
- When interacting with the DB, ensure cleanup in
```
tearDown
```
  like
```
src/tests/test_crud.py:139
```
  .
External libs: prefer real calls for
```
asammdf
```
in small cases; for heavy paths, you may stub or mock as needed (see patterns in
```
src/tests/test_log_converter.py
```
).

Making Code Changes & Upgrades

Be surgical: modify only what’s necessary; keep public behavior stable unless the change is part of an accepted feature.
Respect
```
DATA_FOLDER
```
and do not hardcode paths. New functionality should accept injectable paths or read from config.
Logging: use existing loggers (
```
src/log_converter_logger.py:12
```
,
```
src/webserver_logger.py:8
```
). Avoid adding duplicate handlers.
Database:
- SQLite (default): tables are created via
```
Base.metadata.create_all()
```
  when
```
init_and_upgrade_db()
```
  runs.
- Postgres: maintain migrations under
```
src/database/migrations/
```
  and ensure Alembic config stays valid.
Error handling: prefer explicit exceptions with clear messages; update tests to cover error branches.
Types: add type hints where straightforward; keep consistent with current code style.

When you change behavior:

Add/adjust tests first (red → green).
Update
```
readme.md
```
or inline docstrings when behavior is externally visible.

Feature Planning Template

Use this lightweight plan structure before major work:

Problem Statement: what user pain or gap exists?
Scope & Success Criteria: inputs/outputs and what “done” means.
Technical Approach: modules touched, data flows, error/edge cases.
Test Plan: new tests, data fixtures, and how to validate.
Rollout: config flags/migrations, docs, and backcompat notes.

Keep the plan in the PR description or a short

docs/

note.

High-Value Feature Ideas (Suggestions)

Processing
- Streaming CSV/DAT parsing to reduce memory for large logs.
- Stable, streaming hash computation for dedup without loading entire DF.
- Configurable continuation gap/overlap thresholds with explicit status codes.
- Enforce a finite state machine for
```
processing_status
```
  and centralize transitions.
DBC handling
- Clear precedence/merge order for unit-type vs global DBCs; surface which file decoded which signal.
- Validation command to lint DBC folder layout.
Web UI/API
- Authentication and basic RBAC (the UI is currently unsecured; see
```
readme.md
```
  note).
- Pagination, filtering, and download endpoints hardened against path issues.
- Small JSON API for logs, vehicles, and comments for automation.
Developer Experience
- CLI entry points:
```
process-new-files
```
  ,
```
backfill
```
  ,
```
reindex
```
  .
- Type hints across
```
src/database/
```
  and helpers; enable static checks in devcontainer.
- CI workflow running
```
unittest
```
  and building images.

If implementing any of these, slice work into small, testable increments.

Implementation Playbook (Execute Features)

Prep
- Run tests clean:
```
python -m unittest discover -s src -p "test_*.py" -v
```
  .
- If DB changes are needed, add SQLAlchemy model changes first; for Postgres, create an Alembic revision.
Build
- Add code in
```
src/
```
  with focused functions. Keep public APIs stable; prefer optional params over breaking changes.
- Add or update tests under
```
src/tests/
```
  and fixtures under
```
src/tests/test_data/
```
  .
- For filesystem work, parameterize paths or use
```
config.DATA_FOLDER
```
  .
Validate
- Run unit tests; if the feature affects end-to-end processing, run a smoke test:
  - ```
  export DATA_FOLDER=$(pwd)/can_data
```
  (or a temp dir)
- ```
python src/log_converter.py
```
    and drop a tiny sample log into
```
can_data/in_logs/
```
    .
Package (optional)
- Build images if needed (see
```
run-docker-compose.md
```
  ).

Coding Conventions

Python style: keep to current code style and structure. Do not add new tools unless requested.
Function naming: verbs for actions (
```
process_log_file
```
), nouns for structures (
```
LogFile
```
).
Avoid one-letter variables; prefer descriptive names consistent with existing modules.
Keep functions small; prefer helper extraction in
```
src/helpers.py
```
or a new focused module.

Quality Gates

Tests: all existing tests must pass; new code must have direct test coverage.
Logging: add debug/info logs at key steps, no noisy prints.
Configurable: use
```
config.py
```
for constants and environment overrides; do not hardcode file system paths.
Backward compatibility: keep DB schema compatible unless a migration is included and documented.

Quick Commands

Install deps:

pip install -r processor/requirements.txt -r webserver/requirements.txt

Run all tests:

python -m unittest discover -s src -p "test_*.py" -v

Processor (local):
```
python src/log_converter.py
```

Web UI (local):

python src/webserver.py

then open

http://localhost:5000

Docker Compose:
```
docker compose up
```
(see
```
run-docker-compose.md
```
)

Repository Notes

Tests already cover helpers, CRUD, and end-to-end processing paths. Prefer following their patterns when extending coverage.
SQLite is the default test DB; Postgres is supported but not required for local testing.
Respect the security note in
```
readme.md
```
: the web interface has no auth; do not expose it publicly.

If anything here conflicts with direct instructions from the task/request, follow the direct instructions.

CANInsight-processing — AGENTS Guide

CANInsight-processing — AGENTS Guide

Project Overview

Environment & Setup

Running Tests

Making Code Changes & Upgrades

Feature Planning Template

High-Value Feature Ideas (Suggestions)

Implementation Playbook (Execute Features)

Coding Conventions

Quality Gates

Quick Commands

Repository Notes

Related Skills

Nano Banana Pro

Markdown Converter

1password

CANInsight-processing — AGENTS Guide

Project Overview

Environment & Setup

Running Tests

Making Code Changes & Upgrades

Feature Planning Template

High-Value Feature Ideas (Suggestions)

Implementation Playbook (Execute Features)

Coding Conventions

Quality Gates

Quick Commands

Repository Notes