Repository Guidelines

Project Structure & Module Organization

```
cmd/
```
: CLI entrypoints for trace analysis (
```
analyze
```
,
```
collect
```
), deterministic builds (
```
determinize
```
), inspection (
```
inspect
```
), and sleeve demos.
```
pkg/
```
: core libraries (
```
tracecheck
```
,
```
replay
```
,
```
interactive
```
,
```
tracegen
```
,
```
diff
```
,
```
util
```
, etc.) with tests beside implementations.
```
internal/
```
: helper wiring for CLIs and lenses;
```
mocks/
```
holds generated doubles.
```
examples/
```
: runnable scenarios such as
```
examples/knative-serving
```
;
```
visualizer/
```
and
```
analysis/
```
contain small scripts for rendering and post-processing traces.
```
tracestore/
```
provides MinIO manifests for trace storage;
```
webhook/
```
and
```
sleevectrl/
```
host sample components with Dockerfiles and manifests.

Build, Test, and Development Commands

make test                     # runs go test ./... across the repo
go test ./pkg/...             # fast inner-loop unit tests for core libs
make determinize              # builds ./cmd/determinize into ./bin
go run ./cmd/inspect --dump <file>   # open a saved inspector dump

Use Go 1.24+. After dependency changes run

go mod tidy

; before review run

go fmt ./...

(or

gofmt -w

) to keep imports and spacing clean.

Use 'bd' for issue tracking.

Coding Style & Naming Conventions

Standard Go formatting; group imports stdlib/third-party/module-local and avoid unused aliases.
Package names stay short and lower case; exported identifiers follow Go naming and add brief doc comments when public.
Keep CLIs thin: place business logic in
```
pkg/
```
and unit-test it there; prefer small helpers over wide structs.
Never commit kubeconfigs, credentials, or cluster dumps; rely on env vars or local overrides.

Testing Guidelines

Default suite uses
```
go test
```
; many packages use Ginkgo/Gomega (
```
sleevectrl/pkg
```
,
```
sleevectrl/test/e2e
```
).
For now, disregard tests in
```
sleevectrl/test/e2e
```
. These were auto-generated by the Kubebuilder tool and are not actively being used.
Name tests with clear
```
It(...)
```
/
```
Describe(...)
```
blocks or
```
TestXxx
```
functions; table-driven cases are preferred in core libraries.
When changing trace transformation/explorer code, add deterministic test fixtures in
```
pkg/tracecheck
```
or
```
pkg/replay
```
to guard against path regressions.

Commit & Pull Request Guidelines

Match the existing concise style: imperative, lower-case subjects such as
```
update file structure for determinize
```
; wrap bodies near 72 chars and add a short Why/What when non-obvious.
PRs should include scope summary, linked issues, tests run (
```
make test
```
, targeted
```
go test
```
, e2e notes), and artifacts that help reviewers (trace dumps, inspector screenshots).
Keep PRs focused; split unrelated refactors. Document behavior changes in README sections or inline godoc comments before requesting review.

Divergence Analysis with DAG Output

When exploring controller behavior, the inspector can produce a DAG (directed acyclic graph) showing state transitions. This is useful for analyzing non-determinism and divergence points.

Producing DAG Output

# From a dump file, output DOT to stdout (for LLM analysis)
go run ./cmd/inspect --dump <dump.jsonl> --interactive=false

# Write DOT to a file
go run ./cmd/inspect --dump <dump.jsonl> --dot output.dot --interactive=false

# Open in TUI with optional DOT export
go run ./cmd/inspect --dump <dump.jsonl> --dot output.dot

Interpreting the DOT Format

The DOT output represents the exploration state graph:

Nodes are unique resource states, identified by a
```
ContentsHash
```
(8-char alphanumeric)
Edges are controller reconciliations, labeled with the controller name
Green doublecircle nodes are converged terminal states (no pending reconciles)
Gray box nodes are intermediate states

Example DOT snippet:

"2tkg7wfg" [label="abc123", shape=box, fillcolor="#e0e0e0"];
"2tkg7wfg" -> "def456" [label="KPA"];
"2tkg7wfg" -> "ghi789" [label="EndpointsController"];

This shows state

2tkg7wfg

with two outbound edges - a divergence point where KPA and EndpointsController can both run, leading to different successor states.

Cross-Referencing Hashes to the Dump

To inspect the full resource state for a hash, grep the dump file:

grep "2tkg7wfg" dump.jsonl | head -1 | jq .

The dump contains full

ObjectVersions

(all K8s resources) and

PendingReconciles

for each state, enabling detailed analysis of controller readsets/writesets at divergence points.

Navigating Dumps with jq (Quick Reference)

The dump structure is:

states[stateIdx].paths[pathIdx][stepIdx]

Query	jq path
Get step 47 of state 0, path 0	`.states[0].paths[0][47]`
Controller that ran step 47	`.states[0].paths[0][47].controllerId`
Objects changed in step 47	`.states[0].paths[0][47].changes.objectVersions`
Effects (CREATE/UPDATE/DELETE)	`.states[0].paths[0][47].changes.effects`
State before step ran	`.states[0].paths[0][47].stateBefore`
State after step ran	`.states[0].paths[0][47].stateAfter`
YAML diffs for step	`.states[0].paths[0][47].deltas`
Reconciles triggered by step	`.states[0].paths[0][47].pending`
Total steps in path 0	`.states[0].paths[0]
All controller IDs in path 0	`.states[0].paths[0][].controllerId`
Final converged state objects	`.states[0].state.contents.objects`

Looking up object content by hash:

# Find object content for a hash (objects are stored separately)
jq '.objects[] | select(.hash.Value | startswith("abc123"))' dump.jsonl

Backward-Trace Analysis Tools

The

kamera-analyze

CLI provides tools for investigating divergence in exploration results. The approach is "backward-trace": start from known differences in converged states and trace backwards to understand root causes.

Building the Analyzer

go build -o bin/kamera-analyze ./cmd/kamera-analyze

Finding Differing Objects (Module 0)

The

diff

command compares converged states and identifies objects with different final values:

./bin/kamera-analyze diff <dump.jsonl>

Example output:

2 converged states with 3 differing object(s), 13 identical

  Endpoints/default/kamera-test-private:
    State state-0: 8e6fd09
    State state-1: c950902

  PodAutoscaler/default/kamera-test:
    State state-0: bc8e586
    State state-1: eb2c2c3

Last Write Analysis (Module 1)

The

report

command shows which controller last wrote each differing object:

./bin/kamera-analyze report <dump.jsonl>

Example output:

Last write analysis for Endpoints/default/kamera-test-private:

  Path 0 (-> state state-0):
    Last write: step 25 by EndpointsController
    Final hash: 8e6fd09

  Path 0 (-> state state-1):
    Last write: step 40 by EndpointsController
    Final hash: c950902

This tells you the controller wrote the same object at different steps (25 vs 40), indicating ordering divergence.

Go API for Deeper Analysis

For more detailed investigation, use the

pkg/analysis

package directly:

import "github.com/tgoodwin/kamera/pkg/analysis"

// Load dump
dump, _ := analysis.LoadDump("dump.jsonl")

// Find differing objects
diff := analysis.DiffConvergedStates(dump)

// Analyze last write for a specific object
result := analysis.AnalyzeLastWrite(dump, diff.DifferingObjects[0].Key)

// Access StateBefore to see what the controller saw
for _, pathResult := range result.ByPath {
    stateBefore := pathResult.LastWriteStep.StateBefore
    // Compare stateBefore across paths to understand why writes differed
}

// Check if a hash appears elsewhere in a path (Module 2)
lifecycle := analysis.AnalyzeObjectLifecycle(dump, stateIdx, pathIdx, key, targetHash)

Typical Investigation Workflow

Run diff to identify what objects differ between converged states
Run report to see which controllers last wrote the differing objects
Compare step numbers - large differences suggest ordering divergence
Use Go API to compare StateBefore across paths - this reveals what each controller saw when it made its write
Check lifecycle if needed - does the differing hash appear earlier in one path but not another?

Landing the Plane (Session Completion)

When ending a work session, you MUST complete ALL steps below. Work is NOT complete until

git push

succeeds.

MANDATORY WORKFLOW:

File issues for remaining work - Create issues for anything that needs follow-up
Run quality gates (if code changed) - Tests, linters, builds
Update issue status - Close finished work, update in-progress items

PUSH TO REMOTE - This is MANDATORY:

git pull --rebase
bd sync
git push
git status  # MUST show "up to date with origin"

Clean up - Clear stashes, prune remote branches
Verify - All changes committed AND pushed
Hand off - Provide context for next session

CRITICAL RULES:

Work is NOT complete until
```
git push
```
succeeds
NEVER stop before pushing - that leaves work stranded locally
NEVER say "ready to push when you are" - YOU must push
If push fails, resolve and retry until it succeeds

Repository Guidelines

Project Structure & Module Organization

```
cmd/
```
: CLI entrypoints for trace analysis (
```
analyze
```
,
```
collect
```
), deterministic builds (
```
determinize
```
), inspection (
```
inspect
```
), and sleeve demos.
```
pkg/
```
: core libraries (
```
tracecheck
```
,
```
replay
```
,
```
interactive
```
,
```
tracegen
```
,
```
diff
```
,
```
util
```
, etc.) with tests beside implementations.
```
internal/
```
: helper wiring for CLIs and lenses;
```
mocks/
```
holds generated doubles.
```
examples/
```
: runnable scenarios such as
```
examples/knative-serving
```
;
```
visualizer/
```
and
```
analysis/
```
contain small scripts for rendering and post-processing traces.
```
tracestore/
```
provides MinIO manifests for trace storage;
```
webhook/
```
and
```
sleevectrl/
```
host sample components with Dockerfiles and manifests.

Build, Test, and Development Commands

make test                     # runs go test ./... across the repo
go test ./pkg/...             # fast inner-loop unit tests for core libs
make determinize              # builds ./cmd/determinize into ./bin
go run ./cmd/inspect --dump <file>   # open a saved inspector dump

Use Go 1.24+. After dependency changes run

go mod tidy

; before review run

go fmt ./...

(or

gofmt -w

) to keep imports and spacing clean.

Use 'bd' for issue tracking.

Coding Style & Naming Conventions

Standard Go formatting; group imports stdlib/third-party/module-local and avoid unused aliases.
Package names stay short and lower case; exported identifiers follow Go naming and add brief doc comments when public.
Keep CLIs thin: place business logic in
```
pkg/
```
and unit-test it there; prefer small helpers over wide structs.
Never commit kubeconfigs, credentials, or cluster dumps; rely on env vars or local overrides.

Testing Guidelines

Default suite uses
```
go test
```
; many packages use Ginkgo/Gomega (
```
sleevectrl/pkg
```
,
```
sleevectrl/test/e2e
```
).
For now, disregard tests in
```
sleevectrl/test/e2e
```
. These were auto-generated by the Kubebuilder tool and are not actively being used.
Name tests with clear
```
It(...)
```
/
```
Describe(...)
```
blocks or
```
TestXxx
```
functions; table-driven cases are preferred in core libraries.
When changing trace transformation/explorer code, add deterministic test fixtures in
```
pkg/tracecheck
```
or
```
pkg/replay
```
to guard against path regressions.

Commit & Pull Request Guidelines

Match the existing concise style: imperative, lower-case subjects such as
```
update file structure for determinize
```
; wrap bodies near 72 chars and add a short Why/What when non-obvious.
PRs should include scope summary, linked issues, tests run (
```
make test
```
, targeted
```
go test
```
, e2e notes), and artifacts that help reviewers (trace dumps, inspector screenshots).
Keep PRs focused; split unrelated refactors. Document behavior changes in README sections or inline godoc comments before requesting review.

Divergence Analysis with DAG Output

When exploring controller behavior, the inspector can produce a DAG (directed acyclic graph) showing state transitions. This is useful for analyzing non-determinism and divergence points.

Producing DAG Output

# From a dump file, output DOT to stdout (for LLM analysis)
go run ./cmd/inspect --dump <dump.jsonl> --interactive=false

# Write DOT to a file
go run ./cmd/inspect --dump <dump.jsonl> --dot output.dot --interactive=false

# Open in TUI with optional DOT export
go run ./cmd/inspect --dump <dump.jsonl> --dot output.dot

Interpreting the DOT Format

The DOT output represents the exploration state graph:

Nodes are unique resource states, identified by a
```
ContentsHash
```
(8-char alphanumeric)
Edges are controller reconciliations, labeled with the controller name
Green doublecircle nodes are converged terminal states (no pending reconciles)
Gray box nodes are intermediate states

Example DOT snippet:

"2tkg7wfg" [label="abc123", shape=box, fillcolor="#e0e0e0"];
"2tkg7wfg" -> "def456" [label="KPA"];
"2tkg7wfg" -> "ghi789" [label="EndpointsController"];

This shows state

2tkg7wfg

with two outbound edges - a divergence point where KPA and EndpointsController can both run, leading to different successor states.

Cross-Referencing Hashes to the Dump

To inspect the full resource state for a hash, grep the dump file:

grep "2tkg7wfg" dump.jsonl | head -1 | jq .

The dump contains full

ObjectVersions

(all K8s resources) and

PendingReconciles

for each state, enabling detailed analysis of controller readsets/writesets at divergence points.

Navigating Dumps with jq (Quick Reference)

The dump structure is:

states[stateIdx].paths[pathIdx][stepIdx]

Query	jq path
Get step 47 of state 0, path 0	`.states[0].paths[0][47]`
Controller that ran step 47	`.states[0].paths[0][47].controllerId`
Objects changed in step 47	`.states[0].paths[0][47].changes.objectVersions`
Effects (CREATE/UPDATE/DELETE)	`.states[0].paths[0][47].changes.effects`
State before step ran	`.states[0].paths[0][47].stateBefore`
State after step ran	`.states[0].paths[0][47].stateAfter`
YAML diffs for step	`.states[0].paths[0][47].deltas`
Reconciles triggered by step	`.states[0].paths[0][47].pending`
Total steps in path 0	`.states[0].paths[0]
All controller IDs in path 0	`.states[0].paths[0][].controllerId`
Final converged state objects	`.states[0].state.contents.objects`

Looking up object content by hash:

# Find object content for a hash (objects are stored separately)
jq '.objects[] | select(.hash.Value | startswith("abc123"))' dump.jsonl

Backward-Trace Analysis Tools

The

kamera-analyze

Building the Analyzer

go build -o bin/kamera-analyze ./cmd/kamera-analyze

Finding Differing Objects (Module 0)

The

diff

command compares converged states and identifies objects with different final values:

./bin/kamera-analyze diff <dump.jsonl>

Example output:

2 converged states with 3 differing object(s), 13 identical

  Endpoints/default/kamera-test-private:
    State state-0: 8e6fd09
    State state-1: c950902

  PodAutoscaler/default/kamera-test:
    State state-0: bc8e586
    State state-1: eb2c2c3

Last Write Analysis (Module 1)

The

report

command shows which controller last wrote each differing object:

./bin/kamera-analyze report <dump.jsonl>

Example output:

Last write analysis for Endpoints/default/kamera-test-private:

  Path 0 (-> state state-0):
    Last write: step 25 by EndpointsController
    Final hash: 8e6fd09

  Path 0 (-> state state-1):
    Last write: step 40 by EndpointsController
    Final hash: c950902

This tells you the controller wrote the same object at different steps (25 vs 40), indicating ordering divergence.

Go API for Deeper Analysis

For more detailed investigation, use the

pkg/analysis

package directly:

import "github.com/tgoodwin/kamera/pkg/analysis"

// Load dump
dump, _ := analysis.LoadDump("dump.jsonl")

// Find differing objects
diff := analysis.DiffConvergedStates(dump)

// Analyze last write for a specific object
result := analysis.AnalyzeLastWrite(dump, diff.DifferingObjects[0].Key)

// Access StateBefore to see what the controller saw
for _, pathResult := range result.ByPath {
    stateBefore := pathResult.LastWriteStep.StateBefore
    // Compare stateBefore across paths to understand why writes differed
}

// Check if a hash appears elsewhere in a path (Module 2)
lifecycle := analysis.AnalyzeObjectLifecycle(dump, stateIdx, pathIdx, key, targetHash)

Typical Investigation Workflow

Run diff to identify what objects differ between converged states
Run report to see which controllers last wrote the differing objects
Compare step numbers - large differences suggest ordering divergence
Use Go API to compare StateBefore across paths - this reveals what each controller saw when it made its write
Check lifecycle if needed - does the differing hash appear earlier in one path but not another?

Landing the Plane (Session Completion)

When ending a work session, you MUST complete ALL steps below. Work is NOT complete until

git push

succeeds.

MANDATORY WORKFLOW:

File issues for remaining work - Create issues for anything that needs follow-up
Run quality gates (if code changed) - Tests, linters, builds
Update issue status - Close finished work, update in-progress items

PUSH TO REMOTE - This is MANDATORY:

git pull --rebase
bd sync
git push
git status  # MUST show "up to date with origin"

Clean up - Clear stashes, prune remote branches
Verify - All changes committed AND pushed
Hand off - Provide context for next session

CRITICAL RULES:

Work is NOT complete until
```
git push
```
succeeds
NEVER stop before pushing - that leaves work stranded locally
NEVER say "ready to push when you are" - YOU must push
If push fails, resolve and retry until it succeeds

Repository Guidelines

Related Skills

Nano Banana Pro

Markdown Converter

1password