diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f3f3cdf..b5731dd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -115,6 +115,11 @@ jobs: ./bin/ai-observer --version ./bin/ai-observer --version | grep -q "${{ steps.version.outputs.version }}" || (echo "Version mismatch!" && exit 1) + - name: Run backend race checks + if: matrix.goos == 'darwin' + working-directory: backend + run: go test -race ./... -timeout 120s + - name: Create archive shell: bash run: | diff --git a/README.md b/README.md index 322d0ed..955f32c 100644 --- a/README.md +++ b/README.md @@ -953,11 +953,11 @@ This means long CLI sessions produce traces with thousands of spans spanning hou ## Metric Availability by Ingestion Mode -Not all metrics are available in every mode. OTLP receives telemetry emitted by each tool's built-in OpenTelemetry instrumentation (in-memory counters, histograms, traces), while `watch` and `import` parse local session files which only contain conversation messages and per-response token/cost data. +Not all metrics are available in every mode. OTLP receives telemetry emitted by each tool's built-in OpenTelemetry instrumentation (in-memory counters, histograms, traces), while `watch` and `import` parse local session files. Historical Claude Code imports can also reconstruct selected activity metrics from tool results and session metadata. ### Claude Code -Local JSONL files store conversation messages and API usage per response. Operational metrics (lines of code, active time, git activity) are tracked in-memory by Claude Code's OTel instrumentation and only emitted over the network — they are never written to disk. +Local JSONL files store conversation messages and API usage per response. Full imports also reconstruct activity from successful tool results, PR links, and turn durations when available. These records do not reproduce all live OTel counters; see [reconstruction rules and limitations](docs/import.md#claude-code-activity-metrics). The incremental watcher imports transcripts and token/cost usage only. | Metric | OTLP (`serve`) | Watch (`watch`) | Import (`import`) | |--------|:-:|:-:|:-:| @@ -965,11 +965,11 @@ Local JSONL files store conversation messages and API usage per response. Operat | `claude_code.cost.usage` | Yes | Yes | Yes | | `claude_code.token.usage_user_facing` | Yes (derived) | Yes | Yes | | `claude_code.cost.usage_user_facing` | Yes (derived) | Yes | Yes | -| `claude_code.session.count` | Yes | — | — | -| `claude_code.lines_of_code.count` | Yes | — | — | -| `claude_code.active_time.total` | Yes | — | — | -| `claude_code.pull_request.count` | Yes | — | — | -| `claude_code.commit.count` | Yes | — | — | +| `claude_code.session.count` | Yes | — | Reconstructed | +| `claude_code.lines_of_code.count` | Yes | — | Reconstructed | +| `claude_code.active_time.total` | Yes | — | Reconstructed | +| `claude_code.pull_request.count` | Yes | — | Reconstructed | +| `claude_code.commit.count` | Yes | — | Reconstructed | | `claude_code.code_edit_tool.decision` | Yes | — | — | | Transcript logs | — | Yes | Yes | diff --git a/backend/go.mod b/backend/go.mod index 2a5b537..7ab85a3 100644 --- a/backend/go.mod +++ b/backend/go.mod @@ -1,9 +1,9 @@ module github.com/tobilg/ai-observer -go 1.26 +go 1.26.0 require ( - github.com/duckdb/duckdb-go/v2 v2.10503.1 + github.com/duckdb/duckdb-go/v2 v2.10505.0 github.com/fsnotify/fsnotify v1.10.1 github.com/go-chi/chi/v5 v5.2.5 github.com/go-chi/cors v1.2.2 @@ -12,16 +12,17 @@ require ( go.opentelemetry.io/proto/otlp v1.10.0 golang.org/x/net v0.52.0 google.golang.org/protobuf v1.36.11 + mvdan.cc/sh/v3 v3.14.1 ) require ( github.com/apache/arrow-go/v18 v18.5.2 // indirect - github.com/duckdb/duckdb-go-bindings v0.10503.0 // indirect - github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10503.0 // indirect - github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10503.0 // indirect - github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10503.0 // indirect - github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10503.0 // indirect - github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10503.0 // indirect + github.com/duckdb/duckdb-go-bindings v0.10505.0 // indirect + github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0 // indirect + github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0 // indirect + github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0 // indirect + github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0 // indirect + github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0 // indirect github.com/go-viper/mapstructure/v2 v2.5.0 // indirect github.com/goccy/go-json v0.10.6 // indirect github.com/google/flatbuffers v25.12.19+incompatible // indirect @@ -33,7 +34,7 @@ require ( golang.org/x/exp v0.0.0-20260312153236-7ab1446f8b90 // indirect golang.org/x/mod v0.34.0 // indirect golang.org/x/sync v0.20.0 // indirect - golang.org/x/sys v0.42.0 // indirect + golang.org/x/sys v0.47.0 // indirect golang.org/x/telemetry v0.0.0-20260316223853-b6b0c46d1ccd // indirect golang.org/x/text v0.35.0 // indirect golang.org/x/tools v0.43.0 // indirect diff --git a/backend/go.sum b/backend/go.sum index 9153cc9..c77ba1e 100644 --- a/backend/go.sum +++ b/backend/go.sum @@ -8,20 +8,20 @@ github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UF github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/duckdb/duckdb-go-bindings v0.10503.0 h1:nBms3habub5GB3N7zV7l1BwFP2mdRGrUepa+s5feZ1E= -github.com/duckdb/duckdb-go-bindings v0.10503.0/go.mod h1:USV+K6f/1gJBElmujc08nbJLSqPde1tqGrYuw/xR518= -github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10503.0 h1:O/yDkGeEZhWcRQwXGCy4SOioPqg8UNJEU/r4p6tMsgg= -github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10503.0/go.mod h1:EnAvZh1kNJHp5yF+M1ZHNEvapnmt6anq1xXHVrAGqMo= -github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10503.0 h1:gTYLRJZd9DcNtT3UkT+5NmMU1BgDZ22eP1vBGMjV2Z0= -github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10503.0/go.mod h1:IGLSeEcFhNeZF16aVjQCULD7TsFZKG5G7SyKJAXKp5c= -github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10503.0 h1:XzD2yzX9trLosZJYrCAgf/WDek6ANJHCQOFaoH7hfzg= -github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10503.0/go.mod h1:KAIynZ0GHCS7X5fRyuFnQMg/SZBPK/bS9OCOVojClxw= -github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10503.0 h1:zYkiCNaQAmneI98Hfjj31TLR9QnPA+8IuablmWPthog= -github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10503.0/go.mod h1:81SGOYoEUs8qaAfSk1wRfM5oobrIJ5KI7AzYhK6/bvQ= -github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10503.0 h1:InTPj+kxhdsFmG8iip6FAqJoH8Oerx0tW4dzfxqkAEk= -github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10503.0/go.mod h1:K25pJL26ARblGDeuAkrdblFvUen92+CwksLtPEHRqqQ= -github.com/duckdb/duckdb-go/v2 v2.10503.1 h1:mExuoVmoLQYVdTiGvsKTAlRPtGBVRutx12B8lctU+NI= -github.com/duckdb/duckdb-go/v2 v2.10503.1/go.mod h1:3trgI3LZo7TTgJYAP9+xCDJYdAF+E+TKE41JWKeWE5I= +github.com/duckdb/duckdb-go-bindings v0.10505.0 h1:/0pPsTLrcCsTGxT0VrHgJWnOcPe1tQL1vrki1v3jbAI= +github.com/duckdb/duckdb-go-bindings v0.10505.0/go.mod h1:HoD5xePkDj3VZbBnVVfxVVYIljZ9khCprWA7FgwIiC4= +github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0 h1:FrMqquFBQlMsi34h2KZgCku54rqA8xEbXZ0NLVDKwYs= +github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0/go.mod h1:EnAvZh1kNJHp5yF+M1ZHNEvapnmt6anq1xXHVrAGqMo= +github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0 h1:lbRbpQwT1MmUhh/VTwukV9K8bxKByV3UghAP3MvsbBo= +github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0/go.mod h1:IGLSeEcFhNeZF16aVjQCULD7TsFZKG5G7SyKJAXKp5c= +github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0 h1:nrsaVYj3XYCRbS2FpdOMD/KHE7egRMr+/NR1IHmjT84= +github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0/go.mod h1:KAIynZ0GHCS7X5fRyuFnQMg/SZBPK/bS9OCOVojClxw= +github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0 h1:qM6oGDgwXBILJGbTY4fCy6QOczLpucUA6yn6g3ORjh4= +github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0/go.mod h1:81SGOYoEUs8qaAfSk1wRfM5oobrIJ5KI7AzYhK6/bvQ= +github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0 h1:DjqZl9rYreHkSOqnqLmkrqH5T8UdQNcxZLJVZzGmXXA= +github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0/go.mod h1:K25pJL26ARblGDeuAkrdblFvUen92+CwksLtPEHRqqQ= +github.com/duckdb/duckdb-go/v2 v2.10505.0 h1:SWwvLn2Qx/RQSnQNupwgIF8VbnJ5A6OQU9lYb/mDETI= +github.com/duckdb/duckdb-go/v2 v2.10505.0/go.mod h1:m0PW4J4FG9hlFlVdXi6Ds9owpyIDaBdE2jyce00fGcE= github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug= @@ -32,6 +32,8 @@ github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-quicktest/qt v1.102.0 h1:HSQxCeh5YZH3EL3W39ixjtyaEhcWSXQHtHnMBzSs474= +github.com/go-quicktest/qt v1.102.0/go.mod h1:p4lGIVX+8Wa6ZPNDvqcxq36XpUDLh42FLetFU7odllI= github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro= github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU= @@ -56,6 +58,10 @@ github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBF github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 h1:AMFGa4R4MiIpspGNG7Z948v4n35fFGB3RR3G/ry4FWs= github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8/go.mod h1:mC1jAcsrzbxHt8iiaC+zU4b1ylILSosueou12R++wfY= github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 h1:+n/aFZefKZp7spd8DFdX7uMikMLXX4oubIzJF4kv/wI= @@ -64,6 +70,8 @@ github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc= +github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ= @@ -92,8 +100,8 @@ golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0= golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw= golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= -golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo= -golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/telemetry v0.0.0-20260316223853-b6b0c46d1ccd h1:QbR6Giw8AyR6v6Vff72jiZRUdZnetfgYRndQuKa806k= golang.org/x/telemetry v0.0.0-20260316223853-b6b0c46d1ccd/go.mod h1:TpUTTEp9frx7rTdLpC9gFG9kdI7zVLFTFFlqaH2Cncw= golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8= @@ -114,3 +122,5 @@ google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBN google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +mvdan.cc/sh/v3 v3.14.1 h1:bXkhQWNHCs0KZEChF8hYS6FC+T2N9mUZLbQv9blditI= +mvdan.cc/sh/v3 v3.14.1/go.mod h1:syYCoFET8w9tvevxiXUtY8/ICrU+l26jHmhJDra3Vwo= diff --git a/backend/internal/importer/claude.go b/backend/internal/importer/claude.go index aee4478..45d11ef 100644 --- a/backend/internal/importer/claude.go +++ b/backend/internal/importer/claude.go @@ -105,14 +105,25 @@ func (p *ClaudeParser) FindSessionFiles(ctx context.Context) ([]string, error) { // ClaudeJSONLEntry represents a single line in Claude Code JSONL files type ClaudeJSONLEntry struct { - Type string `json:"type,omitempty"` // Root type: "assistant", "user", "queue-operation", etc. - Timestamp string `json:"timestamp"` - SessionID string `json:"sessionId,omitempty"` - Version string `json:"version,omitempty"` - Cwd string `json:"cwd,omitempty"` - RequestID string `json:"requestId,omitempty"` - CostUSD *float64 `json:"costUSD,omitempty"` - Message *ClaudeMessage `json:"message,omitempty"` + Type string `json:"type,omitempty"` // Root type: "assistant", "user", "pr-link", "system", etc. + Timestamp string `json:"timestamp"` + SessionID string `json:"sessionId,omitempty"` + Version string `json:"version,omitempty"` + Cwd string `json:"cwd,omitempty"` + RequestID string `json:"requestId,omitempty"` + CostUSD *float64 `json:"costUSD,omitempty"` + Message *ClaudeMessage `json:"message,omitempty"` + ToolUseResult json.RawMessage `json:"toolUseResult,omitempty"` + GitBranch string `json:"gitBranch,omitempty"` + // PR link fields (present when type == "pr-link") + PRNumber int `json:"prNumber,omitempty"` + PRUrl string `json:"prUrl,omitempty"` + PRRepository string `json:"prRepository,omitempty"` + // System entry fields (present when type == "system") + Subtype string `json:"subtype,omitempty"` + DurationMs float64 `json:"durationMs,omitempty"` + // Worktree-state fields (present when type == "worktree-state") + WorktreeSession *claudeWorktreeSession `json:"worktreeSession,omitempty"` } type ClaudeMessage struct { @@ -125,6 +136,8 @@ type ClaudeMessage struct { } type ClaudeContent struct { + ID string `json:"id,omitempty"` + IsError bool `json:"is_error,omitempty"` Type string `json:"type,omitempty"` // "text", "tool_use", "tool_result" Text string `json:"text,omitempty"` // message text Name string `json:"name,omitempty"` // tool name (for tool_use) @@ -140,6 +153,115 @@ type ClaudeUsage struct { CacheReadInputTokens int `json:"cache_read_input_tokens"` } +// claudeEditInput holds the input for Edit tool calls +type claudeEditInput struct { + FilePath string `json:"file_path"` + OldString string `json:"old_string"` + NewString string `json:"new_string"` + ReplaceAll bool `json:"replace_all"` +} + +// claudeWriteInput holds the input for Write tool calls +type claudeWriteInput struct { + FilePath string `json:"file_path"` + Content string `json:"content"` +} + +// claudeMultiEditInput holds the input for MultiEdit tool calls +type claudeMultiEditInput struct { + FilePath string `json:"file_path"` + Edits []struct { + OldString string `json:"old_string"` + NewString string `json:"new_string"` + ReplaceAll bool `json:"replace_all"` + } `json:"edits"` +} + +// claudeWorktreeSession holds fields from a "worktree-state" entry's worktreeSession object. +type claudeWorktreeSession struct { + OriginalCwd string `json:"originalCwd,omitempty"` +} + +// claudeSessionMeta holds session-level metadata collected in a first pass +type claudeSessionMeta struct { + GitBranch string + Cwd string + OriginalCwd string // from worktree-state.worktreeSession.originalCwd + CandidateRepos []string // owner/repo references from pr-link entries (most-frequent first) + PRLinks []claudePRLink +} + +type claudePRLink struct { + Number int + URL string + Repository string + Timestamp time.Time +} + +// collectSessionMeta does a first pass over raw JSONL lines to collect session-level metadata +func (p *ClaudeParser) collectSessionMeta(lines []string) claudeSessionMeta { + meta := claudeSessionMeta{} + seenPRs := make(map[string]bool) + seenRepos := make(map[string]bool) + + for _, line := range lines { + var entry ClaudeJSONLEntry + if err := json.Unmarshal([]byte(line), &entry); err != nil { + continue + } + if entry.GitBranch != "" && meta.GitBranch == "" { + meta.GitBranch = entry.GitBranch + } + if entry.Cwd != "" && meta.Cwd == "" { + meta.Cwd = entry.Cwd + } + if entry.Type == "worktree-state" && entry.WorktreeSession != nil { + if oc := entry.WorktreeSession.OriginalCwd; oc != "" && meta.OriginalCwd == "" { + meta.OriginalCwd = oc + } + } + if entry.Type == "pr-link" && entry.PRNumber > 0 { + ts, err := time.Parse(time.RFC3339Nano, entry.Timestamp) + if err != nil { + continue + } + repo := strings.TrimSpace(entry.PRRepository) + if repo == "" { + repo = githubRepoFromURL(entry.PRUrl) + } + if repo == "" { + continue + } + key := strings.ToLower(repo) + "#" + strconv.Itoa(entry.PRNumber) + if seenPRs[key] { + continue + } + seenPRs[key] = true + if r := repo; !seenRepos[r] { + seenRepos[r] = true + meta.CandidateRepos = append(meta.CandidateRepos, r) + } + meta.PRLinks = append(meta.PRLinks, claudePRLink{Number: entry.PRNumber, URL: entry.PRUrl, Repository: repo, Timestamp: ts}) + } + } + return meta +} + +// extractRepository returns the best repository identifier for a session. +// +// A linked PR is only a candidate: its repository may differ from the working tree. +func extractRepository(meta claudeSessionMeta) string { + return resolveSessionRepository(meta.CandidateRepos, "", meta.OriginalCwd, meta.Cwd) +} + +// countLines returns the number of lines in s (at least 1 for non-empty strings) +func countLines(s string) int { + if s == "" { + return 0 + } + return strings.Count(strings.TrimSuffix(s, "\n"), "\n") + 1 +} + // ParseFile parses a Claude Code JSONL file func (p *ClaudeParser) ParseFile(ctx context.Context, path string) (*ImportResult, error) { file, err := os.Open(path) @@ -161,24 +283,59 @@ func (p *ClaudeParser) ParseFile(ctx context.Context, path string) (*ImportResul buf := make([]byte, 0, 64*1024) scanner.Buffer(buf, 1024*1024) - lineNum := 0 - messageIndex := 0 // Track message order for transcripts - seenRequests := make(map[string]bool) // For deduplication of metrics - + // Collect all non-empty lines + var lines []string for scanner.Scan() { if ctx.Err() != nil { return nil, ctx.Err() } + if line := scanner.Text(); strings.TrimSpace(line) != "" { + lines = append(lines, line) + } + } + if err := scanner.Err(); err != nil { + return nil, fmt.Errorf("reading file: %w", err) + } - lineNum++ - line := scanner.Text() - if strings.TrimSpace(line) == "" { - continue + // First pass: collect session-level metadata (gitBranch, PR info, cwd) + meta := p.collectSessionMeta(lines) + repository := extractRepository(meta) + + for _, pr := range meta.PRLinks { + result.Metrics = append(result.Metrics, createPRMetric(meta, pr)) + } + + messageIndex := 0 + seenRequests := make(map[string]bool) // For deduplication of metrics + activity := newClaudeActivity() + + // Second pass: process records + for _, line := range lines { + if ctx.Err() != nil { + return nil, ctx.Err() } var entry ClaudeJSONLEntry if err := json.Unmarshal([]byte(line), &entry); err != nil { - // Skip malformed lines + continue + } + + // Handle system/turn_duration entries for active time (no message field) + if entry.Type == "system" && entry.Subtype == "turn_duration" && entry.DurationMs > 0 { + ts, err := time.Parse(time.RFC3339Nano, entry.Timestamp) + if err != nil { + ts, err = time.Parse(time.RFC3339, entry.Timestamp) + } + if err == nil { + if result.FirstTime.IsZero() || ts.Before(result.FirstTime) { + result.FirstTime = ts + } + if ts.After(result.LastTime) { + result.LastTime = ts + } + secs := entry.DurationMs / 1000.0 + result.Metrics = append(result.Metrics, createActiveTimeMetric(ts, secs, meta, repository)) + } continue } @@ -219,10 +376,13 @@ func (p *ClaudeParser) ParseFile(ctx context.Context, path string) (*ImportResul } // Create transcript log records from message content - transcriptLogs := p.CreateTranscriptLogs(entry, ts, sessionID, &messageIndex) + transcriptLogs := p.createTranscriptLogs(entry, ts, sessionID, meta, &messageIndex) result.Logs = append(result.Logs, transcriptLogs...) - // For assistant entries with usage data, also create metrics + result.Metrics = append(result.Metrics, activity.process(entry, ts, sessionID, meta, repository)...) + + // Usage and individual tool actions have different identities. A content-only + // assistant entry must not consume the later usage record for its request. if entry.Type == "assistant" && entry.Message.Usage != nil { // Deduplication using messageId:requestId for metrics only dedupKey := fmt.Sprintf("%s:%s", entry.Message.ID, entry.RequestID) @@ -233,7 +393,7 @@ func (p *ClaudeParser) ParseFile(ctx context.Context, path string) (*ImportResul seenRequests[dedupKey] = true } - // Create api_request log record (existing behavior) + // Create api_request log record logRecord := api.LogRecord{ Timestamp: ts, ServiceName: SourceClaude.ServiceName(), @@ -253,27 +413,32 @@ func (p *ClaudeParser) ParseFile(ctx context.Context, path string) (*ImportResul if entry.RequestID != "" { logRecord.LogAttributes["request_id"] = entry.RequestID } + if meta.GitBranch != "" { + logRecord.LogAttributes["git_branch"] = meta.GitBranch + } + if repository != "" { + logRecord.LogAttributes["repository"] = repository + } result.Logs = append(result.Logs, logRecord) - // Create metrics + // Token usage metrics usage := entry.Message.Usage model := entry.Message.Model - // Token usage metrics (creates both regular and user-facing variants) if usage.InputTokens > 0 { - result.Metrics = append(result.Metrics, CreateTokenMetrics(ts, model, "input", float64(usage.InputTokens))...) + result.Metrics = append(result.Metrics, createTokenMetrics(ts, model, "input", float64(usage.InputTokens), meta, repository)...) } if usage.OutputTokens > 0 { - result.Metrics = append(result.Metrics, CreateTokenMetrics(ts, model, "output", float64(usage.OutputTokens))...) + result.Metrics = append(result.Metrics, createTokenMetrics(ts, model, "output", float64(usage.OutputTokens), meta, repository)...) } if usage.CacheCreationInputTokens > 0 { - result.Metrics = append(result.Metrics, CreateTokenMetrics(ts, model, "cacheCreation", float64(usage.CacheCreationInputTokens))...) + result.Metrics = append(result.Metrics, createTokenMetrics(ts, model, "cacheCreation", float64(usage.CacheCreationInputTokens), meta, repository)...) } if usage.CacheReadInputTokens > 0 { - result.Metrics = append(result.Metrics, CreateTokenMetrics(ts, model, "cacheRead", float64(usage.CacheReadInputTokens))...) + result.Metrics = append(result.Metrics, createTokenMetrics(ts, model, "cacheRead", float64(usage.CacheReadInputTokens), meta, repository)...) } - // Cost metrics using pricing mode (creates both regular and user-facing variants) + // Cost metrics tokenUsage := pricing.ClaudeTokenUsage{ InputTokens: int64(usage.InputTokens), OutputTokens: int64(usage.OutputTokens), @@ -282,22 +447,51 @@ func (p *ClaudeParser) ParseFile(ctx context.Context, path string) (*ImportResul } cost := pricing.GetClaudeCostWithMode(p.pricingMode, model, tokenUsage, entry.CostUSD) if cost > 0 { - result.Metrics = append(result.Metrics, CreateCostMetrics(ts, model, cost)...) + result.Metrics = append(result.Metrics, createCostMetrics(ts, model, cost, meta, repository)...) } } result.RecordCount++ } - if err := scanner.Err(); err != nil { - return nil, fmt.Errorf("reading file: %w", err) + // Every emitted event participates in file-level date filtering, including PR + // links which can occur after the last conversation message. + for _, metric := range result.Metrics { + if result.FirstTime.IsZero() || metric.Timestamp.Before(result.FirstTime) { + result.FirstTime = metric.Timestamp + } + if metric.Timestamp.After(result.LastTime) { + result.LastTime = metric.Timestamp + } + } + // Session count metric - one per file, timestamped at first activity + if !result.FirstTime.IsZero() { + result.Metrics = append(result.Metrics, createSessionMetric(result.FirstTime, meta, repository)) } return result, nil } -// CreateTranscriptLogs creates transcript log records from message content +// CreateTranscriptLogs creates transcript log records from message content. +// This public form is used by the file watcher; ParseFile uses createTranscriptLogs +// (private) which also enriches records with session metadata. func (p *ClaudeParser) CreateTranscriptLogs(entry ClaudeJSONLEntry, ts time.Time, sessionID string, messageIndex *int) []api.LogRecord { + return p.createTranscriptLogs(entry, ts, sessionID, claudeSessionMeta{}, messageIndex) +} + +// relativeFilePath returns a path relative to baseDir, falling back to the base filename +func relativeFilePath(filePath, baseDir string) string { + if baseDir == "" { + return filepath.Base(filePath) + } + if rel, err := filepath.Rel(baseDir, filePath); err == nil && !strings.HasPrefix(rel, "..") { + return rel + } + return filepath.Base(filePath) +} + +// createTranscriptLogs creates transcript log records with optional session metadata enrichment. +func (p *ClaudeParser) createTranscriptLogs(entry ClaudeJSONLEntry, ts time.Time, sessionID string, meta claudeSessionMeta, messageIndex *int) []api.LogRecord { var logs []api.LogRecord if entry.Message == nil || len(entry.Message.Content) == 0 { @@ -311,11 +505,11 @@ func (p *ClaudeParser) CreateTranscriptLogs(entry ClaudeJSONLEntry, ts time.Time var body string var role string attrs := map[string]string{ - "event.name": "transcript.message", - "session.id": sessionID, - "message.index": strconv.Itoa(*messageIndex), - "message.role": baseRole, - "import_source": "local_jsonl", + "event.name": "transcript.message", + "session.id": sessionID, + "message.index": strconv.Itoa(*messageIndex), + "message.role": baseRole, + "import_source": "local_jsonl", } if entry.Message.Model != "" { @@ -324,6 +518,12 @@ func (p *ClaudeParser) CreateTranscriptLogs(entry ClaudeJSONLEntry, ts time.Time if entry.Message.ID != "" { attrs["message.id"] = entry.Message.ID } + if meta.GitBranch != "" { + attrs["git_branch"] = meta.GitBranch + } + if repo := extractRepository(meta); repo != "" { + attrs["repository"] = repo + } switch content.Type { case "text": @@ -381,9 +581,16 @@ const ( ClaudeUserFacingTokenUsageMetric = "claude_code.token.usage_user_facing" ClaudeCostMetric = "claude_code.cost.usage" ClaudeUserFacingCostMetric = "claude_code.cost.usage_user_facing" + claudeLinesOfCodeMetric = "claude_code.lines_of_code.count" + claudePullRequestMetric = "claude_code.pull_request.count" + claudeCommitMetric = "claude_code.commit.count" + claudeSessionMetric = "claude_code.session.count" + claudeActiveTimeMetric = "claude_code.active_time.total" ) -// CreateTokenMetric creates a token usage metric with the specified name +// CreateTokenMetric creates a token usage metric with the specified name. +// This public form is used by the file watcher; ParseFile uses createTokenMetric +// (private) which also attaches session attributes. func CreateTokenMetric(ts time.Time, metricName, model, tokenType string, value float64) api.MetricDataPoint { return api.MetricDataPoint{ Timestamp: ts, @@ -399,6 +606,34 @@ func CreateTokenMetric(ts time.Time, metricName, model, tokenType string, value } } +// sessionAttrs builds the common session-level attributes (git_branch, repository). +func sessionAttrs(meta claudeSessionMeta, repository string) map[string]string { + attrs := make(map[string]string) + if meta.GitBranch != "" { + attrs["git_branch"] = meta.GitBranch + } + if repository != "" { + attrs["repository"] = repository + } + return attrs +} + +// createTokenMetric creates a token usage metric with the specified name, including session attributes. +func createTokenMetric(ts time.Time, metricName, model, tokenType string, value float64, meta claudeSessionMeta, repository string) api.MetricDataPoint { + attrs := sessionAttrs(meta, repository) + attrs["type"] = tokenType + attrs["model"] = model + attrs["import_source"] = "local_jsonl" + return api.MetricDataPoint{ + Timestamp: ts, + ServiceName: SourceClaude.ServiceName(), + MetricName: metricName, + MetricType: "sum", + Value: &value, + Attributes: attrs, + } +} + // CreateTokenMetrics creates both regular and user-facing token usage metrics. // JSONL data is already user-facing (only assistant messages with cache tokens), // so both metrics have identical values for consistency with OTLP-derived metrics. @@ -409,7 +644,17 @@ func CreateTokenMetrics(ts time.Time, model, tokenType string, value float64) [] } } -// CreateCostMetric creates a cost metric with the specified name +// createTokenMetrics creates both regular and user-facing token usage metrics with session attributes. +func createTokenMetrics(ts time.Time, model, tokenType string, value float64, meta claudeSessionMeta, repository string) []api.MetricDataPoint { + return []api.MetricDataPoint{ + createTokenMetric(ts, ClaudeTokenUsageMetric, model, tokenType, value, meta, repository), + createTokenMetric(ts, ClaudeUserFacingTokenUsageMetric, model, tokenType, value, meta, repository), + } +} + +// CreateCostMetric creates a cost metric with the specified name. +// This public form is used by the file watcher; ParseFile uses createCostMetric +// (private) which also attaches session attributes. func CreateCostMetric(ts time.Time, metricName, model string, value float64) api.MetricDataPoint { return api.MetricDataPoint{ Timestamp: ts, @@ -424,6 +669,21 @@ func CreateCostMetric(ts time.Time, metricName, model string, value float64) api } } +// createCostMetric creates a cost metric with the specified name, including session attributes. +func createCostMetric(ts time.Time, metricName, model string, value float64, meta claudeSessionMeta, repository string) api.MetricDataPoint { + attrs := sessionAttrs(meta, repository) + attrs["model"] = model + attrs["import_source"] = "local_jsonl" + return api.MetricDataPoint{ + Timestamp: ts, + ServiceName: SourceClaude.ServiceName(), + MetricName: metricName, + MetricType: "sum", + Value: &value, + Attributes: attrs, + } +} + // CreateCostMetrics creates both regular and user-facing cost metrics. // JSONL data is already user-facing (only assistant messages with cache tokens), // so both metrics have identical values for consistency with OTLP-derived metrics. @@ -433,3 +693,94 @@ func CreateCostMetrics(ts time.Time, model string, value float64) []api.MetricDa CreateCostMetric(ts, ClaudeUserFacingCostMetric, model, value), } } + +// createCostMetrics creates both regular and user-facing cost metrics with session attributes. +func createCostMetrics(ts time.Time, model string, value float64, meta claudeSessionMeta, repository string) []api.MetricDataPoint { + return []api.MetricDataPoint{ + createCostMetric(ts, ClaudeCostMetric, model, value, meta, repository), + createCostMetric(ts, ClaudeUserFacingCostMetric, model, value, meta, repository), + } +} + +// createLOCMetric creates a lines_of_code metric for a file edit +func createLOCMetric(ts time.Time, lineType, filePath, fileType string, value float64, meta claudeSessionMeta, repository string) api.MetricDataPoint { + attrs := sessionAttrs(meta, repository) + attrs["type"] = lineType + attrs["file_type"] = fileType + attrs["file_path"] = filePath + attrs["import_source"] = "local_jsonl" + return api.MetricDataPoint{ + Timestamp: ts, + ServiceName: SourceClaude.ServiceName(), + MetricName: claudeLinesOfCodeMetric, + MetricType: "sum", + Value: &value, + Attributes: attrs, + } +} + +// createPRMetric creates a pull_request.count metric for a linked PR +func createPRMetric(meta claudeSessionMeta, pr claudePRLink) api.MetricDataPoint { + one := 1.0 + attrs := sessionAttrs(meta, pr.Repository) + attrs["pr_number"] = strconv.Itoa(pr.Number) + attrs["pr_url"] = pr.URL + attrs["reconstruction"] = "pr_link" + attrs["import_source"] = "local_jsonl" + return api.MetricDataPoint{ + Timestamp: pr.Timestamp, + ServiceName: SourceClaude.ServiceName(), + MetricName: claudePullRequestMetric, + MetricType: "sum", + Value: &one, + Attributes: attrs, + } +} + +// createCommitMetric creates a commit.count metric for a git commit tool call +func createCommitMetric(ts time.Time, meta claudeSessionMeta, repository string) api.MetricDataPoint { + one := 1.0 + attrs := sessionAttrs(meta, repository) + attrs["import_source"] = "local_jsonl" + return api.MetricDataPoint{ + Timestamp: ts, + ServiceName: SourceClaude.ServiceName(), + MetricName: claudeCommitMetric, + MetricType: "sum", + Value: &one, + Attributes: attrs, + } +} + +// createSessionMetric creates a session.count metric (one per JSONL file) +func createSessionMetric(ts time.Time, meta claudeSessionMeta, repository string) api.MetricDataPoint { + one := 1.0 + attrs := sessionAttrs(meta, repository) + attrs["import_source"] = "local_jsonl" + return api.MetricDataPoint{ + Timestamp: ts, + ServiceName: SourceClaude.ServiceName(), + MetricName: claudeSessionMetric, + MetricType: "sum", + Value: &one, + Attributes: attrs, + } +} + +// createActiveTimeMetric creates an active_time.total metric from a turn_duration entry. +// Each turn emits its own data point (consistent with OTLP telemetry). +func createActiveTimeMetric(ts time.Time, secs float64, meta claudeSessionMeta, repository string) api.MetricDataPoint { + attrs := sessionAttrs(meta, repository) + attrs["import_source"] = "local_jsonl" + attrs["type"] = "cli" + attrs["reconstruction"] = "turn_duration" + return api.MetricDataPoint{ + Timestamp: ts, + ServiceName: SourceClaude.ServiceName(), + MetricName: claudeActiveTimeMetric, + MetricUnit: "s", + MetricType: "sum", + Value: &secs, + Attributes: attrs, + } +} diff --git a/backend/internal/importer/claude_activity.go b/backend/internal/importer/claude_activity.go new file mode 100644 index 0000000..4dda093 --- /dev/null +++ b/backend/internal/importer/claude_activity.go @@ -0,0 +1,343 @@ +package importer + +import ( + "encoding/json" + "path/filepath" + "regexp" + "strings" + "time" + + "github.com/tobilg/ai-observer/internal/api" + "mvdan.cc/sh/v3/syntax" +) + +// Tool results may contain either a string or a list of content blocks. Keep +// their text available to both transcript rendering and activity reconstruction. +func (c *ClaudeContent) UnmarshalJSON(data []byte) error { + type plain ClaudeContent + var raw struct { + *plain + Content json.RawMessage `json:"content"` + } + *c = ClaudeContent{} + raw.plain = (*plain)(c) + if err := json.Unmarshal(data, &raw); err != nil { + return err + } + if len(raw.Content) == 0 || string(raw.Content) == "null" { + return nil + } + if err := json.Unmarshal(raw.Content, &c.Content); err == nil { + return nil + } + var blocks []struct{ Type, Text string } + if err := json.Unmarshal(raw.Content, &blocks); err != nil { + return err + } + var texts []string + for _, block := range blocks { + if block.Type == "text" { + texts = append(texts, block.Text) + } + } + c.Content = strings.Join(texts, "\n") + return nil +} + +type claudeToolCall struct { + content ClaudeContent + model string +} + +type claudeToolResult struct { + Type string `json:"type"` + FilePath string `json:"filePath"` + OriginalFile *string `json:"originalFile"` + StructuredPatch []struct { + Lines []string `json:"lines"` + } `json:"structuredPatch"` + Stdout string `json:"stdout"` + Interrupted bool `json:"interrupted"` + ExitCode *int `json:"exitCode"` +} + +type claudeActivity struct { + pending map[string]claudeToolCall + completed map[string]bool +} + +func newClaudeActivity() *claudeActivity { + return &claudeActivity{pending: make(map[string]claudeToolCall), completed: make(map[string]bool)} +} + +// Only completed, successful calls produce activity. Tool IDs are independent +// of API request IDs, since one response can contain several tool calls. +func (a *claudeActivity) process(entry ClaudeJSONLEntry, ts time.Time, sessionID string, meta claudeSessionMeta, repository string) []api.MetricDataPoint { + if entry.Message == nil { + return nil + } + resultCount := 0 + for _, content := range entry.Message.Content { + if content.Type == "tool_result" { + resultCount++ + } + } + var metrics []api.MetricDataPoint + for _, content := range entry.Message.Content { + if entry.Type == "assistant" && content.Type == "tool_use" && content.ID != "" { + key := sessionID + ":" + content.ID + if _, exists := a.pending[key]; !exists && !a.completed[key] { + a.pending[key] = claudeToolCall{content, entry.Message.Model} + } + continue + } + if entry.Type != "user" || content.Type != "tool_result" || content.ToolUseID == "" { + continue + } + key := sessionID + ":" + content.ToolUseID + call, exists := a.pending[key] + if !exists || a.completed[key] { + continue + } + delete(a.pending, key) + a.completed[key] = true + if content.IsError { + continue + } + var result claudeToolResult + // Top-level result metadata is only unambiguous for a single result block. + if resultCount == 1 { + _ = json.Unmarshal(entry.ToolUseResult, &result) + } + if result.Interrupted || (result.ExitCode != nil && *result.ExitCode != 0) { + continue + } + if call.content.Name == "Bash" { + var input struct { + Command string `json:"command"` + } + data, err := json.Marshal(call.content.Input) + if err != nil || json.Unmarshal(data, &input) != nil { + continue + } + if !hasGitCommit(input.Command) { + continue + } + output := content.Content + if result.Stdout != "" { + output = result.Stdout + } + // Git's completion headers distinguish created commits from dry runs, + // failed commands and shell commands which merely mention git commit. + seen := make(map[string]bool) + for _, match := range gitCommitOutput.FindAllStringSubmatch(output, -1) { + if seen[match[1]] { + continue + } + seen[match[1]] = true + metric := createCommitMetric(ts, meta, repository) + metric.Attributes["reconstruction"] = "successful_tool_result" + metrics = append(metrics, metric) + } + continue + } + path, added, removed, ok := completedEditLines(call.content, result, content.Content) + if !ok { + continue + } + for _, delta := range []struct { + kind string + value int + }{{"added", added}, {"removed", removed}} { + if delta.value == 0 { + continue + } + metric := createLOCMetric(ts, delta.kind, relativeFilePath(path, meta.Cwd), filepath.Ext(path), float64(delta.value), meta, repository) + metric.Attributes["model"] = call.model + metric.Attributes["reconstruction"] = "successful_tool_result" + metrics = append(metrics, metric) + } + } + return metrics +} + +func completedEditLines(call ClaudeContent, result claudeToolResult, output string) (path string, added, removed int, ok bool) { + if call.Name != "Edit" && call.Name != "Write" && call.Name != "MultiEdit" { + return + } + data, err := json.Marshal(call.Input) + if err != nil { + return + } + var file struct { + Path string `json:"file_path"` + } + if json.Unmarshal(data, &file) != nil || file.Path == "" { + return + } + path = file.Path + if result.FilePath != "" && result.FilePath != path { + return + } + // A recorded patch includes unchanged context and all replace_all matches. + if result.StructuredPatch != nil { + for _, hunk := range result.StructuredPatch { + for _, line := range hunk.Lines { + switch { + case strings.HasPrefix(line, "+"): + added++ + case strings.HasPrefix(line, "-"): + removed++ + case strings.HasPrefix(line, " "), strings.HasPrefix(line, "\\"): + default: + return path, 0, 0, false + } + } + } + return path, added, removed, true + } + switch call.Name { + case "Edit": + var input claudeEditInput + if json.Unmarshal(data, &input) != nil || input.ReplaceAll { + return + } + added, removed, ok = changedLines(input.OldString, input.NewString) + case "Write": + var input claudeWriteInput + if json.Unmarshal(data, &input) != nil { + return + } + if result.Type == "create" || strings.HasPrefix(output, "File created successfully") { + return path, countLines(input.Content), 0, true + } + if result.OriginalFile != nil { + added, removed, ok = changedLines(*result.OriginalFile, input.Content) + } + // Overwrites without a recorded original or patch cannot be reconstructed. + case "MultiEdit": + var input claudeMultiEditInput + if json.Unmarshal(data, &input) != nil { + return + } + for _, edit := range input.Edits { + if edit.ReplaceAll { + return path, 0, 0, false + } + a, r, valid := changedLines(edit.OldString, edit.NewString) + if !valid { + return path, 0, 0, false + } + added += a + removed += r + } + ok = true + } + return +} + +// changedLines uses the longest common subsequence after trimming shared edges. +// Bound work for unusually large replacements; omit uncertain metrics instead +// of blocking an import or substituting gross replacement sizes. +func changedLines(old, new string) (added, removed int, ok bool) { + lines := func(s string) []string { + if s == "" { + return nil + } + return strings.Split(strings.TrimSuffix(s, "\n"), "\n") + } + a, b := lines(old), lines(new) + for len(a) > 0 && len(b) > 0 && a[0] == b[0] { + a, b = a[1:], b[1:] + } + for len(a) > 0 && len(b) > 0 && a[len(a)-1] == b[len(b)-1] { + a, b = a[:len(a)-1], b[:len(b)-1] + } + if int64(len(a))*int64(len(b)) > 4_000_000 { + return 0, 0, false + } + row := make([]int, len(b)+1) + for _, left := range a { + previous := 0 + for j, right := range b { + saved := row[j+1] + if left == right { + row[j+1] = previous + 1 + } else { + row[j+1] = max(row[j], row[j+1]) + } + previous = saved + } + } + common := row[len(b)] + return len(b) - common, len(a) - common, true +} + +var gitCommitOutput = regexp.MustCompile(`(?m)^\[[^\]\r\n]+ ([0-9a-f]{7,64})\](?: |$)`) + +// Parse shell syntax only; no session command or expansion is ever executed. +func hasGitCommit(command string) bool { + file, err := syntax.NewParser().Parse(strings.NewReader(command), "") + if err != nil { + return false + } + found := false + syntax.Walk(file, func(node syntax.Node) bool { + if found { + return false + } + if _, ok := node.(*syntax.FuncDecl); ok { + return false + } + call, ok := node.(*syntax.CallExpr) + if !ok || len(call.Args) < 2 { + return true + } + args := make([]string, len(call.Args)) + for i, arg := range call.Args { + args[i], _ = literalShellWord(arg.Parts) + } + if args[0] != "git" && args[0] != "/usr/bin/git" && args[0] != "/bin/git" { + return true + } + for i := 1; i < len(args); i++ { + arg := args[i] + switch { + case arg == "-C" || arg == "-c" || arg == "--git-dir" || arg == "--work-tree" || arg == "--namespace": + i++ + case arg == "--no-pager" || arg == "--no-optional-locks" || arg == "--bare": + case strings.HasPrefix(arg, "--git-dir=") || strings.HasPrefix(arg, "--work-tree=") || strings.HasPrefix(arg, "--namespace=") || strings.HasPrefix(arg, "--config-env="): + case strings.HasPrefix(arg, "-C") || strings.HasPrefix(arg, "-c"): + default: + found = arg == "commit" + return !found + } + } + return true + }) + return found +} + +func literalShellWord(parts []syntax.WordPart) (string, bool) { + var result strings.Builder + for _, part := range parts { + switch p := part.(type) { + case *syntax.Lit: + result.WriteString(p.Value) + case *syntax.SglQuoted: + if p.Dollar { + return "", false + } + result.WriteString(p.Value) + case *syntax.DblQuoted: + s, ok := literalShellWord(p.Parts) + if !ok { + return "", false + } + result.WriteString(s) + default: + return "", false + } + } + return result.String(), true +} diff --git a/backend/internal/importer/claude_activity_test.go b/backend/internal/importer/claude_activity_test.go new file mode 100644 index 0000000..bcbaacd --- /dev/null +++ b/backend/internal/importer/claude_activity_test.go @@ -0,0 +1,399 @@ +package importer + +import ( + "context" + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/tobilg/ai-observer/internal/storage" +) + +type activityJSON = map[string]any + +func activityAssistant(id, req string, content []any, usage any) activityJSON { + return activityJSON{"type": "assistant", "timestamp": "2026-09-01T23:59:00Z", "sessionId": "review-session", "requestId": req, "costUSD": 0.05, + "message": activityJSON{"id": id, "model": "claude-sonnet-4-20250514", "role": "assistant", "content": content, "usage": usage}} +} + +func activityTool(id, name string, input activityJSON) activityJSON { + return activityJSON{"type": "tool_use", "id": id, "name": name, "input": input} +} + +func activityResult(id, content string, failed bool) activityJSON { + return activityJSON{"type": "user", "timestamp": "2026-09-01T23:59:01Z", "sessionId": "review-session", + "message": activityJSON{"role": "user", "content": []any{activityJSON{"type": "tool_result", "tool_use_id": id, "content": content, "is_error": failed}}}} +} + +func activityFile(t *testing.T, entries ...activityJSON) string { + t.Helper() + path := filepath.Join(t.TempDir(), "review-session.jsonl") + f, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + enc := json.NewEncoder(f) + for _, entry := range entries { + if err := enc.Encode(entry); err != nil { + t.Fatal(err) + } + } + if err := f.Close(); err != nil { + t.Fatal(err) + } + return path +} + +func activityParse(t *testing.T, entries ...activityJSON) *ImportResult { + t.Helper() + r, err := NewClaudeParser().ParseFile(context.Background(), activityFile(t, entries...)) + if err != nil { + t.Fatal(err) + } + return r +} + +func activitySum(r *ImportResult, name, kind string) float64 { + var sum float64 + for _, m := range r.Metrics { + if m.MetricName == name && (kind == "" || m.Attributes["type"] == kind) && m.Value != nil { + sum += *m.Value + } + } + return sum +} + +func TestClaudeActivityLateUsage(t *testing.T) { + r := activityParse(t, + activityAssistant("msg-1", "req-1", []any{activityJSON{"type": "text", "text": "Working"}}, nil), + activityAssistant("msg-1", "req-1", nil, activityJSON{"input_tokens": 100, "output_tokens": 50}), + ) + if got := activitySum(r, "claude_code.token.usage", ""); got != 150 { + t.Errorf("token total: got %v, want 150", got) + } + if got := activitySum(r, "claude_code.cost.usage", ""); got <= 0 { + t.Errorf("cost: got %v, want positive", got) + } +} + +func TestClaudeActivityDistinctToolBlocksSameRequest(t *testing.T) { + r := activityParse(t, + activityAssistant("msg-1", "req-1", []any{activityTool("tool-1", "Write", activityJSON{"file_path": "/project/a.go", "content": "a"})}, nil), + activityAssistant("msg-1", "req-1", []any{activityTool("tool-2", "Write", activityJSON{"file_path": "/project/b.go", "content": "b"})}, nil), + activityResult("tool-1", "File created successfully", false), + activityResult("tool-2", "File created successfully", false), + ) + if got := activitySum(r, "claude_code.lines_of_code.count", "added"); got != 2 { + t.Errorf("two distinct one-line writes: got %v added, want 2", got) + } +} + +func TestClaudeActivityCommitDetection(t *testing.T) { + for _, tc := range []struct { + name, command, result string + failed bool + want float64 + }{ + {"echo", "echo 'git commit'", "git commit", false, 0}, + {"failed", "git commit -m test", "nothing to commit, working tree clean", true, 0}, + {"git_option", "git -C /project commit -m test", "[main abc1234] test", false, 1}, + } { + t.Run(tc.name, func(t *testing.T) { + r := activityParse(t, activityAssistant("msg-1", "req-1", []any{activityTool("tool-1", "Bash", activityJSON{"command": tc.command})}, nil), activityResult("tool-1", tc.result, tc.failed)) + if got := activitySum(r, "claude_code.commit.count", ""); got != tc.want { + t.Errorf("%q: got %v commits, want %v", tc.command, got, tc.want) + } + }) + } +} + +func TestClaudeActivityFailedEdit(t *testing.T) { + r := activityParse(t, + activityAssistant("msg-1", "req-1", []any{activityTool("tool-1", "Edit", activityJSON{"file_path": "/project/a.go", "old_string": "old", "new_string": "new"})}, nil), + activityResult("tool-1", "String to replace not found in file", true), + ) + if got := activitySum(r, "claude_code.lines_of_code.count", ""); got != 0 { + t.Errorf("failed edit: got %v changed lines, want 0", got) + } +} + +func TestClaudeActivityLOCDelta(t *testing.T) { + t.Run("unchanged_context", func(t *testing.T) { + r := activityParse(t, + activityAssistant("msg-1", "req-1", []any{activityTool("tool-1", "Edit", activityJSON{"file_path": "/project/a.go", "old_string": "keep\nold\nkeep", "new_string": "keep\nnew\nkeep"})}, nil), + activityResult("tool-1", "The file has been updated successfully", false), + ) + for _, kind := range []string{"added", "removed"} { + if got := activitySum(r, "claude_code.lines_of_code.count", kind); got != 1 { + t.Errorf("one-line replacement: got %v %s, want 1", got, kind) + } + } + }) + t.Run("trailing_newline", func(t *testing.T) { + r := activityParse(t, + activityAssistant("msg-1", "req-1", []any{activityTool("tool-1", "Write", activityJSON{"file_path": "/project/a.go", "content": "a\nb\n"})}, nil), + activityResult("tool-1", "File created successfully", false), + ) + if got := activitySum(r, "claude_code.lines_of_code.count", "added"); got != 2 { + t.Errorf("two-line file: got %v added, want 2", got) + } + }) +} + +func TestClaudeActivityMultiplePRs(t *testing.T) { + r := activityParse(t, + activityJSON{"type": "pr-link", "timestamp": "2026-09-01T12:00:00Z", "prNumber": 7, "prRepository": "acme/app", "prUrl": "https://github.com/acme/app/pull/7"}, + activityJSON{"type": "pr-link", "timestamp": "2026-09-01T12:00:01Z", "prNumber": 8, "prRepository": "acme/app", "prUrl": "https://github.com/acme/app/pull/8"}, + activityJSON{"type": "pr-link", "timestamp": "2026-09-01T12:00:02Z", "prNumber": 7, "prRepository": "acme/app", "prUrl": "https://github.com/acme/app/pull/7"}, + ) + if got := activitySum(r, "claude_code.pull_request.count", ""); got != 2 { + t.Errorf("two distinct PRs plus repeated link: got %v, want 2", got) + } +} + +func TestClaudeActivityPRDateRange(t *testing.T) { + file := activityFile(t, + activityAssistant("msg-1", "req-1", []any{activityJSON{"type": "text", "text": "Opening a PR"}}, nil), + activityJSON{"type": "pr-link", "timestamp": "2026-09-02T00:01:00Z", "prNumber": 7, "prRepository": "acme/app", "prUrl": "https://github.com/acme/app/pull/7"}, + ) + p := NewClaudeParser() + p.configPaths = []string{filepath.Dir(file)} + r, err := p.ParseFile(context.Background(), file) + if err != nil { + t.Fatal(err) + } + if got := activitySum(r, "claude_code.pull_request.count", ""); got != 1 { + t.Fatalf("fixture PR count = %v", got) + } + store, err := storage.NewDuckDBStore(":memory:") + if err != nil { + t.Fatal(err) + } + defer store.Close() + i := NewImporter(store, false) + i.RegisterParser(p) + from, _ := time.Parse(time.RFC3339, "2026-09-02T00:00:00Z") + if err := i.Import(context.Background(), []SourceType{SourceClaude}, Options{FromDate: &from, SkipConfirm: true}); err != nil { + t.Fatal(err) + } + var count int + if err := store.DB().QueryRow("SELECT count(*) FROM otel_metrics WHERE MetricName = 'claude_code.pull_request.count'").Scan(&count); err != nil { + t.Fatal(err) + } + if count != 1 { + t.Errorf("PR inside selected date range: imported %d, want 1 (parser LastTime=%s)", count, r.LastTime) + } +} + +func TestClaudeActivityRepositoryAttribution(t *testing.T) { + dir := t.TempDir() + for _, args := range [][]string{{"init", "--quiet", dir}, {"-C", dir, "config", "remote.origin.url", "https://github.com/acme/local-project.git"}} { + if out, err := exec.Command("git", args...).CombinedOutput(); err != nil { + t.Fatalf("git fixture: %v: %s", err, out) + } + } + a := activityAssistant("msg-1", "req-1", nil, activityJSON{"input_tokens": 100}) + a["cwd"] = dir + r := activityParse(t, a, activityJSON{"type": "pr-link", "timestamp": "2026-09-02T00:01:00Z", "prNumber": 7, "prRepository": "other/external-project", "prUrl": "https://github.com/other/external-project/pull/7"}) + for _, m := range r.Metrics { + if m.MetricName == "claude_code.token.usage" && m.Attributes["repository"] != "acme/local-project" { + t.Errorf("usage in local git repository attributed to %q, want acme/local-project", m.Attributes["repository"]) + } + } +} + +func TestClaudeActivitySuccessfulResultsAndDeduplication(t *testing.T) { + for _, tc := range []struct { + name string + result activityJSON + want float64 + }{ + {"missing", nil, 0}, + {"successful", activityResult("tool-1", "File created successfully", false), 1}, + {"failed", activityResult("tool-1", "Permission denied", true), 0}, + {"interrupted", func() activityJSON { + r := activityResult("tool-1", "File created successfully", false) + r["toolUseResult"] = activityJSON{"interrupted": true} + return r + }(), 0}, + } { + t.Run(tc.name, func(t *testing.T) { + call := activityAssistant("msg-1", "req-1", []any{activityTool("tool-1", "Write", activityJSON{"file_path": "/project/a.go", "content": "a"})}, activityJSON{"input_tokens": 100}) + entries := []activityJSON{call, call} + if tc.result != nil { + entries = append(entries, tc.result, tc.result, call, tc.result) + } + r := activityParse(t, entries...) + if got := activitySum(r, claudeLinesOfCodeMetric, "added"); got != tc.want { + t.Errorf("added = %v, want %v", got, tc.want) + } + if got := activitySum(r, ClaudeTokenUsageMetric, "input"); got != 100 { + t.Errorf("duplicate usage: got %v, want 100", got) + } + }) + } +} + +func TestClaudeActivityRecordedEdits(t *testing.T) { + for _, tc := range []struct { + name, tool string + input, result activityJSON + added, removed float64 + }{ + {"write_create", "Write", activityJSON{"content": "a\nb\n"}, activityJSON{"type": "create", "originalFile": nil}, 2, 0}, + {"write_overwrite", "Write", activityJSON{"content": "keep\nnew\nkeep\n"}, activityJSON{"type": "update", "originalFile": "keep\nold\nkeep\n"}, 1, 1}, + {"write_unknown_original", "Write", activityJSON{"content": "a\nb\n"}, activityJSON{"type": "update"}, 0, 0}, + {"replace_all_patch", "Edit", activityJSON{"old_string": "old", "new_string": "new", "replace_all": true}, activityJSON{"structuredPatch": []any{activityJSON{"lines": []string{" context", "-old", "+new", " more", "-old", "+new"}}}}, 2, 2}, + {"replace_all_missing_patch", "Edit", activityJSON{"old_string": "old", "new_string": "new", "replace_all": true}, nil, 0, 0}, + {"no_change", "Edit", activityJSON{"old_string": "same\n", "new_string": "same\n"}, nil, 0, 0}, + {"multiedit", "MultiEdit", activityJSON{"edits": []any{activityJSON{"old_string": "line1\nline2", "new_string": "line1\nline2\nline3"}, activityJSON{"old_string": "foo", "new_string": ""}}}, nil, 1, 1}, + {"wrong_file_patch", "Edit", activityJSON{"old_string": "old", "new_string": "new"}, activityJSON{"filePath": "/different/file.go", "structuredPatch": []any{activityJSON{"lines": []string{"-old", "+new"}}}}, 0, 0}, + } { + t.Run(tc.name, func(t *testing.T) { + tc.input["file_path"] = "/project/file.go" + result := activityResult("tool-1", "The file has been updated successfully", false) + result["toolUseResult"] = tc.result + r := activityParse(t, activityAssistant("msg", "req", []any{activityTool("tool-1", tc.tool, tc.input)}, nil), result) + if a, d := activitySum(r, claudeLinesOfCodeMetric, "added"), activitySum(r, claudeLinesOfCodeMetric, "removed"); a != tc.added || d != tc.removed { + t.Errorf("got +%v/-%v, want +%v/-%v", a, d, tc.added, tc.removed) + } + }) + } +} + +func TestClaudeActivityToolResultContentBlocks(t *testing.T) { + result := activityResult("tool-1", "", false) + result["message"].(activityJSON)["content"].([]any)[0].(activityJSON)["content"] = []any{activityJSON{"type": "text", "text": "[main (root-commit) abc1234] test"}, activityJSON{"type": "image", "source": activityJSON{"type": "base64", "data": ""}}} + r := activityParse(t, activityAssistant("msg", "req", []any{activityTool("tool-1", "Bash", activityJSON{"command": "git -c user.name=test commit -m test"})}, nil), result) + if got := activitySum(r, claudeCommitMetric, ""); got != 1 { + t.Errorf("commit count = %v, want 1", got) + } + for _, log := range r.Logs { + if log.LogAttributes["message.role"] == "tool_result" && strings.Contains(log.Body, "abc1234") { + return + } + } + t.Fatal("array tool result missing from transcript") +} + +func TestClaudeActivityCommitCommands(t *testing.T) { + for _, command := range []string{ + "git add . && git commit -m 'test'", "git -C '/some dir' -c user.name=Test commit -m test", + "git --git-dir=/repo/.git --work-tree=/repo commit -m test", "'git' \"commit\" -m test", + } { + t.Run(command, func(t *testing.T) { + r := activityParse(t, activityAssistant("msg", "req", []any{activityTool("tool", "Bash", activityJSON{"command": command})}, nil), activityResult("tool", "[main abc1234] one\n[main def5678] two\n[main abc1234] repeated", false)) + if got := activitySum(r, claudeCommitMetric, ""); got != 2 { + t.Errorf("commits = %v, want 2", got) + } + }) + } + for _, command := range []string{"echo 'git commit'", "cat <<'EOF'\ngit commit\nEOF", "git status # git commit", "f() { git commit; }", "git log --format='git commit'", "echo git commit", "git status; echo 'unterminated"} { + t.Run(command, func(t *testing.T) { + r := activityParse(t, activityAssistant("msg", "req", []any{activityTool("tool", "Bash", activityJSON{"command": command})}, nil), activityResult("tool", "[main abc1234] text", false)) + if got := activitySum(r, claudeCommitMetric, ""); got != 0 { + t.Errorf("commits = %v, want 0", got) + } + }) + } +} + +func TestClaudeActivityPRIdentityAndTimeBounds(t *testing.T) { + r := activityParse(t, + activityJSON{"type": "pr-link", "timestamp": "invalid", "prNumber": 7, "prRepository": "acme/app"}, + activityJSON{"type": "pr-link", "timestamp": "2026-09-01T12:00:00Z", "prNumber": 7, "prRepository": "acme/app"}, + activityJSON{"type": "pr-link", "timestamp": "2026-09-02T12:00:00Z", "prNumber": 7, "prRepository": "other/app"}, + activityJSON{"type": "pr-link", "timestamp": "2026-09-03T12:00:00Z", "prNumber": 7, "prUrl": "https://github.com/acme/app/pull/7"}, + ) + if got := activitySum(r, claudePullRequestMetric, ""); got != 2 { + t.Errorf("PR count = %v, want 2", got) + } + if r.FirstTime.Format(time.RFC3339) != "2026-09-01T12:00:00Z" || r.LastTime.Format(time.RFC3339) != "2026-09-02T12:00:00Z" { + t.Errorf("wrong PR-only bounds: %s to %s", r.FirstTime, r.LastTime) + } + repos := map[string]bool{} + for _, m := range r.Metrics { + if m.MetricName == claudePullRequestMetric { + repos[m.Attributes["repository"]] = true + } + } + if !repos["acme/app"] || !repos["other/app"] { + t.Fatalf("PR repositories: %v", repos) + } +} + +func TestClaudeActivitySessionAndActiveTime(t *testing.T) { + message := activityAssistant("msg", "req", []any{activityJSON{"type": "text", "text": "hello"}}, nil) + message["gitBranch"] = "feature" + r := activityParse(t, message, + activityJSON{"type": "system", "subtype": "turn_duration", "timestamp": "2026-09-02T00:01:00Z", "durationMs": 30000}, + activityJSON{"type": "system", "subtype": "turn_duration", "timestamp": "2026-09-02T00:02:00Z", "durationMs": 15000}, + activityJSON{"type": "system", "subtype": "turn_duration", "timestamp": "2026-09-02T00:03:00Z", "durationMs": -1}, + ) + if got := activitySum(r, claudeSessionMetric, ""); got != 1 { + t.Errorf("session count = %v", got) + } + if got := activitySum(r, claudeActiveTimeMetric, "cli"); got != 45 { + t.Errorf("CLI active seconds = %v, want 45", got) + } + for _, m := range r.Metrics { + if m.Attributes["git_branch"] != "feature" { + t.Errorf("missing branch: %v", m.Attributes) + } + if m.MetricName == claudeActiveTimeMetric && m.MetricUnit != "s" { + t.Errorf("active time unit = %q", m.MetricUnit) + } + } + for _, log := range r.Logs { + if log.LogAttributes["git_branch"] != "feature" { + t.Errorf("missing transcript branch: %v", log.LogAttributes) + } + if log.LogAttributes["pr_number"] != "" { + t.Error("session transcript should not inherit an unrelated PR") + } + } +} + +func TestClaudeActivityDiffBoundaries(t *testing.T) { + for _, tc := range []struct { + old, new string + added, removed int + }{ + {"", "a\nb\n", 2, 0}, {"a\n", "", 0, 1}, {"a\nold\nkeep\nold2\nz", "a\nnew\nkeep\nnew2\nz", 2, 2}, {"same\n", "same", 0, 0}, {"\n", "", 0, 1}, + } { + a, r, ok := changedLines(tc.old, tc.new) + if !ok || a != tc.added || r != tc.removed { + t.Errorf("diff %q -> %q: %d/%d/%v", tc.old, tc.new, a, r, ok) + } + } + if _, _, ok := changedLines(strings.Repeat("a\n", 2500), strings.Repeat("b\n", 2500)); ok { + t.Error("large diff should be omitted") + } +} + +func TestCodexRepositoryFromSessionMetadata(t *testing.T) { + for _, remote := range []string{"https://github.com/acme/project.git", "git@github.com:acme/project.git", "https://example-token@github.com/acme/project.git"} { + t.Run(remote, func(t *testing.T) { + file := activityFile(t, + activityJSON{"type": "session_meta", "timestamp": "2026-09-01T12:00:00Z", "payload": activityJSON{"id": "codex-session", "cwd": "/missing/unrelated", "model": "gpt-4o", "git": activityJSON{"repository_url": remote}}}, + activityJSON{"type": "event_msg", "timestamp": "2026-09-01T12:01:00Z", "payload": activityJSON{"type": "token_count", "info": activityJSON{"total_token_usage": activityJSON{"input_tokens": 100, "output_tokens": 50}}}}, + ) + r, err := NewCodexParser().ParseFile(context.Background(), file) + if err != nil { + t.Fatal(err) + } + if len(r.Metrics) == 0 { + t.Fatal("no Codex usage metrics") + } + for _, m := range r.Metrics { + if m.Attributes["repository"] != "acme/project" { + t.Errorf("repository = %q", m.Attributes["repository"]) + } + } + }) + } +} diff --git a/backend/internal/importer/codex.go b/backend/internal/importer/codex.go index 816b3e3..07f8860 100644 --- a/backend/internal/importer/codex.go +++ b/backend/internal/importer/codex.go @@ -95,8 +95,8 @@ type CodexJSONLEntry struct { type CodexResponseItem struct { Type string `json:"type"` // For "message" type - Role string `json:"role,omitempty"` - Content []CodexContentItem `json:"content,omitempty"` + Role string `json:"role,omitempty"` + Content []CodexContentItem `json:"content,omitempty"` // For "function_call" type Name string `json:"name,omitempty"` Arguments string `json:"arguments,omitempty"` @@ -119,22 +119,28 @@ type CodexReasoningSummary struct { Text string `json:"text,omitempty"` } +// codexGitInfo contains git metadata from session_meta +type codexGitInfo struct { + RepositoryURL string `json:"repository_url"` +} + // CodexSessionMeta represents session metadata type CodexSessionMeta struct { - ID string `json:"id"` - Timestamp string `json:"timestamp"` - Cwd string `json:"cwd"` - Originator string `json:"originator"` - CliVersion string `json:"cli_version"` - ModelProvider string `json:"model_provider"` - Model string `json:"model"` + ID string `json:"id"` + Timestamp string `json:"timestamp"` + Cwd string `json:"cwd"` + Originator string `json:"originator"` + CliVersion string `json:"cli_version"` + ModelProvider string `json:"model_provider"` + Model string `json:"model"` + Git *codexGitInfo `json:"git,omitempty"` } // CodexEventMsg represents an event message payload // The actual JSON structure has "info" at the same level as "type", not nested in another "payload" type CodexEventMsg struct { - Type string `json:"type"` - Info *CodexTokenInfo `json:"info"` // Present for token_count events + Type string `json:"type"` + Info *CodexTokenInfo `json:"info"` // Present for token_count events } // CodexTokenInfo contains token usage information @@ -176,6 +182,7 @@ func (p *CodexParser) ParseFile(ctx context.Context, path string) (*ImportResult var sessionMeta *CodexSessionMeta var currentModel string var lastTokenCount *CodexTokenCount + var repository string messageIndex := 0 // Track message order for transcripts for scanner.Scan() { @@ -222,6 +229,13 @@ func (p *CodexParser) ParseFile(ctx context.Context, path string) (*ImportResult currentModel = meta.Model } + // Resolve repository using git URL from session metadata (authoritative for Codex). + gitRepoURL := "" + if meta.Git != nil { + gitRepoURL = meta.Git.RepositoryURL + } + repository = resolveSessionRepository(nil, gitRepoURL, "", meta.Cwd) + // Create session start log logRecord := api.LogRecord{ Timestamp: ts, @@ -230,17 +244,20 @@ func (p *CodexParser) ParseFile(ctx context.Context, path string) (*ImportResult SeverityNumber: 9, Body: "conversation_starts", LogAttributes: map[string]string{ - "event.name": "codex.conversation_starts", - "session.id": meta.ID, - "model": meta.Model, - "model_provider": meta.ModelProvider, - "cli_version": meta.CliVersion, - "import_source": "local_jsonl", + "event.name": "codex.conversation_starts", + "session.id": meta.ID, + "model": meta.Model, + "model_provider": meta.ModelProvider, + "cli_version": meta.CliVersion, + "import_source": "local_jsonl", }, } if meta.Cwd != "" { logRecord.LogAttributes["cwd"] = meta.Cwd } + if repository != "" { + logRecord.LogAttributes["repository"] = repository + } result.Logs = append(result.Logs, logRecord) result.RecordCount++ } @@ -288,29 +305,29 @@ func (p *CodexParser) ParseFile(ctx context.Context, path string) (*ImportResult // Create metrics for non-zero deltas if deltaInput > 0 { - result.Metrics = append(result.Metrics, CreateCodexTokenMetric(ts, currentModel, "input", float64(deltaInput))) + result.Metrics = append(result.Metrics, createCodexTokenMetric(ts, currentModel, "input", float64(deltaInput), repository)) } if deltaOutput > 0 { - result.Metrics = append(result.Metrics, CreateCodexTokenMetric(ts, currentModel, "output", float64(deltaOutput))) + result.Metrics = append(result.Metrics, createCodexTokenMetric(ts, currentModel, "output", float64(deltaOutput), repository)) } if deltaCacheCreation > 0 { - result.Metrics = append(result.Metrics, CreateCodexTokenMetric(ts, currentModel, "cache_creation", float64(deltaCacheCreation))) + result.Metrics = append(result.Metrics, createCodexTokenMetric(ts, currentModel, "cache_creation", float64(deltaCacheCreation), repository)) } if deltaCacheRead > 0 { - result.Metrics = append(result.Metrics, CreateCodexTokenMetric(ts, currentModel, "cache_read", float64(deltaCacheRead))) + result.Metrics = append(result.Metrics, createCodexTokenMetric(ts, currentModel, "cache_read", float64(deltaCacheRead), repository)) } if deltaReasoning > 0 { - result.Metrics = append(result.Metrics, CreateCodexTokenMetric(ts, currentModel, "reasoning", float64(deltaReasoning))) + result.Metrics = append(result.Metrics, createCodexTokenMetric(ts, currentModel, "reasoning", float64(deltaReasoning), repository)) } if deltaTool > 0 { - result.Metrics = append(result.Metrics, CreateCodexTokenMetric(ts, currentModel, "tool", float64(deltaTool))) + result.Metrics = append(result.Metrics, createCodexTokenMetric(ts, currentModel, "tool", float64(deltaTool), repository)) } // Calculate and add cost metric // Note: cache_read is used for cost calculation (cache_creation tokens are billed at input rate) cost := pricing.CalculateCodexCost(currentModel, int64(deltaInput), int64(deltaCacheRead), int64(deltaOutput)) if cost != nil && *cost > 0 { - result.Metrics = append(result.Metrics, CreateCodexCostMetric(ts, currentModel, *cost)) + result.Metrics = append(result.Metrics, createCodexCostMetric(ts, currentModel, *cost, repository)) } lastTokenCount = tokenCount @@ -518,33 +535,55 @@ func (p *CodexParser) ParseFile(ctx context.Context, path string) (*ImportResult return result, nil } -// CreateCodexTokenMetric creates a token usage metric for Codex -func CreateCodexTokenMetric(ts time.Time, model, tokenType string, value float64) api.MetricDataPoint { +// createCodexTokenMetric creates a token usage metric for Codex with optional repository attribution. +func createCodexTokenMetric(ts time.Time, model, tokenType string, value float64, repository string) api.MetricDataPoint { + attrs := map[string]string{ + "type": tokenType, + "model": model, + "import_source": "local_jsonl", + } + if repository != "" { + attrs["repository"] = repository + } return api.MetricDataPoint{ Timestamp: ts, ServiceName: SourceCodex.ServiceName(), MetricName: "codex_cli_rs.token.usage", MetricType: "sum", Value: &value, - Attributes: map[string]string{ - "type": tokenType, - "model": model, - "import_source": "local_jsonl", - }, + Attributes: attrs, } } -// CreateCodexCostMetric creates a cost usage metric for Codex -func CreateCodexCostMetric(ts time.Time, model string, cost float64) api.MetricDataPoint { +// CreateCodexTokenMetric creates a token usage metric for Codex. +// This public form is used by the file watcher; ParseFile uses createCodexTokenMetric +// (private) which also attaches repository attribution. +func CreateCodexTokenMetric(ts time.Time, model, tokenType string, value float64) api.MetricDataPoint { + return createCodexTokenMetric(ts, model, tokenType, value, "") +} + +// createCodexCostMetric creates a cost usage metric for Codex with optional repository attribution. +func createCodexCostMetric(ts time.Time, model string, cost float64, repository string) api.MetricDataPoint { + attrs := map[string]string{ + "model": model, + "import_source": "local_jsonl", + } + if repository != "" { + attrs["repository"] = repository + } return api.MetricDataPoint{ Timestamp: ts, ServiceName: SourceCodex.ServiceName(), MetricName: "codex_cli_rs.cost.usage", MetricType: "sum", Value: &cost, - Attributes: map[string]string{ - "model": model, - "import_source": "local_jsonl", - }, + Attributes: attrs, } } + +// CreateCodexCostMetric creates a cost usage metric for Codex. +// This public form is used by the file watcher; ParseFile uses createCodexCostMetric +// (private) which also attaches repository attribution. +func CreateCodexCostMetric(ts time.Time, model string, cost float64) api.MetricDataPoint { + return createCodexCostMetric(ts, model, cost, "") +} diff --git a/backend/internal/importer/importer_test.go b/backend/internal/importer/importer_test.go index 3dfaf6a..928d425 100644 --- a/backend/internal/importer/importer_test.go +++ b/backend/internal/importer/importer_test.go @@ -846,6 +846,26 @@ func TestImportUnregisteredParser(t *testing.T) { } } +// TestRelativeFilePath tests relative path extraction +func TestRelativeFilePath(t *testing.T) { + tests := []struct { + filePath string + baseDir string + expected string + }{ + {"/home/user/project/main.go", "/home/user/project", "main.go"}, + {"/home/user/project/sub/file.ts", "/home/user/project", "sub/file.ts"}, + {"/home/user/project/main.go", "", "main.go"}, + {"/other/path/file.py", "/home/user/project", "file.py"}, // outside base -> basename + } + for _, tc := range tests { + got := relativeFilePath(tc.filePath, tc.baseDir) + if got != tc.expected { + t.Errorf("relativeFilePath(%q, %q) = %q, want %q", tc.filePath, tc.baseDir, got, tc.expected) + } + } +} + // Helper functions func floatPtr(f float64) *float64 { return &f diff --git a/backend/internal/importer/repos.go b/backend/internal/importer/repos.go new file mode 100644 index 0000000..ccd73eb --- /dev/null +++ b/backend/internal/importer/repos.go @@ -0,0 +1,281 @@ +package importer + +import ( + "os" + "os/exec" + "path/filepath" + "regexp" + "strconv" + "strings" + "sync" +) + +// Repository attribution logic. +// +// Ported from ducktrace/claude_analysis/repos.py and prmatch.py. +// Maps a session to a canonical "owner/repo" (or bare repo name) using the +// strongest available signal. +// +// Signal priority (best first): +// 1. Codex session_meta.git.repository_url (authoritative) +// 2. On-disk git remote from cwd (real remote, authoritative) +// 3. Candidate repos whose name matches cwd name or path segment +// 4. Bare cwd directory name +// 5. First candidate repository reference + +var ( + githubURLRe = regexp.MustCompile(`(?i)(?:^|://|@)(?:[^/@]+@)?github\.com[:/]+([^/]+/[^/?#]+)`) + schemeRe = regexp.MustCompile(`(?i)^[a-z][a-z0-9+.-]*://`) + userAtRe = regexp.MustCompile(`^[^@/]+@`) + + ghPRRe = regexp.MustCompile(`(?i)\bgh\s+pr\s+(create|merge|close|reopen|checkout|view|diff|review|list|edit|comment|ready|status)\b`) + repoFlagRe = regexp.MustCompile(`(?i)--repo[ =]\s*['"]?([^\s'"]+)`) + pullURLRe = regexp.MustCompile(`(?i)https?://github\.com/([^/\s]+/[^/\s]+?)/pull/(\d+)`) + // repoURLRe catches any github.com/owner/repo reference not already matched by pullURLRe. + // The lazy second component combined with the required terminator stops at path separators. + repoURLRe = regexp.MustCompile(`(?i)github\.com[:/]+([^/\s]+/[^/\s]+?)(?:\.git)?(?:[/\s)"',` + "`" + `]|$)`) +) + +var placeholderRepos = map[string]bool{ + "owner/repo": true, "owner/name": true, "org/repo": true, + "user/repo": true, "your-org/your-repo": true, + "username/repo": true, "owner/repository": true, +} + +var gitRemoteCache sync.Map + +// githubRepoFromURL extracts owner/repo from any GitHub URL (https or ssh), +// stripping embedded credentials and a trailing .git. Returns "" on no match. +func githubRepoFromURL(url string) string { + if url == "" { + return "" + } + m := githubURLRe.FindStringSubmatch(url) + if m == nil { + return "" + } + return strings.TrimSuffix(m[1], ".git") +} + +// normalizeGitURL normalizes any git remote URL to owner/repo when possible. +// Handles git@host:owner/repo.git, https://host/owner/repo.git, and +// https://@host/owner/repo.git (credentials are dropped). Falls back +// to the last two path segments for non-GitHub hosts. +func normalizeGitURL(url string) string { + if gh := githubRepoFromURL(url); gh != "" { + return gh + } + if url == "" { + return "" + } + u := schemeRe.ReplaceAllString(strings.TrimSpace(url), "") + u = userAtRe.ReplaceAllString(u, "") + u = strings.Replace(u, ":", "/", 1) // ssh host:path -> host/path + u = strings.TrimSuffix(u, ".git") + var parts []string + for _, p := range strings.Split(u, "/") { + if p != "" { + parts = append(parts, p) + } + } + if len(parts) >= 3 { // host/owner/repo... + return parts[len(parts)-2] + "/" + parts[len(parts)-1] + } + return u +} + +// resolveRepoName returns the repo name for a working dir, resolving git +// worktrees to the main repo's directory name. If the path no longer exists +// we just return its basename. +func resolveRepoName(cwd string) string { + if cwd == "" { + return "" + } + // A worktree's .git is a file (not a dir) containing "gitdir: ". + content, err := os.ReadFile(filepath.Join(cwd, ".git")) + if err == nil { + text := strings.TrimSpace(string(content)) + if strings.HasPrefix(text, "gitdir:") { + gitdir := strings.TrimSpace(strings.TrimPrefix(text, "gitdir:")) + parts := strings.Split(filepath.ToSlash(gitdir), "/") + for i, seg := range parts { + if seg == ".git" && i > 0 { + return filepath.Base(filepath.Join(parts[:i]...)) + } + } + } + } + return filepath.Base(cwd) +} + +// cwdGitRemote returns the owner/repo of the cwd's origin remote, cached per +// path. This is authoritative when the directory still exists locally - the +// real remote is correct even when the local dir name differs from the repo name. +func cwdGitRemote(path string) string { + if path == "" { + return "" + } + if cached, ok := gitRemoteCache.Load(path); ok { + return cached.(string) + } + out, err := exec.Command("git", "-C", path, "config", "--get", "remote.origin.url").Output() + result := "" + if err == nil { + result = normalizeGitURL(strings.TrimSpace(string(out))) + } + gitRemoteCache.Store(path, result) + return result +} + +// resolveSessionRepository returns the best repository id for a session. +// +// candidateRepos are owner/repo references seen for the session - from pr-link +// entries and text-mined references - ideally most-frequent first. +// +// The working directory says which repo we're in; candidates supply the owner/ +// prefix the cwd alone can't. A candidate whose repo-part matches the cwd name +// or a path segment is the highest-confidence answer. A candidate that doesn't +// match is treated as a stray and used only when there's no cwd at all. +func resolveSessionRepository(candidateRepos []string, gitRepoURL, originalCwd, cwd string) string { + // Codex carries the cwd's actual git remote - authoritative. + if gitRepoURL != "" { + if norm := normalizeGitURL(gitRepoURL); norm != "" { + return norm + } + } + + path := originalCwd + if path == "" { + path = cwd + } + + // On-disk git remote is authoritative (handles dir name != repo name). + if onDisk := cwdGitRemote(path); onDisk != "" { + return onDisk + } + + cwdName := resolveRepoName(path) + var candidates []string + for _, r := range candidateRepos { + if r != "" { + candidates = append(candidates, r) + } + } + + if len(candidates) > 0 && path != "" { + segments := make(map[string]bool) + for _, seg := range strings.Split(filepath.ToSlash(path), "/") { + if seg != "" { + segments[strings.ToLower(seg)] = true + } + } + for _, r := range candidates { + parts := strings.Split(r, "/") + name := strings.ToLower(parts[len(parts)-1]) + if name == strings.ToLower(cwdName) || segments[name] { + return r + } + } + } + + if cwdName != "" { + return cwdName + } + if len(candidates) > 0 { + return candidates[0] + } + return "" +} + +// isPlaceholder returns true if repo is a documentation/example placeholder. +func isPlaceholder(repo string) bool { + return placeholderRepos[strings.ToLower(repo)] || + strings.Contains(repo, "<") || strings.Contains(repo, ">") +} + +func normalizeRepoFlag(val string) string { + if gh := githubRepoFromURL(val); gh != "" { + if isPlaceholder(gh) { + return "" + } + return gh + } + val = strings.TrimRight(strings.TrimSpace(val), "/") + var parts []string + for _, p := range strings.Split(val, "/") { + if p != "" { + parts = append(parts, p) + } + } + if len(parts) >= 2 && !strings.Contains(parts[len(parts)-2], ".") { + repo := strings.TrimSuffix(parts[len(parts)-2]+"/"+parts[len(parts)-1], ".git") + if isPlaceholder(repo) { + return "" + } + return repo + } + return "" +} + +// extractedRefs holds PR/repo references mined from free text. +type extractedRefs struct { + PRActions []string + PRURLs []string + PRNumbers []int + Repos []string +} + +// extractRefs scans one or more strings for gh pr commands, PR URLs, +// --repo flags, and GitHub repo references. +func extractRefs(texts ...string) extractedRefs { + result := extractedRefs{} + seenActions := map[string]bool{} + seenURLs := map[string]bool{} + seenNumbers := map[int]bool{} + seenRepos := map[string]bool{} + + for _, text := range texts { + if text == "" { + continue + } + for _, m := range ghPRRe.FindAllStringSubmatch(text, -1) { + action := "gh pr " + strings.ToLower(m[1]) + if !seenActions[action] { + seenActions[action] = true + result.PRActions = append(result.PRActions, action) + } + } + for _, m := range pullURLRe.FindAllStringSubmatch(text, -1) { + repo := strings.TrimSuffix(m[1], ".git") + if isPlaceholder(repo) { + continue + } + if !seenRepos[repo] { + seenRepos[repo] = true + result.Repos = append(result.Repos, repo) + } + if num, err := strconv.Atoi(m[2]); err == nil && !seenNumbers[num] { + seenNumbers[num] = true + result.PRNumbers = append(result.PRNumbers, num) + } + if url := m[0]; !seenURLs[url] { + seenURLs[url] = true + result.PRURLs = append(result.PRURLs, url) + } + } + for _, m := range repoFlagRe.FindAllStringSubmatch(text, -1) { + if r := normalizeRepoFlag(m[1]); r != "" && !seenRepos[r] { + seenRepos[r] = true + result.Repos = append(result.Repos, r) + } + } + for _, m := range repoURLRe.FindAllStringSubmatch(text, -1) { + r := strings.TrimSuffix(m[1], ".git") + if !isPlaceholder(r) && !seenRepos[r] { + seenRepos[r] = true + result.Repos = append(result.Repos, r) + } + } + } + return result +} diff --git a/backend/internal/logger/logger.go b/backend/internal/logger/logger.go index f1c8b8d..13669d7 100644 --- a/backend/internal/logger/logger.go +++ b/backend/internal/logger/logger.go @@ -4,13 +4,21 @@ import ( "context" "log/slog" "os" + "sync" ) -// Global logger instance -var defaultLogger *slog.Logger +// Protect both lazy initialization and explicit reconfiguration. Callers keep +// the returned logger and emit records after the lock has been released. +var ( + defaultLoggerMu sync.Mutex + defaultLogger *slog.Logger +) // Initialize sets up the global structured logger func Initialize(level slog.Level) { + defaultLoggerMu.Lock() + defer defaultLoggerMu.Unlock() + opts := &slog.HandlerOptions{ Level: level, } @@ -21,6 +29,9 @@ func Initialize(level slog.Level) { // InitializeText sets up a text-based logger (better for development) func InitializeText(level slog.Level) { + defaultLoggerMu.Lock() + defer defaultLoggerMu.Unlock() + opts := &slog.HandlerOptions{ Level: level, } @@ -31,8 +42,13 @@ func InitializeText(level slog.Level) { // Logger returns the default logger func Logger() *slog.Logger { + defaultLoggerMu.Lock() + defer defaultLoggerMu.Unlock() + if defaultLogger == nil { - Initialize(slog.LevelInfo) + handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}) + defaultLogger = slog.New(handler) + slog.SetDefault(defaultLogger) } return defaultLogger } diff --git a/backend/internal/logger/logger_test.go b/backend/internal/logger/logger_test.go new file mode 100644 index 0000000..e367e0b --- /dev/null +++ b/backend/internal/logger/logger_test.go @@ -0,0 +1,112 @@ +package logger + +import ( + "context" + "log/slog" + "sync" + "testing" +) + +// These tests run serially and join all workers before restoring global state. +func resetLogger(t *testing.T) { + t.Helper() + defaultLoggerMu.Lock() + previous, previousSlog := defaultLogger, slog.Default() + defaultLogger = nil + defaultLoggerMu.Unlock() + t.Cleanup(func() { + defaultLoggerMu.Lock() + defer defaultLoggerMu.Unlock() + defaultLogger = previous + slog.SetDefault(previousSlog) + }) +} + +func TestLoggerConcurrentFirstUse(t *testing.T) { + resetLogger(t) + const workers = 64 + start := make(chan struct{}) + results := make(chan *slog.Logger, workers) + var wg sync.WaitGroup + for range workers { + wg.Go(func() { + <-start + log := Logger() + log.Debug("suppressed first-use message") + results <- log + }) + } + close(start) + wg.Wait() + close(results) + current := Logger() + for log := range results { + if log != current { + t.Error("concurrent first use returned different default loggers") + } + } + if current != slog.Default() { + t.Error("package logger and slog default differ") + } +} + +func TestLoggerConcurrentReconfiguration(t *testing.T) { + resetLogger(t) + Initialize(slog.LevelWarn) + start := make(chan struct{}) + var wg sync.WaitGroup + for worker := range 16 { + wg.Go(func() { + <-start + for range 100 { + switch worker % 4 { + case 0: + Initialize(slog.LevelWarn) + case 1: + InitializeText(slog.LevelError) + case 2: + DebugContext(context.Background(), "suppressed concurrent message") + default: + WithRequestID("test").Debug("suppressed child logger message") + } + } + }) + } + close(start) + wg.Wait() + if Logger() != slog.Default() { + t.Error("reconfiguration left package logger and slog default inconsistent") + } +} + +func TestLoggerConfiguration(t *testing.T) { + for _, tc := range []struct { + name string + configure func() + text bool + level slog.Level + }{ + {"default", func() {}, false, slog.LevelInfo}, + {"json_before_first_use", func() { Initialize(slog.LevelWarn) }, false, slog.LevelWarn}, + {"text_before_first_use", func() { InitializeText(slog.LevelDebug) }, true, slog.LevelDebug}, + {"reconfigure_after_first_use", func() { Logger(); InitializeText(slog.LevelError) }, true, slog.LevelError}, + } { + t.Run(tc.name, func(t *testing.T) { + resetLogger(t) + tc.configure() + log := Logger() + _, isText := log.Handler().(*slog.TextHandler) + _, isJSON := log.Handler().(*slog.JSONHandler) + if isText != tc.text || isJSON == tc.text { + t.Fatalf("unexpected handler %T", log.Handler()) + } + ctx := context.Background() + if !log.Enabled(ctx, tc.level) || log.Enabled(ctx, tc.level-1) { + t.Error("configured log threshold was not preserved") + } + if log != slog.Default() { + t.Error("package logger and slog default differ") + } + }) + } +} diff --git a/backend/internal/storage/duckdb_strings_test.go b/backend/internal/storage/duckdb_strings_test.go new file mode 100644 index 0000000..96beaaa --- /dev/null +++ b/backend/internal/storage/duckdb_strings_test.go @@ -0,0 +1,65 @@ +package storage + +import ( + "bytes" + "strings" + "testing" +) + +// DuckDB stores up to 12 bytes inline and longer values through a pointer. +// Exercise both paths with -race/checkptr enabled: duckdb-go v2.10503.1 could +// read the latter pointer from a misaligned Go stack copy and abort the process. +// Upstream fix: duckdb/duckdb-go@d5b9df8f289bda95d840a8c5fc186541a132fa86. +func TestDuckDBStringAndBlobRoundTrip(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + if _, err := store.DB().Exec(`CREATE TABLE string_boundary (id INTEGER, text_value VARCHAR, blob_value BLOB)`); err != nil { + t.Fatal(err) + } + + lengths := []int{0, 1, 11, 12, 13, 64, 4096} + texts := make([]string, len(lengths)) + blobs := make([][]byte, len(lengths)) + for i, length := range lengths { + texts[i] = strings.Repeat("x", length) + blobs[i] = make([]byte, length) + for j := range blobs[i] { + blobs[i][j] = byte(j * 37) // Include NUL and non-UTF-8 bytes. + } + if _, err := store.DB().Exec(`INSERT INTO string_boundary VALUES (?, ?, ?)`, i, texts[i], blobs[i]); err != nil { + t.Fatal(err) + } + } + + rows, err := store.DB().Query(`SELECT text_value, blob_value FROM string_boundary ORDER BY id`) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + var gotTexts []string + var gotBlobs [][]byte + for rows.Next() { + var text string + var blob []byte + if err := rows.Scan(&text, &blob); err != nil { + t.Fatal(err) + } + gotTexts = append(gotTexts, text) + gotBlobs = append(gotBlobs, blob) + } + if err := rows.Err(); err != nil { + t.Fatal(err) + } + if err := rows.Close(); err != nil { + t.Fatal(err) + } + if len(gotTexts) != len(lengths) { + t.Fatalf("read %d rows, want %d", len(gotTexts), len(lengths)) + } + // Values must also remain valid after the driver's result buffers are freed. + for i, length := range lengths { + if gotTexts[i] != texts[i] || !bytes.Equal(gotBlobs[i], blobs[i]) { + t.Errorf("VARCHAR/BLOB round trip corrupted a %d-byte value", length) + } + } +} diff --git a/docs/import.md b/docs/import.md index c0b82fb..ecef8cd 100644 --- a/docs/import.md +++ b/docs/import.md @@ -29,6 +29,24 @@ ai-observer import claude-code --pricing-mode calculate ai-observer import all --force --dry-run ``` +## Claude Code activity metrics + +Full `import claude-code` reconstructs selected activity metrics from JSONL records. The incremental `watch` command continues to import transcripts and token/cost usage only. Imported activity is historical reconstruction, not a complete copy of Claude Code's live OpenTelemetry counters. + +| Metric | Reconstruction | +|--------|----------------| +| `claude_code.lines_of_code.count` | Successful `Edit`, `Write`, and `MultiEdit` calls matched to their tool results by tool ID. Recorded patches provide additions and deletions without unchanged context. Otherwise, single replacements use a line diff; file overwrites require the original contents. A trailing newline does not add an extra line. | +| `claude_code.commit.count` | Successful Bash results containing Git commit completion headers, paired with parsed `git commit` commands, including `git -C` and `git -c`. Each distinct completion hash in a result counts once. Session commands are parsed, never executed. | +| `claude_code.pull_request.count` | One event per distinct repository and PR number found in structured `pr-link` entries within each file, timestamped at the link. Linking an existing PR is not proof of creation; `reconstruction=pr_link` identifies this proxy. | +| `claude_code.session.count` | One event per non-empty parsed session file, at its first activity. This counts imported files, including separate agent files, rather than CLI launches or resumes. | +| `claude_code.active_time.total` | Recorded positive `turn_duration` values converted from milliseconds to seconds, with `type=cli`, unit `s`, and `reconstruction=turn_duration`. User typing/reading time is unavailable; recorded turn duration is an approximation of CLI active time. | + +Tool activity without a matching successful result is omitted. LOC is also omitted for overwrites without original contents or a patch, `replace_all` without a patch, and unusually large fallback diffs (over four million line comparisons after shared edges are removed). Commit aliases, wrappers, dynamically constructed commands, quiet commits, and unrecognized/localized completion output may be omitted. These omissions favor reporting supported activity over inventing counts from incomplete records. + +Repository attribution uses the working tree's Git remote when available, then matching PR repository candidates, then the directory name. A linked PR in another repository keeps that repository on its own metric and does not override session usage attribution. Codex imports prefer the Git URL recorded in session metadata. On another machine or without Git installed, attribution may fall back to a directory name. + +Imported metrics carry `import_source=local_jsonl`. Avoid summing overlapping imports and live telemetry as if they were independent activity. The normal import state skips unchanged files; forcing an import does not deduplicate against existing metrics. Use the existing `--purge` workflow when replacing previously imported data. + ## Options | Option | Description |