From b127fb8873525228177773a703a739979005857f Mon Sep 17 00:00:00 2001 From: Ilya Isaev Date: Sun, 20 Sep 2026 17:07:28 +0700 Subject: [PATCH] Add github group filter --- .env.example | 4 + README.md | 99 ++++++++-- internal/collector/collector.go | 174 +++++++++++++++++ internal/collector/collector_test.go | 246 ++++++++++++++++++++++++ internal/collector/fakes_test.go | 143 ++++++++++++++ internal/config/config.go | 14 ++ internal/database/database_test.go | 276 +++++++++++++++++++++++++++ internal/database/queries.go | 46 ++++- internal/database/teams.go | 188 ++++++++++++++++++ internal/github/queries.go | 127 ++++++++++++ internal/github/types.go | 25 +++ internal/http/handlers.go | 75 +++++++- internal/http/handlers_test.go | 143 ++++++++++++++ internal/http/render.go | 17 ++ internal/models/models.go | 25 +++ migrations/0004_teams.sql | 41 ++++ web/static/css/app.css | 7 + web/templates/contributors.html | 18 +- web/templates/partials.html | 26 +++ web/templates/repositories.html | 22 ++- web/templates/status.html | 2 + 21 files changed, 1682 insertions(+), 36 deletions(-) create mode 100644 internal/database/teams.go create mode 100644 migrations/0004_teams.sql diff --git a/.env.example b/.env.example index 1c9f218..7b68522 100644 --- a/.env.example +++ b/.env.example @@ -23,6 +23,10 @@ HISTORY_MONTHS=12 # rankings, totals and charts, and their activity is not collected. INCLUDE_ARCHIVED=false +# Import organization teams and their membership. Needs the read:org scope on +# the token; set to false if your token cannot read teams. +SYNC_TEAMS=true + # Logins excluded from contributor rankings, comma separated. Accounts GitHub # itself reports as bots are excluded automatically. EXCLUDED_USERS=dependabot[bot],renovate[bot] diff --git a/README.md b/README.md index 4779dee..f0937ef 100644 --- a/README.md +++ b/README.md @@ -31,14 +31,15 @@ statistics in a server-rendered Bootstrap 5 dashboard. 10. [Incremental synchronization](#10-incremental-synchronization) 11. [Date filters](#11-date-filters) 12. [Contributor ranking](#12-contributor-ranking) -13. [PDF reports](#13-pdf-reports) -14. [Bot exclusions](#14-bot-exclusions) -15. [Archived repository handling](#15-archived-repository-handling) -16. [GitHub GraphQL rate-limit handling](#16-github-graphql-rate-limit-handling) -17. [Security considerations](#17-security-considerations) -18. [Troubleshooting](#18-troubleshooting) -19. [Metric definitions](#19-metric-definitions) -20. [Development](#20-development) +13. [Teams](#13-teams) +14. [PDF reports](#14-pdf-reports) +15. [Bot exclusions](#15-bot-exclusions) +16. [Archived repository handling](#16-archived-repository-handling) +17. [GitHub GraphQL rate-limit handling](#17-github-graphql-rate-limit-handling) +18. [Security considerations](#18-security-considerations) +19. [Troubleshooting](#19-troubleshooting) +20. [Metric definitions](#20-metric-definitions) +21. [Development](#21-development) --- @@ -58,6 +59,8 @@ It provides: - **Repository list and detail pages** — contributors, commits, PRs, reviews, last activity, archived status. Every column header on the list sorts the table; clicking the active column flips the direction. +- **Team filter** — scope the contributors and repositories pages to a GitHub + team. - **Status page** — sync health, GraphQL rate limit, stored row counts, run history. - **Global date filter** — 7 or 30 days, 3/6/12 months or a custom range, applied consistently across every page. @@ -126,6 +129,8 @@ web/static/ CSS and Chart.js wiring | `commits` | Default-branch commits; unique on `(repository_id, github_oid)`. | | `pull_requests` | Unique on `github_id` and on `(repository_id, number)`. | | `pull_request_reviews` | Review submissions; unique on `github_id`. | +| `teams` | Organization teams, unique on `github_id` and on `(organization_id, slug)`. | +| `team_members` / `team_repositories` | Team membership and repository access, replaced wholesale on each sync. | | `contributor_daily_stats` | Daily read model keyed by `(date, organization_id, contributor_id, repository_id)`. | | `sync_runs` | One row per synchronization attempt. | | `schema_migrations` | Applied migration versions. | @@ -181,6 +186,7 @@ responses. | `SYNC_INTERVAL` | no | `30m` | Worker period. Minimum `1m`. | | `HISTORY_MONTHS` | no | `12` | How far back the first sync reaches (1–120). | | `INCLUDE_ARCHIVED` | no | `false` | Include archived repositories in analytics. | +| `SYNC_TEAMS` | no | `true` | Import organization teams and membership. Needs `read:org`. | | `EXCLUDED_USERS` | no | — | Comma-separated logins excluded from rankings. | | `APP_PORT` | no | `8080` | Host port published by Docker Compose. | | `LISTEN_ADDR` | no | `:8080` | Address the server binds inside the container. | @@ -214,7 +220,8 @@ DevPulse only reads. It never requires, requests or performs a write. - `Pull requests` — pull requests and their reviews. - `Metadata` — mandatory, granted automatically. - **Organization permissions** (**Read-only**): - - `Members` — resolve contributor logins and display names. + - `Members` — resolve contributor logins and display names, and read teams and + their membership. Without it, set `SYNC_TEAMS=false`. ### Classic personal access token @@ -322,8 +329,9 @@ and: 2. Pages through its repositories (archived ones are stored too). 3. Loads `HISTORY_MONTHS` (default 12) of default-branch commits, pull requests and reviews per repository. -4. Persists everything in PostgreSQL with upserts. -5. Builds the `contributor_daily_stats` aggregates for the days it touched. +4. Imports the organization's teams, their members and their repositories. +5. Persists everything in PostgreSQL with upserts. +6. Builds the `contributor_daily_stats` aggregates for the days it touched. For a large organization the first run can take a long time and consume a lot of GraphQL budget. Watch progress on `/status` or with `docker compose logs -f app`. @@ -428,7 +436,53 @@ their reviews — all for the selected period. --- -## 13. PDF reports +## 13. Teams + +DevPulse imports the organization's **teams** (what GitHub calls a group): their +members and the repositories each team has access to. Membership is mirrored, +not accumulated — when somebody leaves a team on GitHub they stop being a member +here on the next synchronization, and a deleted team is removed. + +A **team filter** appears on the contributors and repositories pages: + +``` +/contributors?team=platform +/repositories?team=platform&sort=commits&dir=desc +``` + +The two pages scope differently, because they answer different questions: + +| Page | `?team=x` means | +|---|---| +| `/contributors` | Only members of team *x*, counting their activity across the **whole** organization. | +| `/repositories` | Only repositories team *x* has access to, with figures covering **every** contributor, not only team members. | + +Each page states which rule it is applying, so the numbers are never ambiguous. +The filter is carried through sorting, pagination and date changes, and an +unknown or deleted slug quietly falls back to the unfiltered view rather than +breaking a shared link. + +Teams need the **`read:org`** scope. If your token does not have it, team +synchronization fails while commits, pull requests and reviews are still +collected; the run is marked `partial` and the status page explains it. To turn +the feature off entirely: + +```dotenv +SYNC_TEAMS=false +``` + +### Limitations + +- The dashboard and the PDF report are organization-wide; the team filter is not + applied to them, and the report link deliberately does not carry a team. +- Only teams visible to the token are imported. Secret teams the token cannot + read are invisible to DevPulse. +- A repository a team can reach that is outside `GITHUB_ORG` is ignored, like + every other foreign repository. + +--- + +## 14. PDF reports Every page carries a **Report** button in the header. It offers the period the page is currently showing, plus each preset: @@ -488,7 +542,7 @@ Unicode font would fix this at the cost of a larger binary. --- -## 14. Bot exclusions +## 15. Bot exclusions Two mechanisms, both applied at query time: @@ -504,7 +558,7 @@ in the database and can be restored by removing the login from the list. --- -## 15. Archived repository handling +## 16. Archived repository handling Archived repositories are **always stored**, so nothing disappears from history. @@ -522,7 +576,7 @@ Set `INCLUDE_ARCHIVED=true` to fold them back into every statistic. --- -## 16. GitHub GraphQL rate-limit handling +## 17. GitHub GraphQL rate-limit handling GitHub's GraphQL API uses a point budget (5,000 points/hour for most accounts). @@ -551,7 +605,7 @@ Protections: --- -## 17. Security considerations +## 18. Security considerations **The token.** `GITHUB_TOKEN` is read from the environment into memory. It is never stored in PostgreSQL, never rendered in HTML, never included in API @@ -587,7 +641,7 @@ putting an authenticating proxy in front of it. --- -## 18. Troubleshooting +## 19. Troubleshooting **`GITHUB_TOKEN must be set` at startup** Configuration validation failed. The message lists every missing or malformed @@ -624,6 +678,11 @@ commits in particular). Unavailable values count as zero. That is intended. Repository errors are logged with the repository name, the run continues, and its status becomes `partial`. The errors are visible on `/status`. +**Team synchronization fails but everything else works** +The token lacks the `read:org` scope, or the organization restricts team +visibility. The run is marked `partial` and contribution data is unaffected. +Grant the scope, or set `SYNC_TEAMS=false` to stop trying. + **`synchronization postponed until the GitHub rate limit recovers`** The point budget fell below `MIN_RATE_LIMIT_REMAINING`. Wait for `resetAt` (shown on `/status`), lengthen `SYNC_INTERVAL`, or reduce `HISTORY_MONTHS`. @@ -638,7 +697,7 @@ re-synchronizes from scratch. --- -## 19. Metric definitions +## 20. Metric definitions Also available in the UI under *"What each metric means"*. @@ -664,7 +723,7 @@ the pull request **author**; review metrics to the **reviewer**. Days are UTC. --- -## 20. Development +## 21. Development The repository is plain Go with no code generation. @@ -690,6 +749,8 @@ Test coverage includes date-range calculation, organization filtering and isolation, contributor aggregation, ranking and sorting, bot exclusion, archived repository exclusion, duplicate-synchronization prevention, upsert idempotency, configuration validation, GraphQL pagination (including nested review pages), +team import (membership replacement, deleted teams, nested pagination) and team +filtering including cross-organization isolation, rate-limit handling, template escaping of GitHub-provided strings, PDF report generation for every period (including table widths that must fit the page), and an end-to-end check that neither rendering a page nor generating a report ever diff --git a/internal/collector/collector.go b/internal/collector/collector.go index 8d237fb..528a77e 100644 --- a/internal/collector/collector.go +++ b/internal/collector/collector.go @@ -26,6 +26,9 @@ type GitHubAPI interface { FetchPullRequests(ctx context.Context, owner, name string, pageSize int, after string) (*github.Page[github.PullRequest], error) FetchRepositoryReviews(ctx context.Context, owner, name string, pageSize, reviewPageSize int, after string) (*github.Page[github.PullRequestReviews], error) FetchPullRequestReviews(ctx context.Context, owner, name string, number, pageSize int, after string) (*github.ReviewPage, error) + FetchTeams(ctx context.Context, login string, pageSize, memberPageSize, repoPageSize int, after string) (*github.Page[github.Team], error) + FetchTeamMembers(ctx context.Context, login, slug string, pageSize int, after string) (*github.Page[github.Actor], error) + FetchTeamRepositories(ctx context.Context, login, slug string, pageSize int, after string) (*github.Page[github.TeamRepository], error) FetchActors(ctx context.Context, ids []string) ([]github.Actor, error) RateLimit() github.RateLimit CheckBudget() error @@ -45,6 +48,11 @@ type Store interface { PullRequestIDsByNumber(ctx context.Context, repoID int64) (map[int]int64, error) UpsertReviews(ctx context.Context, reviews []models.PullRequestReview) (int, error) ContributorsMissingDetails(ctx context.Context, limit int) ([]string, error) + UpsertTeam(ctx context.Context, t *models.Team) (int64, error) + ReplaceTeamMembers(ctx context.Context, orgID, teamID int64, contributorIDs []int64) error + ReplaceTeamRepositories(ctx context.Context, orgID, teamID int64, repositoryIDs []int64) error + DeleteTeamsNotIn(ctx context.Context, orgID int64, githubIDs []string) (int64, error) + RepositoryIDsByGitHubID(ctx context.Context, orgID int64) (map[string]int64, error) SetRepositoryWatermark(ctx context.Context, repoID int64, w database.Watermark, at time.Time) error RebuildDailyStats(ctx context.Context, orgID int64, repoIDs []int64, from, to time.Time) (int64, error) StartSyncRun(ctx context.Context, orgID int64, syncType string) (int64, error) @@ -59,6 +67,9 @@ const ( reviewPRPageSize = 25 reviewPageSize = 50 actorBatchSize = 90 + teamPageSize = 25 + teamMemberPageSize = 50 + teamRepoPageSize = 50 // overlap re-reads a small window before the watermark so records updated in // the seconds around the previous run are not missed. watermarkOverlap = 15 * time.Minute @@ -253,6 +264,18 @@ func (c *Collector) run(ctx context.Context, org *models.Organization, syncType } } + if c.cfg.SyncTeams { + if err := c.SyncTeams(ctx, org); err != nil { + // Team access needs the read:org scope. Missing it must degrade the + // run to partial, not lose the contribution data already collected. + result.Status = models.SyncStatusPartial + result.Errors = append(result.Errors, fmt.Sprintf("teams: %v", err)) + c.log.Error("team synchronization failed; contribution data is unaffected", + "error", err, + "hint", "the token needs the read:org scope, or set SYNC_TEAMS=false") + } + } + if err := c.SyncContributors(ctx); err != nil { result.Status = models.SyncStatusPartial result.Errors = append(result.Errors, fmt.Sprintf("contributor enrichment: %v", err)) @@ -608,6 +631,157 @@ func (c *Collector) storeReviews(ctx context.Context, org *models.Organization, return c.store.UpsertReviews(ctx, batch) } +// SyncTeams mirrors the organization's teams, their members and the +// repositories they have access to. +// +// Membership is replaced rather than merged: people leave teams, and an +// append-only import would keep them as members forever. +func (c *Collector) SyncTeams(ctx context.Context, org *models.Organization) error { + repoIDs, err := c.store.RepositoryIDsByGitHubID(ctx, org.ID) + if err != nil { + return err + } + + var ( + cursor string + seen []string + members int + links int + ) + for { + page, err := c.api.FetchTeams(ctx, c.cfg.GitHubOrg, teamPageSize, teamMemberPageSize, teamRepoPageSize, cursor) + if err != nil { + return fmt.Errorf("list teams: %w", err) + } + for _, node := range page.Nodes { + teamID, err := c.store.UpsertTeam(ctx, &models.Team{ + OrganizationID: org.ID, + GitHubID: node.ID, + Slug: node.Slug, + Name: node.Name, + Description: node.Description, + URL: node.URL, + Privacy: node.Privacy, + }) + if err != nil { + return err + } + seen = append(seen, node.ID) + + memberIDs, err := c.teamMemberIDs(ctx, node) + if err != nil { + return fmt.Errorf("team %s members: %w", node.Slug, err) + } + if err := c.store.ReplaceTeamMembers(ctx, org.ID, teamID, memberIDs); err != nil { + return err + } + members += len(memberIDs) + + repos, err := c.teamRepositoryIDs(ctx, node, repoIDs) + if err != nil { + return fmt.Errorf("team %s repositories: %w", node.Slug, err) + } + if err := c.store.ReplaceTeamRepositories(ctx, org.ID, teamID, repos); err != nil { + return err + } + links += len(repos) + } + if !page.PageInfo.HasNextPage || page.PageInfo.EndCursor == "" { + break + } + cursor = page.PageInfo.EndCursor + } + + removed, err := c.store.DeleteTeamsNotIn(ctx, org.ID, seen) + if err != nil { + return err + } + + c.log.Info("teams synchronized", + "teams", len(seen), + "memberships", members, + "repository_links", links, + "teams_removed", removed, + ) + return nil +} + +// teamMemberIDs resolves a team's members, following nested pagination when the +// team has more members than fit on the first page. +func (c *Collector) teamMemberIDs(ctx context.Context, team github.Team) ([]int64, error) { + ids := make([]int64, 0, len(team.Members.Nodes)) + add := func(actors []github.Actor) error { + for i := range actors { + // A team member need not have contributed anything yet; storing the + // account keeps the filter complete. + id, err := c.contributorID(ctx, &actors[i]) + if err != nil { + return err + } + if id != nil { + ids = append(ids, *id) + } + } + return nil + } + if err := add(team.Members.Nodes); err != nil { + return nil, err + } + + info := team.Members.PageInfo + for info.HasNextPage && info.EndCursor != "" { + page, err := c.api.FetchTeamMembers(ctx, c.cfg.GitHubOrg, team.Slug, teamMemberPageSize, info.EndCursor) + if err != nil { + return nil, err + } + if err := add(page.Nodes); err != nil { + return nil, err + } + info = page.PageInfo + } + return dedupe(ids), nil +} + +// teamRepositoryIDs resolves a team's repositories against the ones already +// imported. A repository the team can reach but that is not part of this +// organization's import is skipped. +func (c *Collector) teamRepositoryIDs(ctx context.Context, team github.Team, known map[string]int64) ([]int64, error) { + ids := make([]int64, 0, len(team.Repositories.Nodes)) + add := func(repos []github.TeamRepository) { + for _, r := range repos { + if id, ok := known[r.ID]; ok { + ids = append(ids, id) + } + } + } + add(team.Repositories.Nodes) + + info := team.Repositories.PageInfo + for info.HasNextPage && info.EndCursor != "" { + page, err := c.api.FetchTeamRepositories(ctx, c.cfg.GitHubOrg, team.Slug, teamRepoPageSize, info.EndCursor) + if err != nil { + return nil, err + } + add(page.Nodes) + info = page.PageInfo + } + return dedupe(ids), nil +} + +// dedupe removes repeats while preserving order. +func dedupe(ids []int64) []int64 { + seen := make(map[int64]struct{}, len(ids)) + out := ids[:0] + for _, id := range ids { + if _, dup := seen[id]; dup { + continue + } + seen[id] = struct{}{} + out = append(out, id) + } + return out +} + // SyncContributors fills in display names and avatars for accounts first seen // through a connection that did not carry them. func (c *Collector) SyncContributors(ctx context.Context) error { diff --git a/internal/collector/collector_test.go b/internal/collector/collector_test.go index 4cf45c0..6bffabd 100644 --- a/internal/collector/collector_test.go +++ b/internal/collector/collector_test.go @@ -28,6 +28,7 @@ func testConfig(org string) *config.Config { SyncInterval: 30 * time.Minute, HistoryMonths: 12, MinRateLimit: 200, + SyncTeams: true, } } @@ -580,3 +581,248 @@ func TestSplitFullName(t *testing.T) { t.Errorf("fallback failed: got %s/%s", owner, name) } } + +func teamMember(id, login string) github.Actor { + return github.Actor{TypeName: "User", ID: id, Login: login, Name: login} +} + +// withTeams adds two teams to the acme fixture: one small, one large enough to +// force nested pagination of both members and repositories. +func withTeams(f *orgFixture) *orgFixture { + f.teams = []github.Team{ + { + ID: "T_platform", Slug: "platform", Name: "Platform", Description: "Core services", + Members: struct { + PageInfo github.PageInfo `json:"pageInfo"` + Nodes []github.Actor `json:"nodes"` + }{Nodes: []github.Actor{ + teamMember("U_alice", "alice"), + teamMember("U_bob", "bob"), + teamMember("U_carol", "carol"), + }}, + Repositories: struct { + PageInfo github.PageInfo `json:"pageInfo"` + Nodes []github.TeamRepository `json:"nodes"` + }{Nodes: []github.TeamRepository{ + {ID: "R_api", Name: "api", NameWithOwner: "acme/api"}, + {ID: "R_web", Name: "web", NameWithOwner: "acme/web"}, + {ID: "R_unknown", Name: "elsewhere", NameWithOwner: "other/elsewhere"}, + }}, + }, + { + ID: "T_design", Slug: "design", Name: "Design", + Members: struct { + PageInfo github.PageInfo `json:"pageInfo"` + Nodes []github.Actor `json:"nodes"` + }{Nodes: []github.Actor{teamMember("U_carol", "carol")}}, + Repositories: struct { + PageInfo github.PageInfo `json:"pageInfo"` + Nodes []github.TeamRepository `json:"nodes"` + }{Nodes: []github.TeamRepository{{ID: "R_web", Name: "web", NameWithOwner: "acme/web"}}}, + }, + } + return f +} + +func TestSyncTeamsStoresMembershipAndRepositories(t *testing.T) { + api := newFakeAPI(50, withTeams(acmeFixture())) + store := newFakeStore() + c := newTestCollector(t, testConfig("acme"), api, store) + + result, err := c.Sync(context.Background()) + if err != nil { + t.Fatal(err) + } + if result.Status != models.SyncStatusSuccess { + t.Fatalf("status = %s, errors = %v", result.Status, result.Errors) + } + + if len(store.teams) != 2 { + t.Fatalf("stored %d teams, want 2", len(store.teams)) + } + platform := store.teams["T_platform"] + if platform == nil || platform.Slug != "platform" || platform.Name != "Platform" { + t.Fatalf("platform team = %+v", platform) + } + org := store.orgsByGitHubID["O_acme"] + if platform.OrganizationID != org.ID { + t.Errorf("team organization_id = %d, want %d", platform.OrganizationID, org.ID) + } + + if got := len(store.teamMembers[platform.ID]); got != 3 { + t.Errorf("platform has %d members, want 3", got) + } + // A repository the team can reach but that is not part of this + // organization's import must be skipped rather than stored dangling. + if got := len(store.teamRepos[platform.ID]); got != 2 { + t.Errorf("platform has %d repository links, want 2 (the foreign repo is skipped)", got) + } +} + +func TestSyncTeamsFollowsNestedPagination(t *testing.T) { + // A page size of 1 forces the collector through the member and repository + // cursors of every team. + api := newFakeAPI(1, withTeams(acmeFixture())) + store := newFakeStore() + cfg := testConfig("acme") + cfg.IncludeArchived = true + c := newTestCollector(t, cfg, api, store) + + if _, err := c.Sync(context.Background()); err != nil { + t.Fatal(err) + } + if len(store.teams) != 2 { + t.Fatalf("stored %d teams, want 2", len(store.teams)) + } + platform := store.teams["T_platform"] + if got := len(store.teamMembers[platform.ID]); got != 3 { + t.Errorf("nested member pagination collected %d of 3 members", got) + } + if got := len(store.teamRepos[platform.ID]); got != 2 { + t.Errorf("nested repository pagination collected %d of 2 links", got) + } +} + +func TestTeamMembershipIsReplacedNotMerged(t *testing.T) { + fixture := withTeams(acmeFixture()) + api := newFakeAPI(50, fixture) + store := newFakeStore() + c := newTestCollector(t, testConfig("acme"), api, store) + + if _, err := c.Sync(context.Background()); err != nil { + t.Fatal(err) + } + platform := store.teams["T_platform"] + if got := len(store.teamMembers[platform.ID]); got != 3 { + t.Fatalf("first sync stored %d members", got) + } + + // Bob and Carol leave the team on GitHub. + fixture.teams[0].Members.Nodes = []github.Actor{teamMember("U_alice", "alice")} + c.contributors = map[string]int64{} + if _, err := c.Sync(context.Background()); err != nil { + t.Fatal(err) + } + if got := len(store.teamMembers[platform.ID]); got != 1 { + t.Errorf("after members left the team still has %d members, want 1", got) + } +} + +func TestDeletedTeamsAreRemoved(t *testing.T) { + fixture := withTeams(acmeFixture()) + api := newFakeAPI(50, fixture) + store := newFakeStore() + c := newTestCollector(t, testConfig("acme"), api, store) + + if _, err := c.Sync(context.Background()); err != nil { + t.Fatal(err) + } + if len(store.teams) != 2 { + t.Fatalf("stored %d teams", len(store.teams)) + } + + fixture.teams = fixture.teams[:1] + if _, err := c.Sync(context.Background()); err != nil { + t.Fatal(err) + } + if len(store.teams) != 1 { + t.Errorf("a team deleted on GitHub is still stored: %d teams remain", len(store.teams)) + } + if _, ok := store.teams["T_design"]; ok { + t.Error("the design team should have been removed") + } +} + +func TestTeamMembersAreStoredEvenWithoutContributions(t *testing.T) { + fixture := withTeams(acmeFixture()) + // Dave is on the team but has never committed anything. + fixture.teams[0].Members.Nodes = append(fixture.teams[0].Members.Nodes, teamMember("U_dave", "dave")) + api := newFakeAPI(50, fixture) + store := newFakeStore() + c := newTestCollector(t, testConfig("acme"), api, store) + + if _, err := c.Sync(context.Background()); err != nil { + t.Fatal(err) + } + if _, ok := store.contributors["U_dave"]; !ok { + t.Error("a team member with no contributions must still be stored, or the filter is incomplete") + } +} + +func TestTeamFailureDoesNotLoseContributionData(t *testing.T) { + api := newFakeAPI(50, withTeams(acmeFixture())) + // A token without read:org cannot list teams. + api.teamsErr = errors.New("Resource not accessible by integration") + store := newFakeStore() + c := newTestCollector(t, testConfig("acme"), api, store) + + result, err := c.Sync(context.Background()) + if err != nil { + t.Fatalf("a team failure must not fail the whole run: %v", err) + } + if result.Status != models.SyncStatusPartial { + t.Errorf("status = %s, want partial", result.Status) + } + if !strings.Contains(strings.Join(result.Errors, " "), "teams") { + t.Errorf("the team error must be reported, got %v", result.Errors) + } + // Commits, pull requests and reviews were still collected. + commits, prs, reviews, _, _ := store.counts() + if commits == 0 || prs == 0 || reviews == 0 { + t.Errorf("contribution data was lost: %d commits, %d PRs, %d reviews", commits, prs, reviews) + } +} + +func TestTeamsCanBeDisabled(t *testing.T) { + api := newFakeAPI(50, withTeams(acmeFixture())) + store := newFakeStore() + cfg := testConfig("acme") + cfg.SyncTeams = false + c := newTestCollector(t, cfg, api, store) + + result, err := c.Sync(context.Background()) + if err != nil { + t.Fatal(err) + } + if result.Status != models.SyncStatusSuccess { + t.Errorf("status = %s, errors = %v", result.Status, result.Errors) + } + if len(store.teams) != 0 { + t.Errorf("SYNC_TEAMS=false still imported %d teams", len(store.teams)) + } +} + +func TestTeamSyncStaysInsideTheConfiguredOrganization(t *testing.T) { + other := withTeams(otherOrgFixture()) + api := newFakeAPI(50, withTeams(acmeFixture()), other) + store := newFakeStore() + c := newTestCollector(t, testConfig("acme"), api, store) + + if _, err := c.Sync(context.Background()); err != nil { + t.Fatal(err) + } + for _, login := range api.askedOrgs { + if !strings.EqualFold(login, "acme") { + t.Errorf("team sync queried organization %q", login) + } + } + org := store.orgsByGitHubID["O_acme"] + for _, team := range store.teams { + if team.OrganizationID != org.ID { + t.Errorf("team %s belongs to organization %d, want %d", team.Slug, team.OrganizationID, org.ID) + } + } +} + +func TestDedupe(t *testing.T) { + got := dedupe([]int64{3, 1, 3, 2, 1}) + want := []int64{3, 1, 2} + if len(got) != len(want) { + t.Fatalf("dedupe = %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("dedupe = %v, want %v", got, want) + } + } +} diff --git a/internal/collector/fakes_test.go b/internal/collector/fakes_test.go index f5e851b..0e6d85d 100644 --- a/internal/collector/fakes_test.go +++ b/internal/collector/fakes_test.go @@ -21,6 +21,7 @@ import ( type orgFixture struct { org github.Organization repos []repoFixture + teams []github.Team } type repoFixture struct { @@ -47,6 +48,9 @@ type fakeAPI struct { // budgetAfter makes CheckBudget fail once this many calls have been made. budgetAfter int + // teamsErr simulates a token without the read:org scope. + teamsErr error + askedOrgs []string askedRepos []string commitCall []commitCall @@ -200,6 +204,74 @@ func (f *fakeAPI) FetchPullRequestReviews(ctx context.Context, owner, name strin return &github.ReviewPage{PullRequestNumber: number, Reviews: nodes, PageInfo: info}, nil } +func (f *fakeAPI) FetchTeams(ctx context.Context, login string, pageSize, memberPageSize, repoPageSize int, after string) (*github.Page[github.Team], error) { + f.mu.Lock() + defer f.mu.Unlock() + f.count() + f.askedOrgs = append(f.askedOrgs, login) + if f.teamsErr != nil { + return nil, f.teamsErr + } + o, ok := f.orgs[strings.ToLower(login)] + if !ok { + return nil, fmt.Errorf("%w: organization %q", github.ErrNotFound, login) + } + // Serve only the first page of each team's nested connections, so the + // collector has to follow the cursors. + nodes := make([]github.Team, 0, len(o.teams)) + for _, t := range o.teams { + trimmed := t + if len(t.Members.Nodes) > f.pageSize { + trimmed.Members.Nodes = t.Members.Nodes[:f.pageSize] + trimmed.Members.PageInfo = github.PageInfo{HasNextPage: true, EndCursor: cursorAt(f.pageSize)} + } + if len(t.Repositories.Nodes) > f.pageSize { + trimmed.Repositories.Nodes = t.Repositories.Nodes[:f.pageSize] + trimmed.Repositories.PageInfo = github.PageInfo{HasNextPage: true, EndCursor: cursorAt(f.pageSize)} + } + nodes = append(nodes, trimmed) + } + page, info := paginate(nodes, f.pageSize, after) + return &github.Page[github.Team]{Nodes: page, PageInfo: info}, nil +} + +func (f *fakeAPI) teamFixture(login, slug string) (*github.Team, error) { + o, ok := f.orgs[strings.ToLower(login)] + if !ok { + return nil, fmt.Errorf("%w: organization %q", github.ErrNotFound, login) + } + for i := range o.teams { + if o.teams[i].Slug == slug { + return &o.teams[i], nil + } + } + return nil, fmt.Errorf("%w: team %s", github.ErrNotFound, slug) +} + +func (f *fakeAPI) FetchTeamMembers(ctx context.Context, login, slug string, pageSize int, after string) (*github.Page[github.Actor], error) { + f.mu.Lock() + defer f.mu.Unlock() + f.count() + team, err := f.teamFixture(login, slug) + if err != nil { + return nil, err + } + nodes, info := paginate(team.Members.Nodes, f.pageSize, after) + return &github.Page[github.Actor]{Nodes: nodes, PageInfo: info}, nil +} + +func (f *fakeAPI) FetchTeamRepositories(ctx context.Context, login, slug string, pageSize int, after string) (*github.Page[github.TeamRepository], error) { + f.mu.Lock() + defer f.mu.Unlock() + f.count() + team, err := f.teamFixture(login, slug) + if err != nil { + return nil, err + } + nodes, info := paginate(team.Repositories.Nodes, f.pageSize, after) + return &github.Page[github.TeamRepository]{Nodes: nodes, PageInfo: info}, nil +} + func (f *fakeAPI) FetchActors(ctx context.Context, ids []string) ([]github.Actor, error) { f.mu.Lock() defer f.mu.Unlock() @@ -285,6 +357,11 @@ type fakeStore struct { prIDs map[string]int64 // repoID|number reviews map[string]models.PullRequestReview // GitHub node id + teams map[string]*models.Team // by GitHub node id + teamMembers map[int64][]int64 + teamRepos map[int64][]int64 + teamsDeleted int + rebuilds []rebuildCall runs []*models.SyncRun upsertOps int @@ -301,6 +378,9 @@ func newFakeStore() *fakeStore { prs: map[string]models.PullRequest{}, prIDs: map[string]int64{}, reviews: map[string]models.PullRequestReview{}, + teams: map[string]*models.Team{}, + teamMembers: map[int64][]int64{}, + teamRepos: map[int64][]int64{}, } } @@ -495,6 +575,69 @@ func (s *fakeStore) FinishSyncRun(ctx context.Context, run *models.SyncRun) erro return nil } +func (s *fakeStore) UpsertTeam(ctx context.Context, t *models.Team) (int64, error) { + s.mu.Lock() + defer s.mu.Unlock() + s.upsertOps++ + if existing, ok := s.teams[t.GitHubID]; ok { + id := existing.ID + *existing = *t + existing.ID = id + return id, nil + } + cp := *t + cp.ID = s.id() + s.teams[t.GitHubID] = &cp + return cp.ID, nil +} + +func (s *fakeStore) ReplaceTeamMembers(ctx context.Context, orgID, teamID int64, ids []int64) error { + s.mu.Lock() + defer s.mu.Unlock() + s.teamMembers[teamID] = append([]int64(nil), ids...) + return nil +} + +func (s *fakeStore) ReplaceTeamRepositories(ctx context.Context, orgID, teamID int64, ids []int64) error { + s.mu.Lock() + defer s.mu.Unlock() + s.teamRepos[teamID] = append([]int64(nil), ids...) + return nil +} + +func (s *fakeStore) DeleteTeamsNotIn(ctx context.Context, orgID int64, githubIDs []string) (int64, error) { + s.mu.Lock() + defer s.mu.Unlock() + keep := make(map[string]struct{}, len(githubIDs)) + for _, id := range githubIDs { + keep[id] = struct{}{} + } + var removed int64 + for id, team := range s.teams { + if _, ok := keep[id]; ok { + continue + } + delete(s.teams, id) + delete(s.teamMembers, team.ID) + delete(s.teamRepos, team.ID) + removed++ + } + s.teamsDeleted += int(removed) + return removed, nil +} + +func (s *fakeStore) RepositoryIDsByGitHubID(ctx context.Context, orgID int64) (map[string]int64, error) { + s.mu.Lock() + defer s.mu.Unlock() + out := map[string]int64{} + for githubID, r := range s.repos { + if r.OrganizationID == orgID { + out[githubID] = r.ID + } + } + return out, nil +} + // lastRun returns the most recently recorded synchronization run. func (s *fakeStore) lastRun() *models.SyncRun { s.mu.Lock() diff --git a/internal/config/config.go b/internal/config/config.go index feab479..dd387f9 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -25,6 +25,9 @@ type Config struct { HistoryMonths int IncludeArchived bool + // SyncTeams imports organization teams and their membership. It needs the + // read:org scope; turn it off for a token that does not have it. + SyncTeams bool // ExcludedUsers holds lower-cased logins excluded from rankings. ExcludedUsers []string @@ -70,6 +73,7 @@ func FromEnv(get func(string) string) (*Config, error) { SyncOnStartup: true, LogLevel: "info", IncludeArchived: false, + SyncTeams: true, } var errs []string @@ -118,6 +122,15 @@ func FromEnv(get func(string) string) (*Config, error) { } } + if v := strings.TrimSpace(get("SYNC_TEAMS")); v != "" { + b, err := strconv.ParseBool(v) + if err != nil { + errs = append(errs, fmt.Sprintf("SYNC_TEAMS is not a boolean (%q)", v)) + } else { + cfg.SyncTeams = b + } + } + cfg.ExcludedUsers = ParseExcludedUsers(get("EXCLUDED_USERS")) if v := strings.TrimSpace(get("LISTEN_ADDR")); v != "" { @@ -192,6 +205,7 @@ func (c *Config) Redacted() map[string]any { "sync_interval": c.SyncInterval.String(), "history_months": c.HistoryMonths, "include_archived": c.IncludeArchived, + "sync_teams": c.SyncTeams, "excluded_users": c.ExcludedUsers, "listen_addr": c.ListenAddr, "min_rate_limit": c.MinRateLimit, diff --git a/internal/database/database_test.go b/internal/database/database_test.go index 2f45de7..09b2f52 100644 --- a/internal/database/database_test.go +++ b/internal/database/database_test.go @@ -1015,3 +1015,279 @@ func TestRepositoryRankingIgnoresAnInjectedSortKey(t *testing.T) { t.Fatal("the repositories table disappeared: the sort key reached SQL") } } + +func mustTeam(t *testing.T, db *DB, org *models.Organization, githubID, slug, name string) int64 { + t.Helper() + id, err := db.UpsertTeam(context.Background(), &models.Team{ + OrganizationID: org.ID, GitHubID: githubID, Slug: slug, Name: name, + }) + if err != nil { + t.Fatalf("upsert team: %v", err) + } + return id +} + +func TestTeamPersistenceIsIdempotentAndReplacesMembership(t *testing.T) { + db := testDB(t) + ctx := context.Background() + org := mustOrg(t, db, "O_1", "acme") + repoA := mustRepo(t, db, org, "R_a", "api", false) + repoB := mustRepo(t, db, org, "R_b", "web", false) + alice := mustContributor(t, db, "U_alice", "alice", false) + bob := mustContributor(t, db, "U_bob", "bob", false) + + teamID := mustTeam(t, db, org, "T_1", "platform", "Platform") + + // Re-importing the same team must not create a second row. + if again := mustTeam(t, db, org, "T_1", "platform", "Platform Team"); again != teamID { + t.Errorf("re-import created a new team id %d, want %d", again, teamID) + } + if got := countRows(t, db, "teams"); got != 1 { + t.Errorf("teams = %d rows, want 1", got) + } + + for i := 0; i < 3; i++ { + if err := db.ReplaceTeamMembers(ctx, org.ID, teamID, []int64{alice, bob}); err != nil { + t.Fatal(err) + } + if err := db.ReplaceTeamRepositories(ctx, org.ID, teamID, []int64{repoA, repoB}); err != nil { + t.Fatal(err) + } + } + if got := countRows(t, db, "team_members"); got != 2 { + t.Errorf("team_members = %d rows after repeats, want 2", got) + } + if got := countRows(t, db, "team_repositories"); got != 2 { + t.Errorf("team_repositories = %d rows after repeats, want 2", got) + } + + // Bob leaves and repoB is unassigned: the stored set must shrink. + if err := db.ReplaceTeamMembers(ctx, org.ID, teamID, []int64{alice}); err != nil { + t.Fatal(err) + } + if err := db.ReplaceTeamRepositories(ctx, org.ID, teamID, []int64{repoA}); err != nil { + t.Fatal(err) + } + if got := countRows(t, db, "team_members"); got != 1 { + t.Errorf("team_members = %d, want 1: membership must be replaced, not merged", got) + } + if got := countRows(t, db, "team_repositories"); got != 1 { + t.Errorf("team_repositories = %d, want 1", got) + } + + // An empty set clears the team without deleting it. + if err := db.ReplaceTeamMembers(ctx, org.ID, teamID, nil); err != nil { + t.Fatal(err) + } + if got := countRows(t, db, "team_members"); got != 0 { + t.Errorf("team_members = %d, want 0", got) + } + if got := countRows(t, db, "teams"); got != 1 { + t.Error("clearing membership must not delete the team") + } +} + +func TestListAndResolveTeams(t *testing.T) { + db := testDB(t) + ctx := context.Background() + org := mustOrg(t, db, "O_1", "acme") + repo := mustRepo(t, db, org, "R_a", "api", false) + alice := mustContributor(t, db, "U_alice", "alice", false) + + platform := mustTeam(t, db, org, "T_1", "platform", "Platform") + mustTeam(t, db, org, "T_2", "design", "Design") + if err := db.ReplaceTeamMembers(ctx, org.ID, platform, []int64{alice}); err != nil { + t.Fatal(err) + } + if err := db.ReplaceTeamRepositories(ctx, org.ID, platform, []int64{repo}); err != nil { + t.Fatal(err) + } + + teams, err := db.ListTeams(ctx, org.ID) + if err != nil { + t.Fatal(err) + } + if len(teams) != 2 { + t.Fatalf("listed %d teams, want 2", len(teams)) + } + // Ordered by display name: Design before Platform. + if teams[0].Name != "Design" { + t.Errorf("teams are not ordered by name: %s first", teams[0].Name) + } + for _, team := range teams { + if team.Slug == "platform" { + if team.MemberCount != 1 || team.RepositoryCount != 1 { + t.Errorf("platform counts = %d members / %d repos, want 1/1", + team.MemberCount, team.RepositoryCount) + } + } + } + + found, err := db.TeamBySlug(ctx, org.ID, "PLATFORM") + if err != nil || found == nil { + t.Fatalf("TeamBySlug is case sensitive: %v %v", found, err) + } + missing, err := db.TeamBySlug(ctx, org.ID, "nope") + if err != nil { + t.Fatal(err) + } + if missing != nil { + t.Error("an unknown slug must resolve to nil, not an error") + } + + if n, err := db.CountTeams(ctx, org.ID); err != nil || n != 2 { + t.Errorf("CountTeams = %d, %v", n, err) + } +} + +func TestDeleteTeamsNotIn(t *testing.T) { + db := testDB(t) + ctx := context.Background() + org := mustOrg(t, db, "O_1", "acme") + keep := mustTeam(t, db, org, "T_1", "platform", "Platform") + mustTeam(t, db, org, "T_2", "design", "Design") + alice := mustContributor(t, db, "U_alice", "alice", false) + if err := db.ReplaceTeamMembers(ctx, org.ID, keep, []int64{alice}); err != nil { + t.Fatal(err) + } + + removed, err := db.DeleteTeamsNotIn(ctx, org.ID, []string{"T_1"}) + if err != nil { + t.Fatal(err) + } + if removed != 1 { + t.Errorf("removed %d teams, want 1", removed) + } + if got := countRows(t, db, "teams"); got != 1 { + t.Errorf("teams = %d, want 1", got) + } + // The surviving team keeps its membership. + if got := countRows(t, db, "team_members"); got != 1 { + t.Errorf("team_members = %d, want 1", got) + } + + // An organization whose teams all disappeared ends up with none. + if _, err := db.DeleteTeamsNotIn(ctx, org.ID, nil); err != nil { + t.Fatal(err) + } + if got := countRows(t, db, "teams"); got != 0 { + t.Errorf("teams = %d, want 0", got) + } +} + +func TestTeamFilterScopesContributorsAndRepositories(t *testing.T) { + db := testDB(t) + ctx := context.Background() + org := mustOrg(t, db, "O_1", "acme") + teamRepo := mustRepo(t, db, org, "R_a", "api", false) + otherRepo := mustRepo(t, db, org, "R_b", "marketing-site", false) + + alice := mustContributor(t, db, "U_alice", "alice", false) // team member + bob := mustContributor(t, db, "U_bob", "bob", false) // team member + mallory := mustContributor(t, db, "U_m", "mallory", false) // not a member + + // Alice works in both repositories, bob only in the team's repository, and + // mallory only outside it. + mustCommit(t, db, org.ID, teamRepo, &alice, "c1", day(2), 10, 1, 1) + mustCommit(t, db, org.ID, otherRepo, &alice, "c2", day(3), 20, 2, 2) + mustCommit(t, db, org.ID, teamRepo, &bob, "c3", day(2), 5, 1, 1) + mustCommit(t, db, org.ID, otherRepo, &mallory, "c4", day(2), 90, 9, 9) + rebuild(t, db, org.ID, []int64{teamRepo, otherRepo}) + + teamID := mustTeam(t, db, org, "T_1", "platform", "Platform") + if err := db.ReplaceTeamMembers(ctx, org.ID, teamID, []int64{alice, bob}); err != nil { + t.Fatal(err) + } + if err := db.ReplaceTeamRepositories(ctx, org.ID, teamID, []int64{teamRepo}); err != nil { + t.Fatal(err) + } + + f := fullRange() + f.OrganizationID = org.ID + + // Unfiltered: everybody and every repository. + all, err := db.ContributorRanking(ctx, f, "commits", 50, 0) + if err != nil { + t.Fatal(err) + } + if len(all) != 3 { + t.Fatalf("unfiltered ranking has %d rows, want 3", len(all)) + } + + f.TeamID = &teamID + members, err := db.ContributorRanking(ctx, f, "commits", 50, 0) + if err != nil { + t.Fatal(err) + } + if len(members) != 2 { + t.Fatalf("team ranking has %d rows, want the 2 members", len(members)) + } + for _, row := range members { + if row.Login == "mallory" { + t.Error("a non-member appears in the team ranking") + } + } + // Membership scopes the people, not their repositories: alice's work in the + // non-team repository still counts. + for _, row := range members { + if row.Login == "alice" && row.Commits != 2 { + t.Errorf("alice has %d commits under the team filter, want both of them", row.Commits) + } + } + + repos, err := db.RepositoryRanking(ctx, f, metrics.DefaultRepositorySort, metrics.SortDesc) + if err != nil { + t.Fatal(err) + } + if len(repos) != 1 || repos[0].Name != "api" { + t.Fatalf("team repository list = %+v, want only api", repos) + } + // The repository's own figures still cover every contributor. + if repos[0].Commits != 2 { + t.Errorf("api shows %d commits under the team filter, want 2 from all contributors", repos[0].Commits) + } + + // Clearing the filter restores the full view without re-synchronizing. + f.TeamID = nil + repos, err = db.RepositoryRanking(ctx, f, metrics.DefaultRepositorySort, metrics.SortDesc) + if err != nil { + t.Fatal(err) + } + if len(repos) != 2 { + t.Errorf("unfiltered repository list has %d rows, want 2", len(repos)) + } +} + +func TestTeamFilterCannotReachAcrossOrganizations(t *testing.T) { + db := testDB(t) + ctx := context.Background() + a, b := seedTwoOrganizations(t, db) + + // A team in org-a whose id is used against org-b must yield nothing. + teamID := mustTeam(t, db, a, "T_a", "platform", "Platform") + var aliceID int64 + if err := db.Pool.QueryRow(ctx, `SELECT id FROM contributors WHERE login = 'alice'`).Scan(&aliceID); err != nil { + t.Fatal(err) + } + if err := db.ReplaceTeamMembers(ctx, a.ID, teamID, []int64{aliceID}); err != nil { + t.Fatal(err) + } + + f := fullRange() + f.OrganizationID = b.ID + f.TeamID = &teamID + + rows, err := db.ContributorRanking(ctx, f, "commits", 50, 0) + if err != nil { + t.Fatal(err) + } + if len(rows) != 0 { + t.Errorf("a team of another organization returned %d contributors, want none", len(rows)) + } + + // Its own organization still sees it. + f.OrganizationID = a.ID + if rows, err = db.ContributorRanking(ctx, f, "commits", 50, 0); err != nil || len(rows) != 1 { + t.Errorf("org-a team ranking = %d rows, %v", len(rows), err) + } +} diff --git a/internal/database/queries.go b/internal/database/queries.go index b83f6ae..35b5462 100644 --- a/internal/database/queries.go +++ b/internal/database/queries.go @@ -21,6 +21,9 @@ type Filter struct { IncludeArchived bool // ExcludedLogins are lower-cased logins removed from rankings. ExcludedLogins []string + // TeamID scopes the contributor ranking to a team's members and the + // repository list to a team's repositories. Nil means the whole organization. + TeamID *int64 } // args returns the four parameters shared by every analytics statement: @@ -45,6 +48,31 @@ func (f Filter) baseArgs() []any { } } +// contributorArgs adds the optional team to the shared parameters, for the +// statements that scope contributors by team membership: $6 team id. +func (f Filter) contributorArgs() []any { + return append(f.args(), f.TeamID) +} + +// repositoryArgs adds the optional team for statements that scope repositories +// by team access: $6 team id. +func (f Filter) repositoryArgs() []any { + return append(f.args(), f.TeamID) +} + +// teamMemberFilterSQL restricts contributors to the members of one team. A NULL +// parameter disables it, so the same statement serves the unfiltered view. +const teamMemberFilterSQL = ` + AND ($6::bigint IS NULL + OR EXISTS (SELECT 1 FROM team_members tm + WHERE tm.contributor_id = c.id AND tm.team_id = $6))` + +// teamRepositoryFilterSQL restricts repositories to those one team has access to. +const teamRepositoryFilterSQL = ` + AND ($6::bigint IS NULL + OR EXISTS (SELECT 1 FROM team_repositories tr + WHERE tr.repository_id = r.id AND tr.team_id = $6))` + // contributorFilterSQL is the shared WHERE clause applied to aggregate reads. // Bots and explicitly excluded logins never appear in rankings. const contributorFilterSQL = ` @@ -100,7 +128,7 @@ WITH stats AS ( COUNT(DISTINCT s.repository_id) AS repositories FROM contributor_daily_stats s JOIN repositories r ON r.id = s.repository_id - JOIN contributors c ON c.id = s.contributor_id` + contributorFilterSQL + ` + JOIN contributors c ON c.id = s.contributor_id` + contributorFilterSQL + teamMemberFilterSQL + ` %s GROUP BY s.contributor_id ), @@ -149,9 +177,9 @@ func scanContributorStats(rows pgx.Rows) ([]models.ContributorStats, error) { func (db *DB) ContributorRanking(ctx context.Context, f Filter, sortKey string, limit, offset int) ([]models.ContributorStats, error) { metric := metrics.ResolveSort(sortKey) q := fmt.Sprintf(contributorStatsSQL, "") + - fmt.Sprintf("\n ORDER BY %s DESC, commits DESC, lower(c.login) ASC\n LIMIT $6 OFFSET $7", metric.SortExpr) + fmt.Sprintf("\n ORDER BY %s DESC, commits DESC, lower(c.login) ASC\n LIMIT $7 OFFSET $8", metric.SortExpr) - args := append(f.args(), limit, offset) + args := append(f.contributorArgs(), limit, offset) rows, err := db.Pool.Query(ctx, q, args...) if err != nil { return nil, fmt.Errorf("contributor ranking: %w", redact(err)) @@ -315,7 +343,7 @@ SELECT r.id, r.organization_id, r.github_id, r.name, r.full_name, r.url, r.descr FROM repositories r LEFT JOIN stats ON stats.repository_id = r.id WHERE r.organization_id = $1 - AND ($4 OR NOT r.is_archived)` + AND ($4 OR NOT r.is_archived)` + teamRepositoryFilterSQL // RepositoryRanking returns repository rows for the repositories page, ordered // by one column. @@ -330,7 +358,7 @@ func (db *DB) RepositoryRanking(ctx context.Context, f Filter, sortKey, sortDir q := repositoryStatsSQL + fmt.Sprintf( "\n ORDER BY %s %s NULLS LAST, lower(r.name) ASC", col.SortExpr, metrics.SQLDirection(dir)) - rows, err := db.Pool.Query(ctx, q, f.args()...) + rows, err := db.Pool.Query(ctx, q, f.repositoryArgs()...) if err != nil { return nil, fmt.Errorf("repository ranking: %w", redact(err)) } @@ -354,8 +382,8 @@ func (db *DB) RepositoryRanking(ctx context.Context, f Filter, sortKey, sortDir // RepositoryTotals returns period statistics for one repository. func (db *DB) RepositoryTotals(ctx context.Context, f Filter, repositoryID int64) (models.RepositoryStats, error) { - q := repositoryStatsSQL + ` AND r.id = $6` - args := append(f.args(), repositoryID) + q := repositoryStatsSQL + ` AND r.id = $7` + args := append(f.repositoryArgs(), repositoryID) var r models.RepositoryStats var lastActivity *time.Time @@ -417,10 +445,10 @@ SELECT r.id, r.name, r.full_name, r.url, r.is_archived, // RepositoryContributors lists the contributors active in one repository. func (db *DB) RepositoryContributors(ctx context.Context, f Filter, repositoryID int64) ([]models.ContributorStats, error) { - q := fmt.Sprintf(contributorStatsSQL, "AND s.repository_id = $6") + + q := fmt.Sprintf(contributorStatsSQL, "AND s.repository_id = $7") + "\n ORDER BY commits DESC, reviews_submitted DESC, lower(c.login) ASC\n LIMIT 200" - args := append(f.args(), repositoryID) + args := append(f.contributorArgs(), repositoryID) rows, err := db.Pool.Query(ctx, q, args...) if err != nil { return nil, fmt.Errorf("repository contributors: %w", redact(err)) diff --git a/internal/database/teams.go b/internal/database/teams.go new file mode 100644 index 0000000..dd1fd65 --- /dev/null +++ b/internal/database/teams.go @@ -0,0 +1,188 @@ +package database + +import ( + "context" + "errors" + "fmt" + + "github.com/jackc/pgx/v5" + + "github.com/relaxart/dev-pulse/internal/models" +) + +// UpsertTeam stores a team of the configured organization. +func (db *DB) UpsertTeam(ctx context.Context, t *models.Team) (int64, error) { + const q = ` +INSERT INTO teams (organization_id, github_id, slug, name, description, url, privacy) +VALUES ($1,$2,$3,$4,$5,$6,$7) +ON CONFLICT (github_id) DO UPDATE + SET organization_id = EXCLUDED.organization_id, + slug = EXCLUDED.slug, + name = EXCLUDED.name, + description = EXCLUDED.description, + url = EXCLUDED.url, + privacy = EXCLUDED.privacy, + updated_at = now() +RETURNING id` + + var id int64 + err := db.Pool.QueryRow(ctx, q, + t.OrganizationID, t.GitHubID, t.Slug, t.Name, t.Description, t.URL, t.Privacy).Scan(&id) + if err != nil { + return 0, fmt.Errorf("upsert team %s: %w", t.Slug, redact(err)) + } + return id, nil +} + +// ReplaceTeamMembers makes the stored membership match GitHub exactly. +// +// Membership is a set, not an append-only log: people leave teams, so a plain +// upsert would keep former members forever. The swap runs in one transaction so +// a concurrent dashboard query never sees a half-empty team. +func (db *DB) ReplaceTeamMembers(ctx context.Context, orgID, teamID int64, contributorIDs []int64) error { + return db.replaceTeamLinks(ctx, + `DELETE FROM team_members WHERE team_id = $1`, + `INSERT INTO team_members (team_id, contributor_id, organization_id) + SELECT $1, unnest($2::bigint[]), $3 + ON CONFLICT DO NOTHING`, + orgID, teamID, contributorIDs, "members") +} + +// ReplaceTeamRepositories makes the stored repository links match GitHub exactly. +func (db *DB) ReplaceTeamRepositories(ctx context.Context, orgID, teamID int64, repositoryIDs []int64) error { + return db.replaceTeamLinks(ctx, + `DELETE FROM team_repositories WHERE team_id = $1`, + `INSERT INTO team_repositories (team_id, repository_id, organization_id) + SELECT $1, unnest($2::bigint[]), $3 + ON CONFLICT DO NOTHING`, + orgID, teamID, repositoryIDs, "repositories") +} + +func (db *DB) replaceTeamLinks(ctx context.Context, deleteSQL, insertSQL string, orgID, teamID int64, ids []int64, what string) error { + tx, err := db.Pool.Begin(ctx) + if err != nil { + return fmt.Errorf("begin team %s update: %w", what, redact(err)) + } + defer func() { _ = tx.Rollback(ctx) }() + + if _, err := tx.Exec(ctx, deleteSQL, teamID); err != nil { + return fmt.Errorf("clear team %s: %w", what, redact(err)) + } + if len(ids) > 0 { + if _, err := tx.Exec(ctx, insertSQL, teamID, ids, orgID); err != nil { + return fmt.Errorf("store team %s: %w", what, redact(err)) + } + } + if err := tx.Commit(ctx); err != nil { + return fmt.Errorf("commit team %s update: %w", what, redact(err)) + } + return nil +} + +// DeleteTeamsNotIn removes teams that no longer exist on GitHub. Passing an +// empty list deletes every team of the organization, which is what a genuinely +// team-less organization should end up with. +func (db *DB) DeleteTeamsNotIn(ctx context.Context, orgID int64, githubIDs []string) (int64, error) { + if githubIDs == nil { + githubIDs = []string{} + } + tag, err := db.Pool.Exec(ctx, + `DELETE FROM teams WHERE organization_id = $1 AND NOT (github_id = ANY($2))`, orgID, githubIDs) + if err != nil { + return 0, fmt.Errorf("delete removed teams: %w", redact(err)) + } + return tag.RowsAffected(), nil +} + +const teamColumns = ` + t.id, t.organization_id, t.github_id, t.slug, t.name, t.description, t.url, t.privacy, + (SELECT count(*) FROM team_members tm WHERE tm.team_id = t.id), + (SELECT count(*) FROM team_repositories tr WHERE tr.team_id = t.id)` + +func scanTeam(row pgx.Row) (*models.Team, error) { + var t models.Team + err := row.Scan(&t.ID, &t.OrganizationID, &t.GitHubID, &t.Slug, &t.Name, + &t.Description, &t.URL, &t.Privacy, &t.MemberCount, &t.RepositoryCount) + if errors.Is(err, pgx.ErrNoRows) { + return nil, nil + } + if err != nil { + return nil, redact(err) + } + return &t, nil +} + +// ListTeams returns the organization's teams with their member and repository +// counts, ordered for a filter menu. +func (db *DB) ListTeams(ctx context.Context, orgID int64) ([]models.Team, error) { + q := `SELECT` + teamColumns + ` + FROM teams t + WHERE t.organization_id = $1 + ORDER BY lower(COALESCE(NULLIF(t.name, ''), t.slug))` + + rows, err := db.Pool.Query(ctx, q, orgID) + if err != nil { + return nil, fmt.Errorf("list teams: %w", redact(err)) + } + defer rows.Close() + + var out []models.Team + for rows.Next() { + var t models.Team + if err := rows.Scan(&t.ID, &t.OrganizationID, &t.GitHubID, &t.Slug, &t.Name, + &t.Description, &t.URL, &t.Privacy, &t.MemberCount, &t.RepositoryCount); err != nil { + return nil, fmt.Errorf("scan team: %w", err) + } + out = append(out, t) + } + return out, rows.Err() +} + +// TeamBySlug resolves a team slug inside one organization. An unknown slug +// yields a nil team rather than an error, so a stale link degrades to the +// unfiltered view. +func (db *DB) TeamBySlug(ctx context.Context, orgID int64, slug string) (*models.Team, error) { + if slug == "" { + return nil, nil + } + q := `SELECT` + teamColumns + ` + FROM teams t + WHERE t.organization_id = $1 AND lower(t.slug) = lower($2)` + team, err := scanTeam(db.Pool.QueryRow(ctx, q, orgID, slug)) + if err != nil { + return nil, fmt.Errorf("load team %s: %w", slug, err) + } + return team, nil +} + +// CountTeams reports how many teams are stored, for the status page. +func (db *DB) CountTeams(ctx context.Context, orgID int64) (int64, error) { + var n int64 + if err := db.Pool.QueryRow(ctx, + `SELECT count(*) FROM teams WHERE organization_id = $1`, orgID).Scan(&n); err != nil { + return 0, fmt.Errorf("count teams: %w", redact(err)) + } + return n, nil +} + +// RepositoryIDsByGitHubID maps GitHub node ids to internal repository ids, so +// team repository links can be resolved without one query per repository. +func (db *DB) RepositoryIDsByGitHubID(ctx context.Context, orgID int64) (map[string]int64, error) { + rows, err := db.Pool.Query(ctx, + `SELECT github_id, id FROM repositories WHERE organization_id = $1`, orgID) + if err != nil { + return nil, fmt.Errorf("load repository ids: %w", redact(err)) + } + defer rows.Close() + + out := map[string]int64{} + for rows.Next() { + var githubID string + var id int64 + if err := rows.Scan(&githubID, &id); err != nil { + return nil, fmt.Errorf("scan repository id: %w", err) + } + out[githubID] = id + } + return out, rows.Err() +} diff --git a/internal/github/queries.go b/internal/github/queries.go index 554c945..24c8986 100644 --- a/internal/github/queries.go +++ b/internal/github/queries.go @@ -111,6 +111,133 @@ func (c *Client) FetchRepositories(ctx context.Context, login string, pageSize i }, nil } +// teamMemberFields selects a team member. Team members are concrete Users, so +// unlike a pull request author they can select id and name directly. +const teamMemberFields = `__typename id login name avatarUrl url` + +const teamsQuery = ` +query DevPulseTeams($login: String!, $first: Int!, $after: String, $members: Int!, $repos: Int!) { + organization(login: $login) { + teams(first: $first, after: $after, orderBy: {field: NAME, direction: ASC}) { + pageInfo { hasNextPage endCursor } + nodes { + id + slug + name + description + url + privacy + members(first: $members) { + pageInfo { hasNextPage endCursor } + nodes { ` + teamMemberFields + ` } + } + repositories(first: $repos) { + pageInfo { hasNextPage endCursor } + nodes { id name nameWithOwner } + } + } + } + }` + rateLimitFragment + ` +}` + +// FetchTeams returns one page of the configured organization's teams, each with +// its first page of members and repositories. +func (c *Client) FetchTeams(ctx context.Context, login string, pageSize, memberPageSize, repoPageSize int, after string) (*Page[Team], error) { + var out struct { + Organization *struct { + Teams struct { + PageInfo PageInfo `json:"pageInfo"` + Nodes []Team `json:"nodes"` + } `json:"teams"` + } `json:"organization"` + } + vars := map[string]any{ + "login": login, + "first": pageSize, + "after": cursor(after), + "members": memberPageSize, + "repos": repoPageSize, + } + if err := c.do(ctx, "teams", teamsQuery, vars, &out); err != nil { + return nil, err + } + if out.Organization == nil { + return nil, fmt.Errorf("%w: organization %q", ErrNotFound, login) + } + t := out.Organization.Teams + return &Page[Team]{Nodes: t.Nodes, PageInfo: t.PageInfo}, nil +} + +const teamMembersQuery = ` +query DevPulseTeamMembers($login: String!, $slug: String!, $first: Int!, $after: String) { + organization(login: $login) { + team(slug: $slug) { + members(first: $first, after: $after) { + pageInfo { hasNextPage endCursor } + nodes { ` + teamMemberFields + ` } + } + } + }` + rateLimitFragment + ` +}` + +// FetchTeamMembers pages the members of a single team. +func (c *Client) FetchTeamMembers(ctx context.Context, login, slug string, pageSize int, after string) (*Page[Actor], error) { + var out struct { + Organization *struct { + Team *struct { + Members struct { + PageInfo PageInfo `json:"pageInfo"` + Nodes []Actor `json:"nodes"` + } `json:"members"` + } `json:"team"` + } `json:"organization"` + } + vars := map[string]any{"login": login, "slug": slug, "first": pageSize, "after": cursor(after)} + if err := c.do(ctx, "teamMembers", teamMembersQuery, vars, &out); err != nil { + return nil, err + } + if out.Organization == nil || out.Organization.Team == nil { + return nil, fmt.Errorf("%w: team %s/%s", ErrNotFound, login, slug) + } + m := out.Organization.Team.Members + return &Page[Actor]{Nodes: m.Nodes, PageInfo: m.PageInfo}, nil +} + +const teamRepositoriesQuery = ` +query DevPulseTeamRepositories($login: String!, $slug: String!, $first: Int!, $after: String) { + organization(login: $login) { + team(slug: $slug) { + repositories(first: $first, after: $after) { + pageInfo { hasNextPage endCursor } + nodes { id name nameWithOwner } + } + } + }` + rateLimitFragment + ` +}` + +// FetchTeamRepositories pages the repositories a single team has access to. +func (c *Client) FetchTeamRepositories(ctx context.Context, login, slug string, pageSize int, after string) (*Page[TeamRepository], error) { + var out struct { + Organization *struct { + Team *struct { + Repositories struct { + PageInfo PageInfo `json:"pageInfo"` + Nodes []TeamRepository `json:"nodes"` + } `json:"repositories"` + } `json:"team"` + } `json:"organization"` + } + vars := map[string]any{"login": login, "slug": slug, "first": pageSize, "after": cursor(after)} + if err := c.do(ctx, "teamRepositories", teamRepositoriesQuery, vars, &out); err != nil { + return nil, err + } + if out.Organization == nil || out.Organization.Team == nil { + return nil, fmt.Errorf("%w: team %s/%s", ErrNotFound, login, slug) + } + r := out.Organization.Team.Repositories + return &Page[TeamRepository]{Nodes: r.Nodes, PageInfo: r.PageInfo}, nil +} + const commitsQuery = ` query DevPulseCommits($owner: String!, $name: String!, $since: GitTimestamp!, $until: GitTimestamp, $first: Int!, $after: String) { repository(owner: $owner, name: $name) { diff --git a/internal/github/types.go b/internal/github/types.go index ae0aaa7..0688d6a 100644 --- a/internal/github/types.go +++ b/internal/github/types.go @@ -68,6 +68,31 @@ type Repository struct { } `json:"defaultBranchRef"` } +// Team is an organization team with its first page of members and repositories. +type Team struct { + ID string `json:"id"` + Slug string `json:"slug"` + Name string `json:"name"` + Description string `json:"description"` + URL string `json:"url"` + Privacy string `json:"privacy"` + Members struct { + PageInfo PageInfo `json:"pageInfo"` + Nodes []Actor `json:"nodes"` + } `json:"members"` + Repositories struct { + PageInfo PageInfo `json:"pageInfo"` + Nodes []TeamRepository `json:"nodes"` + } `json:"repositories"` +} + +// TeamRepository identifies a repository a team has access to. +type TeamRepository struct { + ID string `json:"id"` + Name string `json:"name"` + NameWithOwner string `json:"nameWithOwner"` +} + // Commit is a commit on the default branch. type Commit struct { OID string `json:"oid"` diff --git a/internal/http/handlers.go b/internal/http/handlers.go index 766cdaa..14a4583 100644 --- a/internal/http/handlers.go +++ b/internal/http/handlers.go @@ -8,6 +8,7 @@ import ( "net/http" "net/url" "strconv" + "strings" "time" "github.com/relaxart/dev-pulse/internal/database" @@ -43,6 +44,40 @@ type pageContext struct { Layout layoutData Org *models.Organization Filter database.Filter + // Team is the resolved ?team= selection, nil when unfiltered. + Team *models.Team + // Teams is the organization's team list, for the filter menu. + Teams []models.Team +} + +// resolveTeam loads the team list and the current selection. A team filter is +// only offered on the pages that support it, so callers opt in. +func (s *Server) resolveTeam(r *http.Request, pc *pageContext) error { + if pc.Org == nil { + return nil + } + teams, err := s.db.ListTeams(r.Context(), pc.Org.ID) + if err != nil { + return err + } + pc.Teams = teams + + slug := strings.TrimSpace(r.URL.Query().Get("team")) + if slug == "" { + return nil + } + // An unknown slug degrades to the unfiltered view rather than erroring: a + // team can be renamed or deleted after a link was shared. + team, err := s.db.TeamBySlug(r.Context(), pc.Org.ID, slug) + if err != nil { + return err + } + if team == nil { + return nil + } + pc.Team = team + pc.Filter.TeamID = &team.ID + return nil } func (s *Server) context(w http.ResponseWriter, r *http.Request, nav string) (*pageContext, bool) { @@ -225,6 +260,8 @@ type contributorsView struct { NextPage int HasNext bool IncludeArchived bool + Teams []models.Team + Team *models.Team } func (s *Server) handleContributors(w http.ResponseWriter, r *http.Request) { @@ -246,9 +283,19 @@ func (s *Server) handleContributors(w http.ResponseWriter, r *http.Request) { view.PrevPage = view.Page - 1 view.NextPage = view.Page + 1 view.Title = "Contributors" - // Keep the sort selection in every link on the page. + + if err := s.resolveTeam(r, pc); err != nil { + s.serverError(w, r, err) + return + } + view.Teams, view.Team = pc.Teams, pc.Team + + // Keep the sort and team selections in every link on the page. view.Query = cloneValues(pc.Layout.Range.Query()) view.Query.Set("sort", view.Sort.Key) + if pc.Team != nil { + view.Query.Set("team", pc.Team.Slug) + } if pc.Org == nil { s.render(w, r, http.StatusOK, "contributors.html", view) @@ -356,6 +403,8 @@ type repositoriesView struct { SortDir string SortLabel string IncludeArchived bool + Teams []models.Team + Team *models.Team } func (s *Server) handleRepositories(w http.ResponseWriter, r *http.Request) { @@ -372,11 +421,22 @@ func (s *Server) handleRepositories(w http.ResponseWriter, r *http.Request) { SortKey: col.Key, SortDir: dir, SortLabel: col.Label, - Headers: repositoryHeaders(pc.Layout.Range.Query(), col.Key, dir), } view.Title = "Repositories" - // Keep the chosen column in the period picker and the navigation links. - view.Query = cloneValues(pc.Layout.Range.Query()) + + if err := s.resolveTeam(r, pc); err != nil { + s.serverError(w, r, err) + return + } + view.Teams, view.Team = pc.Teams, pc.Team + + // Keep the chosen column and team in the period picker and the navigation. + base := cloneValues(pc.Layout.Range.Query()) + if pc.Team != nil { + base.Set("team", pc.Team.Slug) + } + view.Headers = repositoryHeaders(base, col.Key, dir) + view.Query = cloneValues(base) view.Query.Set("sort", col.Key) view.Query.Set("dir", dir) @@ -503,6 +563,8 @@ func (s *Server) handleRepository(w http.ResponseWriter, r *http.Request) { type statusView struct { layoutData Counts database.Counts + TeamCount int64 + SyncTeams bool LastRun *models.SyncRun LastSuccess *models.SyncRun Runs []models.SyncRun @@ -537,6 +599,7 @@ func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) { view := statusView{ layoutData: pc.Layout, SyncInterval: s.cfg.SyncInterval.String(), + SyncTeams: s.cfg.SyncTeams, HistoryMonths: s.cfg.HistoryMonths, IncludeArchived: s.cfg.IncludeArchived, ExcludedUsers: s.cfg.ExcludedUsers, @@ -580,6 +643,10 @@ func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) { s.serverError(w, r, err) return } + if view.TeamCount, err = s.db.CountTeams(ctx, pc.Org.ID); err != nil { + s.serverError(w, r, err) + return + } } s.render(w, r, http.StatusOK, "status.html", view) diff --git a/internal/http/handlers_test.go b/internal/http/handlers_test.go index b6a6063..f892884 100644 --- a/internal/http/handlers_test.go +++ b/internal/http/handlers_test.go @@ -34,6 +34,7 @@ type testServer struct { org *models.Organization repoID int64 alice int64 + teamID int64 } func newTestServer(t *testing.T) *testServer { @@ -179,6 +180,25 @@ func (ts *testServer) seed(t *testing.T) { }}); err != nil { t.Fatal(err) } + teamID, err := ts.db.UpsertTeam(ctx, &models.Team{ + OrganizationID: org.ID, GitHubID: "T_1", Slug: "platform", Name: "Platform", + }) + if err != nil { + t.Fatal(err) + } + ts.teamID = teamID + if err := ts.db.ReplaceTeamMembers(ctx, org.ID, teamID, []int64{alice}); err != nil { + t.Fatal(err) + } + if err := ts.db.ReplaceTeamRepositories(ctx, org.ID, teamID, []int64{repoID}); err != nil { + t.Fatal(err) + } + if _, err := ts.db.UpsertTeam(ctx, &models.Team{ + OrganizationID: org.ID, GitHubID: "T_2", Slug: "design", Name: "Design", + }); err != nil { + t.Fatal(err) + } + if _, err := ts.db.RebuildDailyStats(ctx, org.ID, []int64{repoID}, time.Now().AddDate(-2, 0, 0), time.Now()); err != nil { t.Fatal(err) @@ -664,3 +684,126 @@ func TestOnlyTheRepositoriesTableIsSortable(t *testing.T) { t.Error("the dashboard should not advertise sortable headers") } } + +func TestTeamFilterIsOfferedOnBothPages(t *testing.T) { + ts := newTestServer(t) + for _, path := range []string{"/contributors", "/repositories"} { + body := ts.get(t, path+"?period=12m").Body.String() + if !strings.Contains(body, "All teams") { + t.Errorf("%s does not offer the team filter", path) + } + for _, slug := range []string{"platform", "design"} { + if !strings.Contains(body, "team="+slug) { + t.Errorf("%s does not list the %q team", path, slug) + } + } + } + // The dashboard is deliberately organization-wide. + if strings.Contains(ts.get(t, "/?period=12m").Body.String(), "All teams") { + t.Error("the dashboard should not advertise a team filter it does not apply") + } +} + +func TestTeamFilterScopesEachPage(t *testing.T) { + ts := newTestServer(t) + + // platform owns the api repository and alice; design owns neither. + platform := ts.get(t, "/repositories?period=12m&team=platform").Body.String() + if !strings.Contains(platform, ">api<") { + t.Error("the platform team should list its api repository") + } + if strings.Contains(platform, ">legacy<") { + t.Error("a repository outside the team is still listed") + } + + design := ts.get(t, "/repositories?period=12m&team=design").Body.String() + if strings.Contains(design, ">api<") { + t.Error("the design team has no repositories but api is listed") + } + if !strings.Contains(design, "access to no imported repositories") { + t.Error("an empty team should explain itself rather than look broken") + } + + contributors := ts.get(t, "/contributors?period=12m&team=design").Body.String() + if strings.Contains(contributors, "@alice") { + t.Error("alice is not a member of design but appears in its ranking") + } + if !strings.Contains(contributors, "No members of this team were active") { + t.Error("an empty team ranking should explain itself") + } + if !strings.Contains(ts.get(t, "/contributors?period=12m&team=platform").Body.String(), "@alice") { + t.Error("alice is a member of platform but is missing from its ranking") + } + + if *ts.githubHits != 0 { + t.Fatal("filtering by team must not call GitHub") + } +} + +func TestTeamFilterSurvivesSortingAndPeriodChanges(t *testing.T) { + ts := newTestServer(t) + + body := ts.get(t, "/contributors?period=3m&team=platform&sort=approvals").Body.String() + if !strings.Contains(body, ``) { + t.Error("the period picker drops the team filter") + } + // Sort links keep the team. + if !strings.Contains(body, "team=platform") || !strings.Contains(body, "sort=commits") { + t.Error("the sort menu drops the team filter") + } + + repos := ts.get(t, "/repositories?period=3m&team=platform&sort=reviews&dir=asc").Body.String() + if !strings.Contains(repos, ``) { + t.Error("the repositories period picker drops the team filter") + } + for _, q := range headerHrefs(t, repos) { + if q.Get("team") != "platform" { + t.Errorf("the %q header link drops the team filter: %v", q.Get("sort"), q) + } + } +} + +func TestUnknownTeamDegradesToTheFullView(t *testing.T) { + ts := newTestServer(t) + // A renamed or deleted team should not 404 a shared link. + for _, path := range []string{"/contributors", "/repositories"} { + w := ts.get(t, path+"?period=12m&team=does-not-exist") + if w.Code != http.StatusOK { + t.Errorf("%s with an unknown team = %d, want 200", path, w.Code) + } + if !strings.Contains(w.Body.String(), "All teams") { + t.Errorf("%s with an unknown team did not fall back to the full view", path) + } + } + if !strings.Contains(ts.get(t, "/repositories?period=12m&team=does-not-exist").Body.String(), ">legacy<") { + t.Error("the unfiltered repository list should be shown for an unknown team") + } +} + +func TestReportStaysOrganizationWide(t *testing.T) { + ts := newTestServer(t) + // The report is documented as organization-wide, so its link must not carry + // a team the report would then ignore. + body := ts.get(t, "/contributors?period=3m&team=platform").Body.String() + idx := strings.Index(body, "/report.pdf?") + if idx < 0 { + t.Fatal("no report link on the page") + } + link := body[idx : idx+strings.Index(body[idx:], `"`)] + if strings.Contains(link, "team=") { + t.Errorf("the report link carries a team filter it does not apply: %s", link) + } + + w := ts.get(t, "/report.pdf?period=12m&team=platform") + if w.Code != http.StatusOK || !bytes.HasPrefix(w.Body.Bytes(), []byte("%PDF-")) { + t.Errorf("/report.pdf with a team parameter = %d", w.Code) + } +} + +func TestStatusPageReportsTeams(t *testing.T) { + ts := newTestServer(t) + body := ts.get(t, "/status").Body.String() + if !strings.Contains(body, "Teams") { + t.Error("the status page does not report the team count") + } +} diff --git a/internal/http/render.go b/internal/http/render.go index 0a5ef60..ebd4fbd 100644 --- a/internal/http/render.go +++ b/internal/http/render.go @@ -79,6 +79,7 @@ func templateFuncs() template.FuncMap { "sortLink": sortLink, "pageLink": pageLink, "reportLink": reportLink, + "teamLink": teamLink, "dict": dict, "initials": initials, "stateBadge": stateBadge, @@ -333,6 +334,22 @@ func reportLink(q url.Values, periodKey string) template.URL { return template.URL("/report.pdf?" + v.Encode()) } +// teamLink builds a link that selects a team, or clears the selection when slug +// is empty. Changing the filter returns to the first page. +func teamLink(q url.Values, slug string) template.URL { + v := cloneValues(q) + v.Del("page") + if slug == "" { + v.Del("team") + } else { + v.Set("team", slug) + } + if len(v) == 0 { + return template.URL("") + } + return template.URL("?" + v.Encode()) +} + // dict builds a map inside a template, for partial invocation. func dict(values ...any) (map[string]any, error) { if len(values)%2 != 0 { diff --git a/internal/models/models.go b/internal/models/models.go index 0636707..588643d 100644 --- a/internal/models/models.go +++ b/internal/models/models.go @@ -48,6 +48,31 @@ type Contributor struct { IsBot bool } +// Team is a GitHub organization team. GitHub calls these "teams"; they are what +// most people mean by a group. +type Team struct { + ID int64 + OrganizationID int64 + GitHubID string + Slug string + Name string + Description string + URL string + Privacy string + + // Counts are filled in by listing queries, not by the collector. + MemberCount int64 + RepositoryCount int64 +} + +// DisplayName prefers the human name and falls back to the slug. +func (t Team) DisplayName() string { + if t.Name != "" { + return t.Name + } + return t.Slug +} + // Commit is one commit on a repository's default branch. type Commit struct { OrganizationID int64 diff --git a/migrations/0004_teams.sql b/migrations/0004_teams.sql new file mode 100644 index 0000000..32be657 --- /dev/null +++ b/migrations/0004_teams.sql @@ -0,0 +1,41 @@ +-- GitHub organization teams ("groups") and their membership. +-- +-- Teams belong to exactly one organization, like every other imported entity, +-- and the join tables carry organization_id too so a team filter can never +-- reach across organizations. + +CREATE TABLE teams ( + id BIGSERIAL PRIMARY KEY, + organization_id BIGINT NOT NULL REFERENCES organizations (id) ON DELETE CASCADE, + github_id TEXT NOT NULL UNIQUE, + slug TEXT NOT NULL, + name TEXT NOT NULL DEFAULT '', + description TEXT NOT NULL DEFAULT '', + url TEXT NOT NULL DEFAULT '', + privacy TEXT NOT NULL DEFAULT '', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE (organization_id, slug) +); + +CREATE INDEX teams_organization_idx ON teams (organization_id); + +CREATE TABLE team_members ( + team_id BIGINT NOT NULL REFERENCES teams (id) ON DELETE CASCADE, + contributor_id BIGINT NOT NULL REFERENCES contributors (id) ON DELETE CASCADE, + organization_id BIGINT NOT NULL REFERENCES organizations (id) ON DELETE CASCADE, + PRIMARY KEY (team_id, contributor_id) +); + +CREATE INDEX team_members_contributor_idx ON team_members (contributor_id); +CREATE INDEX team_members_organization_idx ON team_members (organization_id); + +CREATE TABLE team_repositories ( + team_id BIGINT NOT NULL REFERENCES teams (id) ON DELETE CASCADE, + repository_id BIGINT NOT NULL REFERENCES repositories (id) ON DELETE CASCADE, + organization_id BIGINT NOT NULL REFERENCES organizations (id) ON DELETE CASCADE, + PRIMARY KEY (team_id, repository_id) +); + +CREATE INDEX team_repositories_repository_idx ON team_repositories (repository_id); +CREATE INDEX team_repositories_organization_idx ON team_repositories (organization_id); diff --git a/web/static/css/app.css b/web/static/css/app.css index 5e7f170..8ee1ce4 100644 --- a/web/static/css/app.css +++ b/web/static/css/app.css @@ -322,3 +322,10 @@ main { flex: 1 0 auto; } :root[data-bs-theme="dark"] .table > thead th { border-bottom-color: var(--md-outline); } :root[data-bs-theme="dark"] .row-archived { opacity: .55; } :root[data-bs-theme="dark"] code { color: #f7a8b8; } + +/* ---------- Team filter ---------- */ + +.team-menu { max-height: 70vh; overflow-y: auto; min-width: 260px; } +.team-menu .dropdown-item { white-space: normal; } +.team-menu-meta { display: block; font-size: .72rem; color: var(--md-text-muted); } +.team-menu .dropdown-item.active .team-menu-meta { color: rgba(255, 255, 255, .8); } diff --git a/web/templates/contributors.html b/web/templates/contributors.html index 26e9e89..db882d0 100644 --- a/web/templates/contributors.html +++ b/web/templates/contributors.html @@ -5,9 +5,14 @@

Top contributors

-

{{.Org}} · {{.Range.Label}} · sorted by {{.Sort.Label}}

+

+ {{.Org}}{{if .Team}} · team {{.Team.DisplayName}}{{end}} · + {{.Range.Label}} · sorted by {{.Sort.Label}} +

+
+ {{template "teamFilter" .}} +

Bots and logins listed in EXCLUDED_USERS are left out of this ranking. {{if not .IncludeArchived}}Archived repositories are excluded too.{{end}} + {{if .Team}} + Showing the {{.Team.DisplayName}} team: its {{num .Team.MemberCount}} members, + counting their activity across the whole organization. + {{end}}

@@ -76,7 +86,11 @@

Top contributors

{{else}} - {{template "emptyState" "No contributor activity in this period."}} + {{if .Team}} + {{template "emptyState" "No members of this team were active in this period."}} + {{else}} + {{template "emptyState" "No contributor activity in this period."}} + {{end}} {{end}}
diff --git a/web/templates/partials.html b/web/templates/partials.html index 17f56e3..a001335 100644 --- a/web/templates/partials.html +++ b/web/templates/partials.html @@ -81,6 +81,32 @@ {{end}} +{{define "teamFilter"}} +{{if .Teams}} + +{{end}} +{{end}} + {{define "statCard"}}
diff --git a/web/templates/repositories.html b/web/templates/repositories.html index 33d866f..61bbb75 100644 --- a/web/templates/repositories.html +++ b/web/templates/repositories.html @@ -5,10 +5,13 @@

Repositories

- {{.Org}} · {{.Range.Label}} · sorted by {{.SortLabel}} + {{.Org}}{{if .Team}} · team {{.Team.DisplayName}}{{end}} · + {{.Range.Label}} · sorted by {{.SortLabel}} ({{if eq .SortDir "asc"}}ascending{{else}}descending{{end}})

+
+ {{template "teamFilter" .}}
{{if .IncludeArchived}} inventory_2 @@ -18,8 +21,17 @@

Repositories

INCLUDE_ARCHIVED=false — archived repositories are stored and listed, but excluded from rankings and totals. {{end}}
+
+{{if .Team}} +

+ Showing the {{num .Team.RepositoryCount}} repositories the + {{.Team.DisplayName}} team has access to. Their figures still + cover every contributor, not only team members. +

+{{end}} +
{{if .Rows}} @@ -57,7 +69,13 @@

Repositories

{{end}} - {{else}}{{template "emptyState" "No repositories have been synchronized yet."}}{{end}} + {{else}} + {{if .Team}} + {{template "emptyState" "This team has access to no imported repositories."}} + {{else}} + {{template "emptyState" "No repositories have been synchronized yet."}} + {{end}} + {{end}}
{{end}} diff --git a/web/templates/status.html b/web/templates/status.html index 469337a..849946f 100644 --- a/web/templates/status.html +++ b/web/templates/status.html @@ -80,6 +80,8 @@

Synchronization & system status

Repositories
{{num .Counts.Repositories}} ({{num .Counts.ArchivedRepositories}} archived)
Contributors
{{num .Counts.Contributors}}
+
Teams
+
{{if .SyncTeams}}{{num .TeamCount}}{{else}}not synchronized (SYNC_TEAMS=false){{end}}
Commits
{{num .Counts.Commits}}
Pull requests
{{num .Counts.PullRequests}}
Reviews
{{num .Counts.Reviews}}