diff --git a/README.md b/README.md index d321742..b14098a 100644 --- a/README.md +++ b/README.md @@ -45,17 +45,19 @@ One binary, no config to write, no browser. - Creates, edits and deletes namespaces. - Shows the versions of a job, reverts to any of them, tags them, and marks them stable or unstable. - Explains why a job is not placed, has the scheduler evaluate it again, and - places its failed allocations again without waiting for their delay. Opens - the allocations an evaluation placed or changed. + places its failed allocations again without waiting for their delay. Shows + what an evaluation planned for each group, and opens the allocations it + placed or changed. - Lists the launches of periodic and parameterized jobs, shows when a periodic job runs next, runs it now, and dispatches a parameterized job with its meta and payload. - Starts, stops, scales and restarts jobs and allocations, restarts every task of an allocation with its prestart and sidecar tasks, restarts and signals single tasks, and opens a shell in a task. -- Drains clients and shows their CPU and memory over time. Purges clients - that are down, stopped jobs, and collects the garbage of a client or of - the whole cluster. +- Drains clients, shows their CPU and memory over time, marks the ones that + are full, shows when the identity of a client expires and renews it. + Purges clients that are down, stopped jobs, and collects the garbage of a + client or of the whole cluster. - Shows the health of the servers and how many the cluster can lose. - Shows how the scheduler places work, and edits it with check-and-set. - Lists the scaling policies and shows what the autoscaler did with the diff --git a/docs/clients.md b/docs/clients.md index 712974b..93f1450 100644 --- a/docs/clients.md +++ b/docs/clients.md @@ -11,6 +11,13 @@ variables and node pools. For the keys of each screen, see status, eligibility, drain state, CPU and memory use, and address of each client. +A client that holds as many allocations as its limit allows (`node_max_allocs`, +Nomad 1.10.2 and later) is full: the scheduler places nothing more on it, and +its row has the attention color, like a client that is draining or +ineligible. With the event stream on, the list shows a client that fills up +or frees a place within 30 seconds: Nomad streams a change of the client, +not of the allocations placed on it. + - `ctrl-d` drains a client, or stops draining it, after you confirm. - `i` marks a client as eligible or ineligible for new work. - `ctrl-p` purges a client that is down, after you confirm: the cluster @@ -28,6 +35,11 @@ With several clients marked, these keys act on all of them that fit: - its status, address, datacenter, node pool, version and whether it takes new work; +- when the client has a limit, how many allocations it holds of it, like + `3 of 10`. Allocations that ended or are being stopped do not count; +- when the client has an identity, when it expires, like `Identity expires in + 23h` (Nomad 1.11 and later). On an older cluster, or an older client, none + of it shows; - charts of its CPU and memory use, read every 5 seconds while the screen is open. The charts keep the last 20 minutes. `u` reads them in percents or in numbers, scaled up to gigahertz and gibibytes. @@ -42,7 +54,10 @@ From the client screen: - `ctrl-h` lists its host volumes; - `a` shows its attributes; - `m` shows its metadata. `e` opens the metadata in your editor, and urga - saves it to the client when you close the editor. + saves it to the client when you close the editor; +- `i` shows the claims of its identity; +- `ctrl-n` renews its identity after you confirm, and the client asks for a + new one at its next heartbeat. ## Servers diff --git a/docs/jobs.md b/docs/jobs.md index d3a8ef4..5d2950a 100644 --- a/docs/jobs.md +++ b/docs/jobs.md @@ -154,7 +154,10 @@ The plan shows, from top to bottom: 2. **Warnings** from the cluster. 3. **What will happen when you submit this job**: for each task group, how many allocations will be created, stopped, migrated, updated in place, - recreated or deployed as canaries, and how many stay unchanged. + recreated or deployed as canaries, rescheduled now or after a delay, + disconnected or reconnected, and how many stay unchanged. It also says how + many allocations of other jobs the plan preempts. Rescheduled, disconnected + and reconnected allocations need Nomad 1.11. 4. **Changes**: the difference between the running job and your file, laid out like the job file. Removed lines are red and start with `-`, added lines are green and start with `+`. @@ -220,7 +223,9 @@ nothing of a system job. `:evaluations` lists the evaluations of the namespace. `enter` opens one: its status, what triggered it, related evaluations, and the placement failures, if -there are any. `a`, on the list or on the evaluation, opens the allocations it +there are any. It also shows what the evaluation planned for each group, in +the words of the plan, and the allocations of other jobs it preempted (Nomad +1.11). `a`, on the list or on the evaluation, opens the allocations it placed or changed. ## Deployments diff --git a/docs/keys.md b/docs/keys.md index 9b8cfcc..06ac21c 100644 --- a/docs/keys.md +++ b/docs/keys.md @@ -473,6 +473,8 @@ The client screen lists the allocations of the client. It has the keys of | `a` | Attributes | | | `m` | Metadata | | | `u` | Units: percents or numbers on the charts | | +| `i` | Claims of the identity of the client. Shown when it has one. | | +| `ctrl-n` | Renew the identity of the client. Shown when it has one. | yes | On drivers, `enter` opens the details of a driver. On attributes, metadata, driver details and a server, `c` copies the value under the cursor. On diff --git a/internal/nomad/alloc.go b/internal/nomad/alloc.go index 9c83be0..9a9944b 100644 --- a/internal/nomad/alloc.go +++ b/internal/nomad/alloc.go @@ -125,6 +125,24 @@ func allocsOf(stubs []*api.AllocationListStub) []Alloc { return out } +// HoldsPlace says the allocation counts against the limit of its client: the +// scheduler counts what is meant to run and has not ended. +func (a Alloc) HoldsPlace() bool { return holdsPlace(a.DesiredStatus, a.Status) } + +func holdsPlace(desired, status string) bool { + switch desired { + case api.AllocDesiredStatusStop, api.AllocDesiredStatusEvict: + return false + } + + switch status { + case api.AllocClientStatusComplete, api.AllocClientStatusFailed, api.AllocClientStatusLost: + return false + } + + return true +} + // NodeAllocations lists what one machine of the cluster runs, whichever // namespace the work belongs to. func (c *Client) NodeAllocations(ctx context.Context, nodeID string) ([]Alloc, error) { diff --git a/internal/nomad/evaluation.go b/internal/nomad/evaluation.go index df44d90..3da307b 100644 --- a/internal/nomad/evaluation.go +++ b/internal/nomad/evaluation.go @@ -30,6 +30,12 @@ type EvaluationDetail struct { // Failures are the task groups it could not place, by name. Failures []PlacementFailure + + // Groups are what it planned for each group, and Preempted the + // allocations of other jobs it preempted. A cluster older than Nomad + // 1.11 sends neither. + Groups []PlanGroup + Preempted []Alloc } // PlacementFailure is why the allocations of one task group were not placed: @@ -78,6 +84,11 @@ func newEvaluationDetail(eval *api.Evaluation) EvaluationDetail { Failures: newPlacementFailures(eval.FailedTGAllocs), } + if eval.PlanAnnotations != nil { + detail.Groups = planGroups(eval.PlanAnnotations.DesiredTGUpdates) + detail.Preempted = allocsOf(eval.PlanAnnotations.PreemptedAllocs) + } + for _, stub := range eval.RelatedEvals { if stub == nil { continue diff --git a/internal/nomad/evaluation_test.go b/internal/nomad/evaluation_test.go index 5e1b848..184553e 100644 --- a/internal/nomad/evaluation_test.go +++ b/internal/nomad/evaluation_test.go @@ -123,6 +123,49 @@ func TestFailedPlacement_WhenNoneFailed(t *testing.T) { r.ErrorIs(err, nomad.ErrNoFailures) } +func TestEvaluation_WhatItPlannedForEachGroup(t *testing.T) { + r := require.New(t) + + client, _ := recorder(t, `{ + "ID": "e1", + "PlanAnnotations": { + "DesiredTGUpdates": { + "web": {"Place": 2, "Disconnect": 1}, + "api": {"Reconnect": 1, "RescheduleNow": 1} + }, + "PreemptedAllocs": [ + {"ID": "p1", "Name": "batch.work[0]", "Namespace": "other", "JobID": "batch", "TaskGroup": "work"} + ] + } + }`) + + eval, err := client.Evaluation(context.Background(), "production", "e1") + r.NoError(err) + + r.Equal([]nomad.PlanGroup{ + {Name: "api", Reconnect: 1, RescheduleNow: 1}, + {Name: "web", Place: 2, Disconnect: 1}, + }, eval.Groups) + + r.Len(eval.Preempted, 1) + r.Equal("p1", eval.Preempted[0].ID) + r.Equal("batch.work[0]", eval.Preempted[0].Name) + r.Equal("other", eval.Preempted[0].Namespace) +} + +func TestEvaluation_WithoutPlanAnnotations(t *testing.T) { + r := require.New(t) + + // A cluster older than Nomad 1.11 sends none. + client, _ := recorder(t, `{"ID": "e1"}`) + + eval, err := client.Evaluation(context.Background(), "production", "e1") + r.NoError(err) + + r.Empty(eval.Groups) + r.Empty(eval.Preempted) +} + func TestEvaluationAllocations(t *testing.T) { r := require.New(t) diff --git a/internal/nomad/identity_test.go b/internal/nomad/identity_test.go new file mode 100644 index 0000000..d7e79ea --- /dev/null +++ b/internal/nomad/identity_test.go @@ -0,0 +1,78 @@ +package nomad_test + +import ( + "context" + "net/http" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/ingvarch/urga/internal/nomad" +) + +const identityClaims = `{"Claims": { + "exp": 1792454400, "iat": 1792368000, "iss": "nomad", + "nomad_node_id": "node-1", "nomad_node_pool": "default" +}}` + +func TestNodeIdentity_ReadsTheExpiryAndKeepsTheClaims(t *testing.T) { + r := require.New(t) + + client, _, asked := metaServer(t, identityClaims) + + identity, err := client.NodeIdentity(context.Background(), "node-1") + r.NoError(err) + + r.Equal("/v1/client/identity", asked.URL.Path) + r.Equal("node-1", asked.URL.Query().Get("node_id")) + r.Empty(asked.URL.Query().Get("namespace")) + + r.True(time.Unix(1792454400, 0).Equal(identity.Expires)) + r.Contains(identity.Claims, `"nomad_node_id": "node-1"`) +} + +func TestNodeIdentity_WithoutAnExpiryHasNone(t *testing.T) { + for _, body := range []string{ + `{"Claims": {"iss": "nomad"}}`, + `{"Claims": {"exp": "soon"}}`, + `{"Claims": null}`, + } { + client, _, _ := metaServer(t, body) + + identity, err := client.NodeIdentity(context.Background(), "node-1") + require.NoError(t, err, body) + require.True(t, identity.Expires.IsZero(), body) + } +} + +func TestNodeIdentity_ANotFoundIsNoIdentity(t *testing.T) { + client := refusing(t, http.StatusNotFound, "") + + _, err := client.NodeIdentity(context.Background(), "node-1") + + require.ErrorIs(t, err, nomad.ErrNoIdentity) +} + +func TestNodeIdentity_AnotherErrorIsPassedOn(t *testing.T) { + r := require.New(t) + + client := refusing(t, http.StatusInternalServerError, "rpc error") + + _, err := client.NodeIdentity(context.Background(), "node-1") + + r.Error(err) + r.NotErrorIs(err, nomad.ErrNoIdentity) +} + +func TestRenewNodeIdentity_PostsTheNode(t *testing.T) { + r := require.New(t) + + client, sent, asked := metaServer(t, `{}`) + + r.NoError(client.RenewNodeIdentity(context.Background(), "node-1")) + + r.Equal(http.MethodPost, asked.Method) + r.Equal("/v1/client/identity/renew", asked.URL.Path) + r.JSONEq(`{"NodeID": "node-1"}`, string(*sent)) +} diff --git a/internal/nomad/limits_test.go b/internal/nomad/limits_test.go new file mode 100644 index 0000000..0614c1b --- /dev/null +++ b/internal/nomad/limits_test.go @@ -0,0 +1,405 @@ +package nomad_test + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "net/url" + "slices" + "strings" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/ingvarch/urga/internal/nomad" +) + +func TestAlloc_HoldsPlace(t *testing.T) { + for _, tc := range []struct { + desired, status string + holds bool + }{ + {"run", "running", true}, + {"run", "pending", true}, + {"run", "unknown", true}, + {"run", "complete", false}, + {"run", "failed", false}, + {"run", "lost", false}, + {"stop", "running", false}, + {"evict", "running", false}, + } { + alloc := nomad.Alloc{DesiredStatus: tc.desired, Status: tc.status} + + require.Equal(t, tc.holds, alloc.HoldsPlace(), "%s %s", tc.desired, tc.status) + } +} + +func TestNode_ReadsTheAllocationLimit(t *testing.T) { + r := require.New(t) + + client, _ := recorder(t, `{"ID": "node-1", "NodeMaxAllocs": 10}`) + + node, err := client.Node(context.Background(), "node-1") + r.NoError(err) + r.Equal(10, node.MaxAllocs) +} + +// limitsCluster answers the three requests WithAllocLimits makes, and keeps +// what was asked. +type limitsCluster struct { + // limits is NodeMaxAllocs by client id; a client not in it has none. + limits map[string]int + + // listStatus and gone make the filtered list and one client answer with + // an error. + listStatus int + gone string + + // infoStatus makes the read of every client answer with an error. + infoStatus int + + // allocsStatus makes the allocations list answer with an error. + allocsStatus int + + // together holds the reads of the clients until that many are asked at + // once, or until readsTimeout has passed and stalled is set. + together int + opened chan struct{} + open sync.Once + stalled bool + + // inFlight are the reads of clients asked and not answered yet, and + // most the most of them at once. + inFlight, most int + + allocs string + + // mu guards asked and the counts of reads: the client reads several + // clients at once. + mu sync.Mutex + asked []url.URL +} + +// readsTimeout is how long the reads of the clients wait to be asked +// together before the test fails. A cluster that reads them one at a time +// never gets there. +const readsTimeout = 10 * time.Second + +// heldOpen is how long the reads stay held once together of them are in +// flight, so that one more sent with them is counted too. It cannot fail a +// client that keeps to the limit. +const heldOpen = 200 * time.Millisecond + +// read counts a read of a client in flight and holds it until together of +// them are, and for heldOpen after that. +func (l *limitsCluster) read() { + l.mu.Lock() + l.inFlight++ + l.most = max(l.most, l.inFlight) + all := l.inFlight == l.together + l.mu.Unlock() + + switch { + case l.together == 0: + l.open.Do(func() { close(l.opened) }) + case all: + time.AfterFunc(heldOpen, func() { l.open.Do(func() { close(l.opened) }) }) + } + + select { + case <-l.opened: + case <-time.After(readsTimeout): + l.mu.Lock() + l.stalled = true + l.mu.Unlock() + l.open.Do(func() { close(l.opened) }) + } + + // The read is over before its answer is sent: the next one cannot be + // asked before then. + l.mu.Lock() + l.inFlight-- + l.mu.Unlock() +} + +// reads are the most reads of clients in flight at once, and whether they +// waited too long for that many. +func (l *limitsCluster) reads() (most int, stalled bool) { + l.mu.Lock() + defer l.mu.Unlock() + + return l.most, l.stalled +} + +func (l *limitsCluster) client(t *testing.T) *nomad.Client { + t.Helper() + + l.opened = make(chan struct{}) + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { + l.mu.Lock() + l.asked = append(l.asked, *req.URL) + l.mu.Unlock() + + w.Header().Set("Content-Type", "application/json") + + switch path := req.URL.Path; { + case path == "/v1/nodes": + if l.listStatus != 0 { + http.Error(w, "bad filter", l.listStatus) + + return + } + + stubs := []string{} + for id := range l.limits { + stubs = append(stubs, fmt.Sprintf(`{"ID": %q}`, id)) + } + + _, _ = w.Write([]byte("[" + strings.Join(stubs, ",") + "]")) + case strings.HasPrefix(path, "/v1/node/"): + id := strings.TrimPrefix(path, "/v1/node/") + l.read() + + if id == l.gone { + http.Error(w, "node not found", http.StatusNotFound) + + return + } + + if l.infoStatus != 0 { + http.Error(w, "rpc error", l.infoStatus) + + return + } + + _, _ = fmt.Fprintf(w, `{"ID": %q, "NodeMaxAllocs": %d}`, id, l.limits[id]) + case path == "/v1/allocations": + if l.allocsStatus != 0 { + http.Error(w, "permission denied", l.allocsStatus) + + return + } + + _, _ = w.Write([]byte(l.allocs)) + default: + http.Error(w, "not found", http.StatusNotFound) + } + })) + t.Cleanup(server.Close) + + client, err := nomad.New(nomad.Config{Address: server.URL}) + require.NoError(t, err) + + return client +} + +func (l *limitsCluster) requests() []url.URL { + l.mu.Lock() + defer l.mu.Unlock() + + return slices.Clone(l.asked) +} + +func (l *limitsCluster) askedPaths() []string { + asked := l.requests() + + paths := make([]string, 0, len(asked)) + for _, u := range asked { + paths = append(paths, u.Path) + } + + return paths +} + +const limitAllocs = `[ + {"ID": "a1", "NodeID": "node-1", "DesiredStatus": "run", "ClientStatus": "running"}, + {"ID": "a2", "NodeID": "node-1", "DesiredStatus": "run", "ClientStatus": "pending"}, + {"ID": "a3", "NodeID": "node-1", "DesiredStatus": "run", "ClientStatus": "complete"}, + {"ID": "a4", "NodeID": "node-1", "DesiredStatus": "stop", "ClientStatus": "running"}, + {"ID": "a5", "NodeID": "node-2", "DesiredStatus": "run", "ClientStatus": "running"} +]` + +func TestWithAllocLimits_CountsWhatHoldsAPlace(t *testing.T) { + r := require.New(t) + + cluster := &limitsCluster{limits: map[string]int{"node-1": 2}, allocs: limitAllocs} + client := cluster.client(t) + + nodes := []nomad.Node{{ID: "node-1"}, {ID: "node-2"}} + + got, err := client.WithAllocLimits(context.Background(), nodes) + r.NoError(err) + + // The limited client has its limit and the allocations that hold a place; + // the complete one and the one that is being stopped do not count. + r.Equal([]nomad.Node{{ID: "node-1", MaxAllocs: 2, Allocs: 2}, {ID: "node-2"}}, got) + + // The nodes it was given are not changed. + r.Zero(nodes[0].MaxAllocs) +} + +func TestWithAllocLimits_AsksForTheLimitedClientsOnly(t *testing.T) { + r := require.New(t) + + cluster := &limitsCluster{limits: map[string]int{"node-1": 2}, allocs: limitAllocs} + client := cluster.client(t) + + _, err := client.WithAllocLimits(context.Background(), []nomad.Node{{ID: "node-1"}, {ID: "node-2"}}) + r.NoError(err) + + r.Equal([]string{"/v1/nodes", "/v1/node/node-1", "/v1/allocations"}, cluster.askedPaths()) + + asked := cluster.requests() + r.Equal("NodeMaxAllocs != 0", asked[0].Query().Get("filter")) + r.Equal("*", asked[2].Query().Get("namespace")) + r.Equal(`NodeID == "node-1"`, asked[2].Query().Get("filter")) +} + +func TestWithAllocLimits_FiltersOnEveryLimitedClient(t *testing.T) { + r := require.New(t) + + cluster := &limitsCluster{limits: map[string]int{"node-1": 2}, allocs: `[]`} + client := cluster.client(t) + + // A cluster that answers for node-2 as well, with a limit of its own. + cluster.limits["node-2"] = 5 + + got, err := client.WithAllocLimits(context.Background(), []nomad.Node{{ID: "node-1"}, {ID: "node-2"}}) + r.NoError(err) + r.Equal(5, got[1].MaxAllocs) + + asked := cluster.requests() + r.Equal(`NodeID == "node-1" or NodeID == "node-2"`, asked[len(asked)-1].Query().Get("filter")) +} + +func TestWithAllocLimits_NoLimitedClientAsksForNothingMore(t *testing.T) { + r := require.New(t) + + cluster := &limitsCluster{limits: map[string]int{}} + client := cluster.client(t) + + nodes := []nomad.Node{{ID: "node-1"}} + + got, err := client.WithAllocLimits(context.Background(), nodes) + r.NoError(err) + r.Equal(nodes, got) + r.Equal([]string{"/v1/nodes"}, cluster.askedPaths()) +} + +func TestWithAllocLimits_AClusterThatIgnoresTheFilterMarksNothing(t *testing.T) { + r := require.New(t) + + // An older cluster lists every client, and each read shows no limit. + cluster := &limitsCluster{limits: map[string]int{"node-1": 0}} + client := cluster.client(t) + + nodes := []nomad.Node{{ID: "node-1"}} + + got, err := client.WithAllocLimits(context.Background(), nodes) + r.NoError(err) + r.Equal(nodes, got) + r.Equal([]string{"/v1/nodes", "/v1/node/node-1"}, cluster.askedPaths()) +} + +func TestWithAllocLimits_AClusterThatRefusesTheFilterLeavesTheNodes(t *testing.T) { + r := require.New(t) + + cluster := &limitsCluster{listStatus: http.StatusBadRequest} + client := cluster.client(t) + + nodes := []nomad.Node{{ID: "node-1"}} + + got, err := client.WithAllocLimits(context.Background(), nodes) + r.NoError(err) + r.Equal(nodes, got) +} + +func TestWithAllocLimits_OtherErrorsAreReturned(t *testing.T) { + limits := map[string]int{"node-1": 2} + + for name, cluster := range map[string]*limitsCluster{ + "the filtered list": {listStatus: http.StatusInternalServerError}, + "a client": {limits: limits, infoStatus: http.StatusInternalServerError}, + "the allocations": {limits: limits, allocsStatus: http.StatusInternalServerError}, + } { + _, err := cluster.client(t).WithAllocLimits(context.Background(), []nomad.Node{{ID: "node-1"}}) + require.Error(t, err, name) + } +} + +func TestWithAllocLimits_ReadsOnlyTheClientsItWasGiven(t *testing.T) { + r := require.New(t) + + // node-9 is of another pool: the filtered list names it, the list it + // was given does not. + cluster := &limitsCluster{limits: map[string]int{"node-1": 2, "node-9": 5}, allocs: limitAllocs} + + got, err := cluster.client(t).WithAllocLimits(context.Background(), []nomad.Node{{ID: "node-1"}, {ID: "node-2"}}) + r.NoError(err) + r.Equal([]nomad.Node{{ID: "node-1", MaxAllocs: 2, Allocs: 2}, {ID: "node-2"}}, got) + + r.NotContains(cluster.askedPaths(), "/v1/node/node-9") + + asked := cluster.requests() + r.Equal(`NodeID == "node-1"`, asked[len(asked)-1].Query().Get("filter")) +} + +func TestWithAllocLimits_AClientThatIsGoneHasNoLimit(t *testing.T) { + r := require.New(t) + + cluster := &limitsCluster{ + limits: map[string]int{"node-1": 2, "node-2": 3}, + gone: "node-2", + allocs: limitAllocs, + } + client := cluster.client(t) + + got, err := client.WithAllocLimits(context.Background(), []nomad.Node{{ID: "node-1"}, {ID: "node-2"}}) + r.NoError(err) + r.Equal([]nomad.Node{{ID: "node-1", MaxAllocs: 2, Allocs: 2}, {ID: "node-2"}}, got) +} + +func TestWithAllocLimits_ATokenThatMayNotListAllocationsLeavesTheNodes(t *testing.T) { + r := require.New(t) + + cluster := &limitsCluster{limits: map[string]int{"node-1": 2}, allocsStatus: http.StatusForbidden} + client := cluster.client(t) + + nodes := []nomad.Node{{ID: "node-1"}} + + got, err := client.WithAllocLimits(context.Background(), nodes) + r.NoError(err) + r.Equal(nodes, got) +} + +func TestWithAllocLimits_ManyLimitedClientsAreReadTogether(t *testing.T) { + r := require.New(t) + + limits := map[string]int{} + nodes := []nomad.Node{} + + for i := range 40 { + id := fmt.Sprintf("node-%02d", i) + limits[id] = 2 + nodes = append(nodes, nomad.Node{ID: id}) + } + + // Eight is how many clients are read at once. + cluster := &limitsCluster{limits: limits, allocs: `[]`, together: 8} + client := cluster.client(t) + + got, err := client.WithAllocLimits(context.Background(), nodes) + r.NoError(err) + + for _, node := range got { + r.Equal(2, node.MaxAllocs, node.ID) + } + + most, stalled := cluster.reads() + r.False(stalled, "fewer than 8 clients were read at once") + r.Equal(8, most) +} diff --git a/internal/nomad/node.go b/internal/nomad/node.go index 29bf685..74ded97 100644 --- a/internal/nomad/node.go +++ b/internal/nomad/node.go @@ -1,10 +1,17 @@ package nomad import ( + "cmp" "context" "encoding/json" + "errors" "fmt" + "maps" + "net/http" + "slices" "sort" + "strings" + "sync" "time" "github.com/hashicorp/nomad/api" @@ -60,6 +67,50 @@ type NodeDetail struct { Attributes map[string]string } +// NodeIdentity is the identity the servers gave a client: when it expires, +// and its claims as JSON, the way a describe shows them. +type NodeIdentity struct { + Expires time.Time + Claims string +} + +// ErrNoIdentity says the cluster has no identity to read for the client: it +// is older than Nomad 1.11, or the client is not connected. +var ErrNoIdentity = errors.New("the client has no identity to read") + +// NodeIdentity reads the identity of a client. A client belongs to no +// namespace. +func (c *Client) NodeIdentity(ctx context.Context, nodeID string) (NodeIdentity, error) { + got, err := c.api.Nodes().Identity().Get(&api.NodeIdentityGetRequest{NodeID: nodeID}, c.query(ctx, "")) + if answered(err, http.StatusNotFound) { + return NodeIdentity{}, ErrNoIdentity + } + + if err != nil { + return NodeIdentity{}, err + } + + claims, err := asJSON(got.Claims) + if err != nil { + return NodeIdentity{}, err + } + + identity := NodeIdentity{Claims: claims} + if exp, ok := got.Claims["exp"].(float64); ok { + identity.Expires = time.Unix(int64(exp), 0) + } + + return identity, nil +} + +// RenewNodeIdentity asks a client to get a new identity at its next +// heartbeat. +func (c *Client) RenewNodeIdentity(ctx context.Context, nodeID string) error { + _, err := c.api.Nodes().Identity().Renew(&api.NodeIdentityRenewRequest{NodeID: nodeID}, c.query(ctx, "")) + + return err +} + // Node reads one client from the cluster instead of from the list, so a // screen open on it shows its current state. func (c *Client) Node(ctx context.Context, nodeID string) (Node, error) { @@ -70,7 +121,7 @@ func (c *Client) Node(ctx context.Context, nodeID string) (Node, error) { // The answer holds more than a list entry; copying it into a list stub // lets newNode build both. - return newNode(&api.NodeListStub{ + full := newNode(&api.NodeListStub{ ID: node.ID, Name: node.Name, Datacenter: node.Datacenter, @@ -81,7 +132,140 @@ func (c *Client) Node(ctx context.Context, nodeID string) (Node, error) { Drain: node.Drain, Address: node.HTTPAddr, NodeResources: node.NodeResources, - }), nil + }) + full.MaxAllocs = node.NodeMaxAllocs + + return full, nil +} + +// WithAllocLimits returns the clients with the allocation limit of those that +// have one, and how many allocations hold a place on them. The list does not +// carry the limit, so it reads the clients the cluster says have one, and +// the allocations of those only. +func (c *Client) WithAllocLimits(ctx context.Context, nodes []Node) ([]Node, error) { + q := c.query(ctx, "") + q.Filter = "NodeMaxAllocs != 0" + + stubs, _, err := c.api.Nodes().List(q) + if err != nil { + // A cluster older than 1.10.2 may refuse a filter on a field it lacks. + if answered(err, http.StatusBadRequest) { + return nodes, nil + } + + return nil, err + } + + limits, err := c.allocLimits(ctx, nodes, stubs) + if err != nil { + return nil, err + } + + if len(limits) == 0 { + return nodes, nil + } + + held, err := c.heldPlaces(ctx, limits) + // A token that reads clients but no job in any namespace cannot count + // the allocations; the clients go without a limit. + if Forbidden(err) { + return nodes, nil + } + + if err != nil { + return nil, err + } + + out := slices.Clone(nodes) + for i, node := range out { + if limit, ok := limits[node.ID]; ok { + out[i].MaxAllocs, out[i].Allocs = limit, held[node.ID] + } + } + + return out, nil +} + +// limitReaders is how many clients are read at once. +const limitReaders = 8 + +// allocLimits reads the limit of each client of nodes that the filtered list +// named, a few at a time. A client that is gone since the list is left out. +func (c *Client) allocLimits(ctx context.Context, nodes []Node, stubs []*api.NodeListStub) (map[string]int, error) { + ours := make(map[string]bool, len(nodes)) + for _, node := range nodes { + ours[node.ID] = true + } + + var ( + mu sync.Mutex + wg sync.WaitGroup + firstErr error + limits = map[string]int{} + slots = make(chan struct{}, limitReaders) + ) + + for _, stub := range stubs { + if !ours[stub.ID] { + continue + } + + slots <- struct{}{} + + wg.Go(func() { + defer func() { <-slots }() + + node, _, err := c.api.Nodes().Info(stub.ID, c.query(ctx, "")) + + mu.Lock() + defer mu.Unlock() + + switch { + case answered(err, http.StatusNotFound): + case err != nil: + firstErr = cmp.Or(firstErr, err) + case node.NodeMaxAllocs != 0: + limits[stub.ID] = node.NodeMaxAllocs + } + }) + } + + wg.Wait() + + if firstErr != nil { + return nil, firstErr + } + + return limits, nil +} + +// heldPlaces counts, per client of limits, the allocations that hold a place +// on it. One request asks for the allocations of those clients only. +func (c *Client) heldPlaces(ctx context.Context, limits map[string]int) (map[string]int, error) { + ids := slices.Sorted(maps.Keys(limits)) + + terms := make([]string, 0, len(ids)) + for _, id := range ids { + terms = append(terms, fmt.Sprintf("NodeID == %q", id)) + } + + q := c.query(ctx, AllNamespaces) + q.Filter = strings.Join(terms, " or ") + + allocs, _, err := c.api.Allocations().List(q) + if err != nil { + return nil, err + } + + held := map[string]int{} + + for _, alloc := range allocs { + if holdsPlace(alloc.DesiredStatus, alloc.ClientStatus) { + held[alloc.NodeID]++ + } + } + + return held, nil } // NodeDetail reads everything the cluster holds about one client. diff --git a/internal/nomad/plan.go b/internal/nomad/plan.go index 209b7e1..21908b5 100644 --- a/internal/nomad/plan.go +++ b/internal/nomad/plan.go @@ -54,6 +54,13 @@ type PlanGroup struct { Migrate int Preemptions int + // The scheduler counts an allocation rescheduled now in Place too: these + // four say why, they are not more allocations. + RescheduleNow int + RescheduleLater int + Disconnect int + Reconnect int + // Ignore are the allocations the plan leaves as they are. Ignore int } @@ -91,7 +98,15 @@ func newPlan(answer *api.JobPlanResponse) Plan { return plan } - updates := answer.Annotations.DesiredTGUpdates + plan.Groups = planGroups(answer.Annotations.DesiredTGUpdates) + + return plan +} + +// planGroups puts what the scheduler would do to each group in the order of +// the names; a map returns its keys in random order. +func planGroups(updates map[string]*api.DesiredUpdates) []PlanGroup { + var groups []PlanGroup for _, name := range slices.Sorted(maps.Keys(updates)) { update := updates[name] @@ -99,20 +114,24 @@ func newPlan(answer *api.JobPlanResponse) Plan { continue } - plan.Groups = append(plan.Groups, PlanGroup{ - Name: name, - Place: int(update.Place), - Stop: int(update.Stop), - InPlace: int(update.InPlaceUpdate), - Destructive: int(update.DestructiveUpdate), - Canary: int(update.Canary), - Migrate: int(update.Migrate), - Preemptions: int(update.Preemptions), - Ignore: int(update.Ignore), + groups = append(groups, PlanGroup{ + Name: name, + Place: int(update.Place), + Stop: int(update.Stop), + InPlace: int(update.InPlaceUpdate), + Destructive: int(update.DestructiveUpdate), + Canary: int(update.Canary), + Migrate: int(update.Migrate), + Preemptions: int(update.Preemptions), + RescheduleNow: int(update.RescheduleNow), + RescheduleLater: int(update.RescheduleLater), + Disconnect: int(update.Disconnect), + Reconnect: int(update.Reconnect), + Ignore: int(update.Ignore), }) } - return plan + return groups } // jobChanged says the cluster refused a change planned on a job that has diff --git a/internal/nomad/plan_test.go b/internal/nomad/plan_test.go index 3b5a407..f8e14a8 100644 --- a/internal/nomad/plan_test.go +++ b/internal/nomad/plan_test.go @@ -66,6 +66,21 @@ func TestPlanJob(t *testing.T) { r.Contains(plan.Warnings, "has warnings") } +func TestPlanJob_DisconnectsReconnectsAndReschedules(t *testing.T) { + r := require.New(t) + + client, _ := clusterServer(t, map[string]string{"/v1/job/web/plan": `{ + "Annotations": {"DesiredTGUpdates": { + "web": {"RescheduleNow": 1, "RescheduleLater": 2, "Disconnect": 3, "Reconnect": 4} + }} + }`}) + + plan, err := client.PlanJob(context.Background(), "production", `{"ID": "web", "Name": "web"}`, nomad.JobVariables{}, nil) + r.NoError(err) + + r.Equal([]nomad.PlanGroup{{Name: "web", RescheduleNow: 1, RescheduleLater: 2, Disconnect: 3, Reconnect: 4}}, plan.Groups) +} + func TestPlanJob_HCLIsReadByTheClusterFirst(t *testing.T) { r := require.New(t) diff --git a/internal/nomad/resources.go b/internal/nomad/resources.go index 8b4644d..c1f983f 100644 --- a/internal/nomad/resources.go +++ b/internal/nomad/resources.go @@ -128,6 +128,13 @@ type Node struct { // measured against them. CPUShares int MemoryMB int + + // MaxAllocs is how many allocations the client takes at most, zero for + // no limit, and Allocs how many hold a place on it. Only WithAllocLimits + // counts Allocs, and only for a client with a limit; Node leaves it + // zero, because the client screen counts its own list. + MaxAllocs int + Allocs int } // Nodes lists the clients of the cluster. diff --git a/internal/ui/app_test.go b/internal/ui/app_test.go index 5d0fc4f..e35a372 100644 --- a/internal/ui/app_test.go +++ b/internal/ui/app_test.go @@ -120,7 +120,7 @@ type fakeClient struct { err error raftErr error - // nodesErr and allocsErr fail the client list or the allocation list + // nodesErr and allocsErr fail the client list or a list of allocations // alone, ahead of err, as a token that may not read one of them does. nodesErr error allocsErr error @@ -129,6 +129,13 @@ type fakeClient struct { // allocations carry the job after a slash. lists map[string][]string + // identity is what a client answers for its identity; nil answers + // nomad.ErrNoIdentity, as a cluster older than 1.11 does. + identity *nomad.NodeIdentity + identityErr error + identityAsked []string + renewed []string + askedNamespace string askedJobID string askedNodeID string @@ -425,6 +432,10 @@ func (f *fakeClient) FailedPlacement(_ context.Context, namespace, jobID string) func (f *fakeClient) NodeAllocations(_ context.Context, nodeID string) ([]nomad.Alloc, error) { f.askedNodeID = nodeID + if f.allocsErr != nil { + return nil, f.allocsErr + } + return f.nodeAllocs, f.err } @@ -467,6 +478,33 @@ func (f *fakeClient) Node(_ context.Context, nodeID string) (nomad.Node, error) return nomad.Node{}, f.err } +func (f *fakeClient) NodeIdentity(_ context.Context, nodeID string) (nomad.NodeIdentity, error) { + f.identityAsked = append(f.identityAsked, nodeID) + + if f.identityErr != nil { + return nomad.NodeIdentity{}, f.identityErr + } + + if f.identity == nil { + return nomad.NodeIdentity{}, nomad.ErrNoIdentity + } + + return *f.identity, nil +} + +func (f *fakeClient) RenewNodeIdentity(_ context.Context, nodeID string) error { + f.wrote("RenewNodeIdentity") + f.renewed = append(f.renewed, nodeID) + + return f.actionErr +} + +func (f *fakeClient) WithAllocLimits(_ context.Context, nodes []nomad.Node) ([]nomad.Node, error) { + f.listed("limits", "") + + return nodes, nil +} + func (f *fakeClient) NodeDetail(_ context.Context, nodeID string) (nomad.NodeDetail, error) { f.askedNodeID = nodeID diff --git a/internal/ui/client.go b/internal/ui/client.go index bf726ce..e50b38f 100644 --- a/internal/ui/client.go +++ b/internal/ui/client.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "image/color" + "strconv" "strings" "time" @@ -23,14 +24,11 @@ const ( hostChartShort = 7 hostChartRest = 2 - // hostPanelRest is the panel around the charts: the details of the - // machine, a line under them and an empty line above the allocations. + // hostPanelRest is the panel around the charts: the first row of the + // details of the machine, a line under them and an empty line above the + // allocations. chartHeight counts the other rows of details. hostPanelRest = 3 - // hostPanelRows is the panel without a chart, which is what a screen - // with no room for one still shows. - hostPanelRows = 2 - // rowsKept are the rows a table under a panel shows whatever else is on // the screen: the allocations of a client, the tasks of an allocation. // The panel shrinks to leave them room. @@ -67,6 +65,10 @@ type hostModel struct { node nomad.Node trail []nomad.ResourceUse + // identity is the identity of the client as last read, nil while it has + // none or is not read yet. + identity *nomad.NodeIdentity + // due is true while the next reading or its timer is pending. The list // arrives again on every poll, and none of those answers may start a // second chain of readings next to the first. @@ -108,6 +110,10 @@ type clientPage struct { absolute bool allocs []nomad.Alloc + + // listed is true once the allocations are read: until then the page + // has none to count. + listed bool } // clientOf is a client with what is known of the machine so far. @@ -134,18 +140,24 @@ func (clientPage) where(env) string { return nomad.AllNamespaces } func (p clientPage) fetch(e env) tea.Cmd { client, nodeID := e.client, p.nodeID - return tea.Batch( + cmds := []tea.Cmd{ fetchList(func(ctx context.Context) ([]nomad.Alloc, error) { return client.NodeAllocations(ctx, nodeID) }, func(items []nomad.Alloc) tea.Msg { return allocsMsg(items) }), fetchHost(client, nodeID), - ) + } + + if hasIdentity(p.host.node) { + cmds = append(cmds, fetchIdentity(client, nodeID)) + } + + return tea.Batch(cmds...) } func (p clientPage) take(msg tea.Msg, e env) (page, outcome, bool) { switch msg := msg.(type) { case allocsMsg: - p.allocs = msg + p.allocs, p.listed = msg, true return p.hostOnce(e) @@ -157,8 +169,24 @@ func (p clientPage) take(msg tea.Msg, e env) (page, outcome, bool) { // The panel shows the machine as last read. This answer is not the // list: it does not clear an error of the list from the status line, // and it does not reschedule the poll. + known := hasIdentity(p.host.node) p.host.node = nomad.Node(msg) + // The identity shows without waiting for the next poll once the + // machine says it is new enough to have one. + if !known && hasIdentity(p.host.node) { + return p, outcome{cmd: fetchIdentity(e.client, p.nodeID), reading: true}, true + } + + return p, outcome{reading: true}, true + + case identityMsg: + if msg.nodeID != p.nodeID { + return p, outcome{}, false + } + + p.host.identity = msg.identity + return p, outcome{reading: true}, true case hostUseMsg: @@ -243,6 +271,8 @@ var clientKeys = append([]pageKey[clientPage]{ {press: "a", label: "Attributes", do: nodeScreen(func(d machine) page { return nodeAttributesPage{machine: d} })}, {press: "m", label: "Meta", do: nodeScreen(func(d machine) page { return nodeMetaPage{nodeID: d.nodeID, name: d.name} })}, {press: "u", label: "Units", do: toggleUnits}, + {press: "i", label: "Identity", do: showIdentity, offered: holdsIdentity}, + {press: "ctrl+n", label: "Renew", do: renewIdentity, writes: true, offered: holdsIdentity}, }, allocKeys[clientPage]()...) func (p clientPage) keys(e env) []keyHint { return hintsOf(p, e, clientKeys) } @@ -278,11 +308,11 @@ func chartWidth(width int) int { return max((width-tableIndent-columnGap)/2, 1) } -// chartHeight is how tall the plot of a chart is in a panel of room rows. A -// short screen gets the short one, and then none at all: the allocations -// come first. -func chartHeight(room int) int { - room -= hostChartRest + hostPanelRest +// chartHeight is how tall the plot of a chart is in a panel of room rows +// that holds details rows of the machine. A short screen gets the short one, +// and then none at all: the allocations come first. +func chartHeight(room, details int) int { + room -= hostChartRest + hostPanelRest + details - 1 switch { case room >= hostChartHeight: @@ -321,25 +351,35 @@ func (m Model) panel(width int) []string { // panel is what the machine is doing, above its allocations. func (p clientPage) panel(_ env, width, room int) []string { - // A box with no room even for the details of the machine gives all its - // rows to the allocations. - if room < hostPanelRows { + details := p.host.details(width, p.held()) + + // A box with no room even for the details of the machine, and the line + // under them, gives all its rows to the allocations. + if room < len(details)+1 { return nil } half := chartWidth(width) - return p.host.view(width, half, chartHeight(room), p.absolute) + return p.host.view(half, chartHeight(room, len(details)), p.absolute, details) } -// view is what a client shows above its allocations: what kind of machine -// it is and what it has been doing since the screen was opened. Each chart -// is half wide and plot tall, and a plot of no height leaves them out. -func (h hostModel) view(width, half, plot int, absolute bool) []string { - // One column of the panel goes to the margin the table keeps. - width -= tableIndent +// held is how many allocations of the client hold a place on it, unknown +// until they are read. +func (p clientPage) held() string { + if !p.listed { + return unknown + } - rows := []string{h.details(width), ""} + return strconv.Itoa(len(keep(p.allocs, nomad.Alloc.HoldsPlace))) +} + +// view is what a client shows above its allocations: what kind of machine +// it is, in the rows of details it starts with, and what it has been doing +// since the screen was opened. Each chart is half wide and plot tall, and a +// plot of no height leaves them out. +func (h hostModel) view(half, plot int, absolute bool, rows []string) []string { + rows = append(rows, panelRow("")) if plot > 0 { cpu, memory := alignLabels(h.chart(cpuMeasure, absolute), h.chart(memoryMeasure, absolute)) @@ -347,22 +387,17 @@ func (h hostModel) view(width, half, plot int, absolute bool) []string { memoryLines := memory.render(half, plot) for i := range cpuLines { - rows = append(rows, pad(cpuLines[i], half)+strings.Repeat(" ", columnGap)+memoryLines[i]) + rows = append(rows, panelRow(pad(cpuLines[i], half)+strings.Repeat(" ", columnGap)+memoryLines[i])) } // A narrow screen gets no charts: half of it has no room for one // with its scale. Then there is nothing to keep apart from the // allocations either. if len(cpuLines) > 0 { - rows = append(rows, "") + rows = append(rows, panelRow("")) } } - // The panel starts where the columns of the table do. - for i, row := range rows { - rows[i] = panelRow(row) - } - return rows } @@ -472,17 +507,31 @@ func (h hostModel) chart(of measure, absolute bool) chart { } // details is the machine itself, in the fields the Nomad interface shows. -func (h hostModel) details(width int) string { +// The first row is kept, blank while the machine is not read yet, so the +// panel does not grow when it is; the next is left out when it has no +// value. held is how many allocations hold a place on it, shown against the +// limit when it has one. The rows start where the columns of the table do. +func (h hostModel) details(width int, held string) []string { node := h.node - return fieldLine([]field{ + rows := []string{panelRow(fieldLine([]field{ {"Status", node.Status}, {"Address", node.Address}, {"Datacenter", node.Datacenter}, {"Pool", node.NodePool}, {"Version", node.Version}, {"Scheduling", eligibilityOf(node)}, - }, width) + }, width-tableIndent))} + + allocations := "" + if node.MaxAllocs > 0 { + allocations = fmt.Sprintf("%s of %d", held, node.MaxAllocs) + } + + return append(rows, fieldLineRows([][]field{{ + {"Allocations", allocations}, + {"Identity", identityText(h.identity)}, + }}, width)...) } // eligibilityOf says whether the machine is taking work. diff --git a/internal/ui/client_test.go b/internal/ui/client_test.go index 46236a5..89f8837 100644 --- a/internal/ui/client_test.go +++ b/internal/ui/client_test.go @@ -644,14 +644,37 @@ func TestClient_ThePanelTakesWhatTheBoxHasRoomFor(t *testing.T) { } } +func TestClient_OpenedByIDKeepsTheRowOfTheMachineUntilItIsRead(t *testing.T) { + r := require.New(t) + + // Opened from an allocation, the page knows the id and the name only. + m, _ := newTestModel(&fakeClient{}).update(openMsg{clientOf(nomad.Node{ID: "node-1", Name: "nomad-server-01"})}) + r.IsType(clientPage{}, m.screen.page) + + before := m.panelHeight() + r.Equal(before, drawnPanel(m)) + + // The row is there, blank, and the panel does not grow when the machine + // answers. + m, _ = m.update(hostMsg(busyClient()[0])) + + r.Contains(bodyOf(m), "10.0.0.2") + r.Equal(before, m.panelHeight()) + r.Equal(before, drawnPanel(m)) +} + func TestPanel_TakesTheRowsItDraws(t *testing.T) { r := require.New(t) client, _ := onClient(t) client, _ = client.update(reading(29)) + limited := limitedClient(t, 3, clientAllocs()) + limited, _ = limited.update(reading(29)) + screens := map[string]Model{ "client": client, + "limited": limited, "tasks": openCanary(t, &fakeClient{jobs: twoJobs(), allocs: twoAllocs()}), "deployment": onDeployment(t, &fakeClient{}), "variable": onVariable(t, &fakeClient{}, leaderVariable()), @@ -709,8 +732,8 @@ func TestClient_ChartsTooNarrowForTheirScaleTakeNoRows(t *testing.T) { // dropped, and so is the empty row under them. m, _ = m.update(key('u')) - r.Equal(hostPanelRows, m.panelHeight()) - r.Equal(hostPanelRows, drawnPanel(m)) + r.Equal(2, m.panelHeight()) + r.Equal(2, drawnPanel(m)) r.NotContains(bodyOf(m), "now") } diff --git a/internal/ui/evaluation.go b/internal/ui/evaluation.go index 929b8a2..54cf849 100644 --- a/internal/ui/evaluation.go +++ b/internal/ui/evaluation.go @@ -162,7 +162,7 @@ func describeEvaluation(client evaluationsClient, namespace, id string) tea.Cmd func evaluationScreen(eval nomad.EvaluationDetail) describeMsg { return describeMsg{ label: fmt.Sprintf("Evaluation %s", shortID(eval.ID)), - content: evaluationText(eval), + lines: evaluationLines(eval), evaluation: evalRef{eval.Namespace, eval.ID}, } } @@ -228,9 +228,10 @@ func placementOf(client evaluationsClient, namespace, jobID string) tea.Cmd { }, func(msg describeMsg) tea.Msg { return msg }) } -// evaluationText is an evaluation as a page: its fields, why each group it -// could not place was not placed, and the evaluations it is chained to. -func evaluationText(eval nomad.EvaluationDetail) string { +// evaluationLines are an evaluation as a page: its fields, why each group it +// could not place was not placed, what it planned for each group, the +// allocations it preempted, and the evaluations it is chained to. +func evaluationLines(eval nomad.EvaluationDetail) []paintedLine { status := eval.Status if eval.StatusDescription != "" { status += ": " + eval.StatusDescription @@ -260,20 +261,35 @@ func evaluationText(eval nomad.EvaluationDetail) string { lines = append(lines, fmt.Sprintf("%-13s %s", field[0], field[1])) } + painted := plainLines(strings.Join(lines, "\n")) + if len(eval.Failures) > 0 { - lines = append(lines, "", "Placement failures", "") - lines = append(lines, placementLines(eval.Failures)...) + painted = append(painted, paintedLine{}, paintedLine{text: "Placement failures"}, paintedLine{}) + painted = append(painted, plainLines(strings.Join(placementLines(eval.Failures), "\n"))...) + } + + if len(eval.Groups) > 0 { + painted = append(painted, paintedLine{}, paintedLine{text: "Planned for each group"}, paintedLine{}) + painted = append(painted, groupLines(eval.Groups)...) + } + + if len(eval.Preempted) > 0 { + painted = append(painted, paintedLine{}, paintedLine{text: "Preempted allocations"}, paintedLine{}) + + for _, a := range eval.Preempted { + painted = append(painted, paintedLine{text: fmt.Sprintf(" %s %s %s", shortID(a.ID), a.Name, a.Namespace)}) + } } if len(eval.Related) > 0 { - lines = append(lines, "", "Related evaluations", "") + painted = append(painted, paintedLine{}, paintedLine{text: "Related evaluations"}, paintedLine{}) for _, related := range eval.Related { - lines = append(lines, fmt.Sprintf(" %s %s %s", shortID(related.ID), related.Status, related.TriggeredBy)) + painted = append(painted, paintedLine{text: fmt.Sprintf(" %s %s %s", shortID(related.ID), related.Status, related.TriggeredBy)}) } } - return strings.Join(lines, "\n") + return painted } // placementLines says why each group was not placed, in the words of the diff --git a/internal/ui/evaluation_test.go b/internal/ui/evaluation_test.go index f43154d..7468534 100644 --- a/internal/ui/evaluation_test.go +++ b/internal/ui/evaluation_test.go @@ -1,6 +1,7 @@ package ui import ( + "slices" "strings" "testing" "time" @@ -116,10 +117,10 @@ func TestEvaluation_OpensTheOneUnderTheCursorWhereItLives(t *testing.T) { r.Equal(other.ID, client.askedID) } -func TestEvaluationText(t *testing.T) { +func TestEvaluationLines(t *testing.T) { r := require.New(t) - text := evaluationText(shortOfRoom()) + text := strings.Join(texts(evaluationLines(shortOfRoom())), "\n") r.Contains(text, "Evaluation 1ad88fd0-6b1c-da67-5e8f-482b5a15af1f") r.Contains(text, "Triggered by queued-allocs") @@ -133,16 +134,81 @@ func TestEvaluationText(t *testing.T) { r.NotContains(text, "Next ") } -func TestEvaluationText_WhenEverythingWasPlaced(t *testing.T) { +func TestEvaluationLines_WhenEverythingWasPlaced(t *testing.T) { r := require.New(t) placed := nomad.EvaluationDetail{Evaluation: nomad.Evaluation{ID: "e1", Status: "complete"}} - text := evaluationText(placed) + text := strings.Join(texts(evaluationLines(placed)), "\n") r.NotContains(text, "Placement failures") + r.NotContains(text, "Planned for each group") + r.NotContains(text, "Preempted allocations") r.True(strings.HasPrefix(text, "Evaluation e1")) } +func plannedEvaluation() nomad.EvaluationDetail { + eval := shortOfRoom() + eval.Groups = []nomad.PlanGroup{ + {Name: "web", Place: 1, Disconnect: 2, RescheduleNow: 1}, + } + eval.Preempted = []nomad.Alloc{{ID: "p1234567-0000-0000-0000-000000000000", Name: "batch.work[0]", Namespace: "other"}} + + return eval +} + +func TestEvaluationLines_WhatItPlannedForEachGroup(t *testing.T) { + r := require.New(t) + + lines := evaluationLines(plannedEvaluation()) + words := texts(lines) + + r.Subset(words, []string{ + "Planned for each group", + ` Task group "web"`, + " ● 1 new allocation will be created", + " ● 1 allocation will be rescheduled now", + " ● 2 allocations will be disconnected (status unknown)", + "Preempted allocations", + " p1234567 batch.work[0] other", + }) + + at := func(text string) int { return slices.Index(words, text) } + + // The failures come first, as in the plan; the related ones last. + r.Less(at("Placement failures"), at("Planned for each group")) + r.Less(at("Planned for each group"), at("Preempted allocations")) + r.Less(at("Preempted allocations"), at("Related evaluations")) + + // The updates have the colours of the plan. + for _, painted := range lines { + if painted.text == " ● 2 allocations will be disconnected (status unknown)" { + r.Equal(opening(styleDestructive), opening(*painted.style)) + } + } +} + +func TestEvaluation_FromTheListShowsWhatItPlanned(t *testing.T) { + r := require.New(t) + + planned := plannedEvaluation() + listed := []nomad.Evaluation{planned.Evaluation} + client := &fakeClient{evaluations: listed, evaluation: planned} + + m := newTestModel(client) + m, _ = m.update(key(':')) + m = typeIn(m, "evals") + m, _ = m.update(enter()) + m, _ = m.update(evaluationsMsg(listed)) + + m, cmd := m.update(enter()) + m = drain(m, cmd) + + page := strings.Join(m.text.lines, "\n") + r.Contains(page, "Planned for each group") + r.Contains(page, " ● 1 allocation will be rescheduled now") + r.Contains(page, " p1234567 batch.work[0] other") +} + // waiting are the jobs of the tests with one allocation of web that waits // for a place. func waiting() []nomad.Job { diff --git a/internal/ui/hints_test.go b/internal/ui/hints_test.go index f6db571..4d3db48 100644 --- a/internal/ui/hints_test.go +++ b/internal/ui/hints_test.go @@ -10,6 +10,7 @@ import ( "slices" "strings" "testing" + "time" tea "charm.land/bubbletea/v2" "github.com/stretchr/testify/require" @@ -62,6 +63,7 @@ func everyScreenClient() *fakeClient { nodeAllocs: clientAllocs(), nodeDetail: clientDetail(), nodeMeta: clientMeta(), + identity: &nomad.NodeIdentity{Expires: time.Now().Add(time.Hour), Claims: "{}"}, versions: threeVersions(), variables: []nomad.Variable{{Path: "nomad/jobs/web", Namespace: "production"}}, nodePools: []nomad.NodePool{{Name: "default"}}, diff --git a/internal/ui/identity.go b/internal/ui/identity.go new file mode 100644 index 0000000..8e3c7f7 --- /dev/null +++ b/internal/ui/identity.go @@ -0,0 +1,80 @@ +package ui + +import ( + "context" + "errors" + + tea "charm.land/bubbletea/v2" + + "github.com/ingvarch/urga/internal/nomad" +) + +// identityFrom is the first Nomad that gives a client an identity. +const identityFrom = "1.11.0" + +// identityMsg is the identity of a client, nil when it has none. It carries +// the id of the client it was requested for. +type identityMsg struct { + nodeID string + identity *nomad.NodeIdentity +} + +// hasIdentity says the client is new enough to have an identity. An older +// one answers an error when asked, so it is not asked, and neither is one +// whose version is not read yet. +func hasIdentity(node nomad.Node) bool { return nomad.AtLeast(node.Version, identityFrom) } + +// fetchIdentity reads the identity of a client. A cluster with none to read +// is not an error. +func fetchIdentity(client nodesClient, nodeID string) tea.Cmd { + return request(func(ctx context.Context) (*nomad.NodeIdentity, error) { + identity, err := client.NodeIdentity(ctx, nodeID) + if errors.Is(err, nomad.ErrNoIdentity) { + return nil, nil + } + + if err != nil { + return nil, err + } + + return &identity, nil + }, func(identity *nomad.NodeIdentity) tea.Msg { return identityMsg{nodeID: nodeID, identity: identity} }) +} + +// identityText is what the panel says of the identity: when it expires, or +// nothing when it has no expiry. +func identityText(identity *nomad.NodeIdentity) string { + if identity == nil || identity.Expires.IsZero() { + return "" + } + + left := expiresIn(identity.Expires) + if left == "expired" { + return left + } + + return "expires " + left +} + +func holdsIdentity(p clientPage, _ env) bool { return p.host.identity != nil } + +// showIdentity opens the claims the page holds. +func showIdentity(p clientPage, _ env) (clientPage, outcome) { + return p, then(describeMsg{ + label: sprintf("Identity (Client: %s)", p.name), + content: p.host.identity.Claims, + }) +} + +// renewIdentity asks the client to get a new identity, after the user +// confirms. +func renewIdentity(p clientPage, e env) (clientPage, outcome) { + client, nodeID := e.client, p.nodeID + + return p, then(askMsg{ + question: sprintf("Really renew the identity of the client %s?", p.name), + apply: act(sprintf("The client %s gets a new identity at its next heartbeat.", p.name), func(ctx context.Context) error { + return client.RenewNodeIdentity(ctx, nodeID) + }), + }) +} diff --git a/internal/ui/identity_test.go b/internal/ui/identity_test.go new file mode 100644 index 0000000..5792d56 --- /dev/null +++ b/internal/ui/identity_test.go @@ -0,0 +1,209 @@ +package ui + +import ( + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/ingvarch/urga/internal/nomad" +) + +const claimsText = "{\n \"nomad_node_id\": \"node-1\"\n}" + +// identityCluster is one client of the given version, with an identity that +// expires after left, or none when left is zero. +func identityCluster(version string, left time.Duration) *fakeClient { + nodes := busyClient() + nodes[0].Version = version + + client := &fakeClient{ + nodes: nodes, + nodeAllocs: clientAllocs(), + use: map[string]nomad.ResourceUse{"node-1": {}}, + } + + if left != 0 { + client.identity = &nomad.NodeIdentity{Expires: time.Now().Add(left), Claims: claimsText} + } + + return client +} + +func onIdentity(t *testing.T, client *fakeClient) Model { + t.Helper() + + m, _ := nodeModelOf(client) + m, cmd := m.update(enter()) + + return drain(m, cmd) +} + +func TestClient_AskedForItsIdentityOnEachPollFromNomad111(t *testing.T) { + r := require.New(t) + + client := identityCluster("1.11.0", time.Hour) + m := onIdentity(t, client) + + // The list said the version already: the open and the machine's answer + // ask once. + r.Equal([]string{"node-1"}, client.identityAsked) + + drain(m, m.fetch()) + + r.Equal([]string{"node-1", "node-1"}, client.identityAsked) +} + +func TestClient_AskedForItsIdentityOnceTheMachineShowsAVersionThatQualifies(t *testing.T) { + r := require.New(t) + + client := identityCluster("1.11.0", time.Hour) + + page := clientPage{nodeID: "node-1", name: "nomad-server-01"} + e := newTestModel(client).env() + + // A version not read yet asks nothing. + drain(newTestModel(client), page.fetch(e)) + r.Empty(client.identityAsked) + + _, out, _ := page.take(hostMsg(nomad.Node{ID: "node-1", Version: "1.11.0"}), e) + r.NotNil(out.cmd) + + msg, ok := out.cmd().(identityMsg) + r.True(ok) + r.Equal("node-1", msg.nodeID) + r.Equal([]string{"node-1"}, client.identityAsked) + + // A machine that is already known to qualify is not asked a second + // time for the same answer. + known := clientPage{nodeID: "node-1", host: hostModel{node: nomad.Node{ID: "node-1", Version: "1.11.0"}}} + _, out, _ = known.take(hostMsg(nomad.Node{ID: "node-1", Version: "1.11.0"}), e) + r.Nil(out.cmd) +} + +func TestClient_NeverAsksAnOlderClientForItsIdentity(t *testing.T) { + r := require.New(t) + + client := identityCluster("1.10.5", time.Hour) + m := onIdentity(t, client) + m = drain(m, m.fetch()) + + r.Empty(client.identityAsked) + r.NotContains(bodyOf(m), "Identity") + r.NotContains(m.flash.text, nomad.ErrNoIdentity.Error()) +} + +func TestClient_ShowsWhenItsIdentityExpires(t *testing.T) { + r := require.New(t) + + m := onIdentity(t, identityCluster("1.11.1", 23*time.Hour+30*time.Minute)) + + r.Contains(bodyOf(m), "Identity expires in 23h") +} + +func TestClient_ShowsAnIdentityThatExpiredAsExpired(t *testing.T) { + r := require.New(t) + + m := onIdentity(t, identityCluster("1.11.1", -time.Hour)) + + r.Contains(bodyOf(m), "Identity expired") + r.NotContains(bodyOf(m), "expires expired") +} + +func TestClient_WithoutAnIdentityShowsNothingOfIt(t *testing.T) { + r := require.New(t) + + m := onIdentity(t, identityCluster("1.11.1", 0)) + m = drain(m, m.fetch()) + + r.NotContains(bodyOf(m), "Identity") + r.NotContains(m.hints(), hint{Key: "", Description: "Identity"}) + r.NotContains(m.hints(), hint{Key: "", Description: "Renew"}) + r.NotContains(m.flash.text, nomad.ErrNoIdentity.Error()) + + // An identity without an expiry has no field either, and still has keys. + client := identityCluster("1.11.1", 0) + client.identity = &nomad.NodeIdentity{Claims: claimsText} + m = onIdentity(t, client) + + r.NotContains(bodyOf(m), "Identity") + r.Contains(m.hints(), hint{Key: "", Description: "Identity"}) +} + +func TestClient_AnIdentityThatCannotBeReadShowsTheError(t *testing.T) { + r := require.New(t) + + client := identityCluster("1.11.1", 0) + m := onIdentity(t, client) + + client.identityErr = errTest + m = drain(m, m.fetch()) + + r.Contains(m.flash.text, errTest.Error()) + r.NotContains(bodyOf(m), "Identity") +} + +func TestClient_DropsTheIdentityOfAnotherNode(t *testing.T) { + r := require.New(t) + + m := onIdentity(t, identityCluster("1.11.1", 0)) + + m, _ = m.update(identityMsg{nodeID: "node-2", identity: &nomad.NodeIdentity{Expires: time.Now().Add(time.Hour)}}) + + r.NotContains(bodyOf(m), "Identity") + + m, _ = m.update(identityMsg{nodeID: "node-1", identity: &nomad.NodeIdentity{Expires: time.Now().Add(time.Hour)}}) + + r.Contains(bodyOf(m), "Identity expires in") +} + +func TestClient_IdentityShowsItsClaims(t *testing.T) { + r := require.New(t) + + client := identityCluster("1.11.1", time.Hour) + m := onIdentity(t, client) + asked := len(client.identityAsked) + + m, cmd := m.update(key('i')) + + r.Nil(cmd) + r.Len(client.identityAsked, asked, "the page holds the claims") + r.Contains(plain(m.render()), "Identity (Client: nomad-server-01)") + r.Contains(plain(m.render()), `"nomad_node_id": "node-1"`) +} + +func TestClient_RenewAsksAndSendsOnYes(t *testing.T) { + r := require.New(t) + + client := identityCluster("1.11.1", time.Hour) + m := onIdentity(t, client) + + m, _ = m.update(ctrlKey('n')) + + r.Equal(overlayConfirm, m.overlay) + r.Contains(plain(m.render()), "Really renew the identity of the client nomad-server-01?") + r.Empty(client.renewed) + + m = confirmed(t, m) + + r.Equal([]string{"node-1"}, client.renewed) + r.Contains(m.flash.text, "The client nomad-server-01 gets a new identity at its next heartbeat.") +} + +func TestClient_ReadOnlyOffersIdentityAndNotRenew(t *testing.T) { + r := require.New(t) + + client := identityCluster("1.11.1", time.Hour) + m, _ := nodeModelOf(client) + m.opts.ReadOnly = true + m, cmd := m.update(enter()) + m = drain(m, cmd) + + r.Contains(m.hints(), hint{Key: "", Description: "Identity"}) + r.NotContains(m.hints(), hint{Key: "", Description: "Renew"}) + + m, cmd = m.update(ctrlKey('n')) + r.Nil(cmd) + r.NotEqual(overlayConfirm, m.overlay) + r.Empty(client.renewed) +} diff --git a/internal/ui/limits_test.go b/internal/ui/limits_test.go new file mode 100644 index 0000000..353d182 --- /dev/null +++ b/internal/ui/limits_test.go @@ -0,0 +1,182 @@ +package ui + +import ( + "strings" + "testing" + + tea "charm.land/bubbletea/v2" + "github.com/stretchr/testify/require" + + "github.com/ingvarch/urga/internal/nomad" +) + +func TestIsFull(t *testing.T) { + for _, tc := range []struct { + node nomad.Node + full bool + }{ + {nomad.Node{MaxAllocs: 3, Allocs: 3}, true}, + {nomad.Node{MaxAllocs: 3, Allocs: 4}, true}, + {nomad.Node{MaxAllocs: 3, Allocs: 2}, false}, + {nomad.Node{Allocs: 2}, false}, + } { + require.Equal(t, tc.full, isFull(tc.node), "%+v", tc.node) + } +} + +func TestNodeRows_AFullClientTakesNoNewWorkAndIsPaintedLikeOneThatDoesNot(t *testing.T) { + r := require.New(t) + + ready := nomad.Node{Status: "ready", Eligibility: "eligible"} + + full, below, none := ready, ready, ready + full.MaxAllocs, full.Allocs = 3, 3 + below.MaxAllocs, below.Allocs = 3, 2 + + rows := nodeRows([]nomad.Node{full, below, none}, nil) + + r.Equal(colorAttention, rows[0].color) + r.Nil(rows[1].color) + r.Nil(rows[2].color) +} + +func TestNodes_TheListAsksForTheLimitsOfWhatItReads(t *testing.T) { + r := require.New(t) + + full := nomad.Node{ID: "node-1", Name: "server-01", Status: "ready", Eligibility: "eligible", MaxAllocs: 2, Allocs: 2} + client := &fakeClient{nodes: []nomad.Node{full}} + + m, cmd := runLine(newTestModel(client), "nodes") + m = drain(m, cmd) + + r.Equal([]string{""}, client.lists["limits"]) + r.Equal(colorAttention, m.list.table.rows[0].color) +} + +func TestNodes_ThePoolListAsksForLimitsToo(t *testing.T) { + r := require.New(t) + + client := &fakeClient{poolNodes: []nomad.Node{{ID: "node-1", Status: "ready", Eligibility: "eligible"}}} + + cmd := nodesPage{pool: "default"}.fetch(newTestModel(client).env()) + drain(newTestModel(client), cmd) + + r.Equal([]string{""}, client.lists["limits"]) +} + +// limitedCluster holds one client with an allocation limit, and allocs on it. +func limitedCluster(limit int, allocs []nomad.Alloc) *fakeClient { + nodes := busyClient() + nodes[0].MaxAllocs = limit + + return &fakeClient{nodes: nodes, nodeAllocs: allocs, use: map[string]nomad.ResourceUse{"node-1": {}}} +} + +func limitedClient(t *testing.T, limit int, allocs []nomad.Alloc) Model { + t.Helper() + + m, _ := nodeModelOf(limitedCluster(limit, allocs)) + m, cmd := m.update(enter()) + + return drain(m, cmd) +} + +func TestClient_DoesNotCountAllocationsItHasNotRead(t *testing.T) { + r := require.New(t) + + // The list carries the limit, and the allocations are not read yet. + m, client := nodeModelOf(limitedCluster(3, clientAllocs())) + m, _ = m.update(enter()) + r.IsType(clientPage{}, m.screen.page) + + r.NotContains(bodyOf(m), "Allocations 0 of 3") + r.Contains(bodyOf(m), "Allocations n/a of 3") + + // Nor when the cluster does not answer for them. + client.allocsErr = errTest + m = drain(m, m.fetch()) + + r.Contains(m.flash.text, errTest.Error()) + r.NotContains(bodyOf(m), "Allocations 0 of 3") + r.Contains(bodyOf(m), "Allocations n/a of 3") +} + +func TestNodes_AListThatCannotBeReadShowsTheErrorAndAsksForNoLimits(t *testing.T) { + r := require.New(t) + + m, client := nodeModel(t, readyNode()) + client.nodesErr = errTest + + m = drain(m, m.fetch()) + + r.Contains(m.flash.text, errTest.Error()) + r.Empty(client.lists["limits"]) +} + +func TestClient_ShowsHowManyAllocationsItHoldsOfItsLimit(t *testing.T) { + r := require.New(t) + + running, ended := clientAllocs()[0], clientAllocs()[0] + ended.Status = "complete" + + stopped := running + stopped.DesiredStatus = "stop" + + m := limitedClient(t, 3, []nomad.Alloc{running, running, ended, stopped}) + + r.Contains(bodyOf(m), "Allocations 2 of 3") +} + +func TestClient_WithoutALimitShowsNoAllocationCount(t *testing.T) { + m, _ := onClient(t) + + require.NotContains(t, bodyOf(m), "Allocations ") +} + +func TestClient_ThePanelOfALimitedClientCountsItsExtraRow(t *testing.T) { + r := require.New(t) + + m := limitedClient(t, 3, clientAllocs()) + m, _ = m.update(reading(29)) + + // The charts get what the extra row leaves, so the tall ones need a row + // more than without a limit, and a box too small for the two rows and + // one more draws no panel. + for _, tc := range []struct { + width, box, panel int + charts bool + }{ + {width: 120, box: 40, panel: 19, charts: true}, + {width: 120, box: 25, panel: 19, charts: true}, + {width: 120, box: 24, panel: 13, charts: true}, + {width: 120, box: 19, panel: 13, charts: true}, + {width: 120, box: 18, panel: 3}, + {width: 120, box: 9, panel: 3}, + {width: 120, box: 8, panel: 0}, + {width: 120, box: 3, panel: 0}, + {width: 20, box: 40, panel: 3}, + } { + sized, _ := m.update(tea.WindowSizeMsg{Width: tc.width, Height: tc.box + screenPadTop + headerHeight + statusHeight}) + _, content := sized.body(sized.innerWidth()) + body := plain(content) + + r.Equal(tc.panel, sized.panelHeight(), "%dx%d", tc.width, tc.box) + r.Equal(tc.panel, drawnPanel(sized), "%dx%d", tc.width, tc.box) + r.Equal(max(tc.box-3-tc.panel, 1), sized.list.table.height, "%dx%d", tc.width, tc.box) + + // A screen this narrow cuts the row. + if tc.width == 120 { + r.Equal(tc.panel > 0, strings.Contains(body, "Allocations 1 of 3"), "%dx%d", tc.width, tc.box) + } + + r.Equal(tc.charts, strings.Contains(body, "50%"), "%dx%d", tc.width, tc.box) + } +} + +func TestOverview_DoesNotAskForLimits(t *testing.T) { + client := &fakeClient{} + + overviewAnswered(client) + + require.Empty(t, client.lists["limits"]) +} diff --git a/internal/ui/navigation_test.go b/internal/ui/navigation_test.go index ca9273d..088d439 100644 --- a/internal/ui/navigation_test.go +++ b/internal/ui/navigation_test.go @@ -285,7 +285,7 @@ func TestAllocations_EachListAnswersItsOwnKeys(t *testing.T) { "job": allocKeys, // The keys of the node first, then those of its allocations. - "client": slices.Concat([]string{"e Events", "ctrl+d Drivers", "ctrl+h Host Volumes", "a Attributes", "m Meta", "u Units"}, allocKeys), + "client": slices.Concat([]string{"e Events", "ctrl+d Drivers", "ctrl+h Host Volumes", "a Attributes", "m Meta", "u Units", "i Identity", "ctrl+n Renew"}, allocKeys), // The deployment first, then what it placed. "deployment": slices.Concat([]string{ diff --git a/internal/ui/node.go b/internal/ui/node.go index 387d7e9..0c962ed 100644 --- a/internal/ui/node.go +++ b/internal/ui/node.go @@ -41,16 +41,28 @@ func (nodesPage) titles() []string { return nodeTitles } func (nodesPage) topics() []string { return []string{nomad.TopicNode} } // fetch reads the clients, of the pool when the page has one: the cluster -// says which clients a pool holds. +// says which clients a pool holds. The list does not carry the allocation +// limit of a client, so the clients are read again for it. func (p nodesPage) fetch(e env) tea.Cmd { client, pool := e.client, p.pool return fetchList(func(ctx context.Context) ([]nomad.Node, error) { + var ( + nodes []nomad.Node + err error + ) + if pool != "" { - return client.PoolNodes(ctx, pool) + nodes, err = client.PoolNodes(ctx, pool) + } else { + nodes, err = client.Nodes(ctx) } - return client.Nodes(ctx) + if err != nil { + return nil, err + } + + return client.WithAllocLimits(ctx, nodes) }, func(items []nomad.Node) tea.Msg { return nodesMsg(items) }) } @@ -274,12 +286,16 @@ func nodeRows(nodes []nomad.Node, usage map[string]nomad.ResourceUse) []tableRow return rows } -// nodeColor marks a node that takes no work: down, draining or ineligible. +// isFull says the client holds as many allocations as its limit allows. +func isFull(n nomad.Node) bool { return n.MaxAllocs > 0 && n.Allocs >= n.MaxAllocs } + +// nodeColor marks a node that takes no work: down, draining, ineligible or +// full. func nodeColor(n nomad.Node) color.Color { switch { case n.Status != "ready": return colorDead - case n.Drain, n.Eligibility != "eligible": + case n.Drain, n.Eligibility != "eligible", isFull(n): return colorAttention } diff --git a/internal/ui/plan.go b/internal/ui/plan.go index 558435a..6a091d8 100644 --- a/internal/ui/plan.go +++ b/internal/ui/plan.go @@ -386,14 +386,7 @@ func planText(plan nomad.Plan, revert bool) []paintedLine { lines = append(lines, paintedLine{text: " No changes detected", style: &styleMuted}) } - for i, group := range plan.Groups { - if i > 0 { - lines = append(lines, paintedLine{}) - } - - lines = append(lines, paintedLine{text: fmt.Sprintf(" Task group %q", group.Name)}) - lines = append(lines, groupUpdates(group)...) - } + lines = append(lines, groupLines(plan.Groups)...) lines = append(lines, paintedLine{}, paintedLine{text: "Changes"}, paintedLine{}) @@ -404,6 +397,23 @@ func planText(plan nomad.Plan, revert bool) []paintedLine { return append(lines, diffLines(plan.Diff)...) } +// groupLines say what the scheduler would do to each group, a blank line +// between the groups. +func groupLines(groups []nomad.PlanGroup) []paintedLine { + lines := []paintedLine{} + + for i, group := range groups { + if i > 0 { + lines = append(lines, paintedLine{}) + } + + lines = append(lines, paintedLine{text: fmt.Sprintf(" Task group %q", group.Name)}) + lines = append(lines, groupUpdates(group)...) + } + + return lines +} + // planVerb is what sending a plan does to the job. func planVerb(revert bool) string { if revert { @@ -428,6 +438,10 @@ func groupUpdates(group nomad.PlanGroup) []paintedLine { {group.Destructive, plural(group.Destructive, "allocation") + " will be recreated (destructive update)", &styleDestructive}, {group.Canary, plural(group.Canary, "canary allocation") + " will be deployed", &styleCanary}, {group.Preemptions, preempted(group.Preemptions), &styleDeleted}, + {group.RescheduleNow, plural(group.RescheduleNow, "allocation") + " will be rescheduled now", &styleTitle}, + {group.RescheduleLater, plural(group.RescheduleLater, "allocation") + " will be rescheduled after a delay", &stylePending}, + {group.Disconnect, plural(group.Disconnect, "allocation") + " will be disconnected (status unknown)", &styleDestructive}, + {group.Reconnect, plural(group.Reconnect, "allocation") + " will be reconnected", &styleAdded}, } lines := []paintedLine{} diff --git a/internal/ui/plan_test.go b/internal/ui/plan_test.go index eacafe1..4fa9e6c 100644 --- a/internal/ui/plan_test.go +++ b/internal/ui/plan_test.go @@ -84,15 +84,7 @@ func TestPlanText(t *testing.T) { }, texts(text)) // Each part in the colour of what it says. - style := func(line string) string { - for _, painted := range text { - if painted.text == line && painted.style != nil { - return opening(*painted.style) - } - } - - return "" - } + style := func(line string) string { return styleOf(text, line) } r.Equal(opening(styleError), style("Placement failures")) r.Equal(opening(styleWarn), style("Warnings")) @@ -123,7 +115,8 @@ func TestPlanText_EveryUpdate(t *testing.T) { r := require.New(t) text := planText(nomad.Plan{Groups: []nomad.PlanGroup{ - {Name: "web", Place: 1, Stop: 2, Migrate: 1, InPlace: 3, Preemptions: 1}, + {Name: "web", Place: 1, Stop: 2, Migrate: 1, InPlace: 3, Preemptions: 1, + RescheduleNow: 1, RescheduleLater: 2, Disconnect: 3, Reconnect: 1}, }}, false) r.Subset(texts(text), []string{ @@ -132,9 +125,40 @@ func TestPlanText_EveryUpdate(t *testing.T) { " ● 1 allocation will be migrated to another node", " ● 3 allocations will be updated in-place (no restart)", " ● 1 allocation of another job will be preempted", + " ● 1 allocation will be rescheduled now", + " ● 2 allocations will be rescheduled after a delay", + " ● 3 allocations will be disconnected (status unknown)", + " ● 1 allocation will be reconnected", }) } +func TestPlanText_RescheduleDisconnectReconnectColours(t *testing.T) { + r := require.New(t) + + text := planText(nomad.Plan{Groups: []nomad.PlanGroup{ + {Name: "web", RescheduleNow: 1, RescheduleLater: 1, Disconnect: 1, Reconnect: 1}, + }}, false) + + style := func(line string) string { return styleOf(text, line) } + + r.Equal(opening(styleTitle), style(" ● 1 allocation will be rescheduled now")) + r.Equal(opening(stylePending), style(" ● 1 allocation will be rescheduled after a delay")) + r.Equal(opening(styleDestructive), style(" ● 1 allocation will be disconnected (status unknown)")) + r.Equal(opening(styleAdded), style(" ● 1 allocation will be reconnected")) +} + +// styleOf is the opening of the style the line of text is painted in, or +// nothing. +func styleOf(text []paintedLine, line string) string { + for _, painted := range text { + if painted.text == line && painted.style != nil { + return opening(*painted.style) + } + } + + return "" +} + func TestPlanText_NothingChanges(t *testing.T) { r := require.New(t) diff --git a/internal/ui/roles.go b/internal/ui/roles.go index 6666efa..fae8b87 100644 --- a/internal/ui/roles.go +++ b/internal/ui/roles.go @@ -124,11 +124,14 @@ type ( // they are grouped in. nodesClient interface { Nodes(ctx context.Context) ([]nomad.Node, error) + WithAllocLimits(ctx context.Context, nodes []nomad.Node) ([]nomad.Node, error) Node(ctx context.Context, nodeID string) (nomad.Node, error) NodeDetail(ctx context.Context, nodeID string) (nomad.NodeDetail, error) NodeMeta(ctx context.Context, nodeID string) ([]nomad.MetaEntry, error) NodeMetaSpec(ctx context.Context, nodeID string) (string, error) SubmitNodeMeta(ctx context.Context, nodeID, source string) error + NodeIdentity(ctx context.Context, nodeID string) (nomad.NodeIdentity, error) + RenewNodeIdentity(ctx context.Context, nodeID string) error NodeUsage(ctx context.Context, nodeID string) (nomad.ResourceUse, error) DrainNode(ctx context.Context, nodeID string, drain bool) error SetNodeEligible(ctx context.Context, nodeID string, eligible bool) error diff --git a/internal/ui/writes_test.go b/internal/ui/writes_test.go index d3f44cc..b86b319 100644 --- a/internal/ui/writes_test.go +++ b/internal/ui/writes_test.go @@ -21,7 +21,7 @@ var ( "SubmitJob", "SubmitNamespace", "SubmitNodeMeta", "SubmitVariable", "StartJob", "StopJob", "LaunchJob", "RevertJobTo", "ScaleJob", "RestartAllocation", "RestartAllTasks", "StopAllocation", "RestartTask", "SignalTask", "PromoteGroups", "PauseDeployment", "DeleteServiceRegistration", - "DrainNode", "SetNodeEligible", + "DrainNode", "SetNodeEligible", "RenewNodeIdentity", "PromoteDeployment", "FailDeployment", "DetachVolume", "ReleaseClaim", "DispatchJob", "EvaluateJob", "RescheduleFailed", "TagVersion", "UntagVersion", "MarkStable", "SetAllocHealth", "ReleaseLock", @@ -36,7 +36,7 @@ var ( "DescribeJob", "DescribeAllocation", "DescribeDeployment", "DescribeService", "Allocations", "NodeAllocations", "Allocation", "Logs", "LogBefore", "Usage", "AllocationUsage", "TaskUsage", "NodeUsage", - "Node", "NodeDetail", "NodeMeta", "NodeMetaSpec", "Nodes", "NodePools", "PoolJobs", "PoolNodes", + "Node", "NodeIdentity", "WithAllocLimits", "NodeDetail", "NodeMeta", "NodeMetaSpec", "Nodes", "NodePools", "PoolJobs", "PoolNodes", "NamespaceSpec", "Namespaces", "Deployments", "Services", "Evaluations", "Evaluation", "EvaluationAllocations", "FailedPlacement", "NewestEvaluation", "Variables", "Variable", "VariableSpec", "Servers", "Server", "RaftPeers", "Events", "Volumes", "Volume", "DescribeVolume", "Plugins", "Plugin", "DescribePlugin",