feat(ui): show what newer clusters report: reschedules in a plan, allocation limits and identities of clients

README.md

@@ -45,17 +45,19 @@ One binary, no config to write, no browser.
 - Creates, edits and deletes namespaces.
 - Shows the versions of a job, reverts to any of them, tags them, and marks them stable or unstable.
 - Explains why a job is not placed, has the scheduler evaluate it again, and
-  places its failed allocations again without waiting for their delay. Opens
-  the allocations an evaluation placed or changed.
+  places its failed allocations again without waiting for their delay. Shows
+  what an evaluation planned for each group, and opens the allocations it
+  placed or changed.
 - Lists the launches of periodic and parameterized jobs, shows when a
   periodic job runs next, runs it now, and dispatches a parameterized job
   with its meta and payload.
 - Starts, stops, scales and restarts jobs and allocations, restarts every
   task of an allocation with its prestart and sidecar tasks, restarts and
   signals single tasks, and opens a shell in a task.
-- Drains clients and shows their CPU and memory over time. Purges clients
-  that are down, stopped jobs, and collects the garbage of a client or of
-  the whole cluster.
+- Drains clients, shows their CPU and memory over time, marks the ones that
+  are full, shows when the identity of a client expires and renews it.
+  Purges clients that are down, stopped jobs, and collects the garbage of a
+  client or of the whole cluster.
 - Shows the health of the servers and how many the cluster can lose.
 - Shows how the scheduler places work, and edits it with check-and-set.
 - Lists the scaling policies and shows what the autoscaler did with the

docs/clients.md

@@ -11,6 +11,13 @@ variables and node pools. For the keys of each screen, see
 status, eligibility, drain state, CPU and memory use, and address of each
 client.
 
+A client that holds as many allocations as its limit allows (`node_max_allocs`,
+Nomad 1.10.2 and later) is full: the scheduler places nothing more on it, and
+its row has the attention color, like a client that is draining or
+ineligible. With the event stream on, the list shows a client that fills up
+or frees a place within 30 seconds: Nomad streams a change of the client,
+not of the allocations placed on it.
+
 - `ctrl-d` drains a client, or stops draining it, after you confirm.
 - `i` marks a client as eligible or ineligible for new work.
 - `ctrl-p` purges a client that is down, after you confirm: the cluster
@@ -28,6 +35,11 @@ With several clients marked, these keys act on all of them that fit:
 
 - its status, address, datacenter, node pool, version and whether it takes
   new work;
+- when the client has a limit, how many allocations it holds of it, like
+  `3 of 10`. Allocations that ended or are being stopped do not count;
+- when the client has an identity, when it expires, like `Identity expires in
+  23h` (Nomad 1.11 and later). On an older cluster, or an older client, none
+  of it shows;
 - charts of its CPU and memory use, read every 5 seconds while the screen is
   open. The charts keep the last 20 minutes. `u` reads them in percents or
   in numbers, scaled up to gigahertz and gibibytes.
@@ -42,7 +54,10 @@ From the client screen:
 - `ctrl-h` lists its host volumes;
 - `a` shows its attributes;
 - `m` shows its metadata. `e` opens the metadata in your editor, and urga
-  saves it to the client when you close the editor.
+  saves it to the client when you close the editor;
+- `i` shows the claims of its identity;
+- `ctrl-n` renews its identity after you confirm, and the client asks for a
+  new one at its next heartbeat.
 
 ## Servers
 

docs/jobs.md

@@ -154,7 +154,10 @@ The plan shows, from top to bottom:
 2. **Warnings** from the cluster.
 3. **What will happen when you submit this job**: for each task group, how
    many allocations will be created, stopped, migrated, updated in place,
-   recreated or deployed as canaries, and how many stay unchanged.
+   recreated or deployed as canaries, rescheduled now or after a delay,
+   disconnected or reconnected, and how many stay unchanged. It also says how
+   many allocations of other jobs the plan preempts. Rescheduled, disconnected
+   and reconnected allocations need Nomad 1.11.
 4. **Changes**: the difference between the running job and your file, laid out
    like the job file. Removed lines are red and start with `-`, added lines
    are green and start with `+`.
@@ -220,7 +223,9 @@ nothing of a system job.
 
 `:evaluations` lists the evaluations of the namespace. `enter` opens one: its
 status, what triggered it, related evaluations, and the placement failures, if
-there are any. `a`, on the list or on the evaluation, opens the allocations it
+there are any. It also shows what the evaluation planned for each group, in
+the words of the plan, and the allocations of other jobs it preempted (Nomad
+1.11). `a`, on the list or on the evaluation, opens the allocations it
 placed or changed.
 
 ## Deployments

docs/keys.md

@@ -473,6 +473,8 @@ The client screen lists the allocations of the client. It has the keys of
 | `a` | Attributes | |
 | `m` | Metadata | |
 | `u` | Units: percents or numbers on the charts | |
+| `i` | Claims of the identity of the client. Shown when it has one. | |
+| `ctrl-n` | Renew the identity of the client. Shown when it has one. | yes |
 
 On drivers, `enter` opens the details of a driver. On attributes, metadata,
 driver details and a server, `c` copies the value under the cursor. On

internal/nomad/alloc.go

@@ -125,6 +125,24 @@ func allocsOf(stubs []*api.AllocationListStub) []Alloc {
 	return out
 }
 
+// HoldsPlace says the allocation counts against the limit of its client: the
+// scheduler counts what is meant to run and has not ended.
+func (a Alloc) HoldsPlace() bool { return holdsPlace(a.DesiredStatus, a.Status) }
+
+func holdsPlace(desired, status string) bool {
+	switch desired {
+	case api.AllocDesiredStatusStop, api.AllocDesiredStatusEvict:
+		return false
+	}
+
+	switch status {
+	case api.AllocClientStatusComplete, api.AllocClientStatusFailed, api.AllocClientStatusLost:
+		return false
+	}
+
+	return true
+}
+
 // NodeAllocations lists what one machine of the cluster runs, whichever
 // namespace the work belongs to.
 func (c *Client) NodeAllocations(ctx context.Context, nodeID string) ([]Alloc, error) {

internal/nomad/evaluation.go

@@ -30,6 +30,12 @@ type EvaluationDetail struct {
 
 	// Failures are the task groups it could not place, by name.
 	Failures []PlacementFailure
+
+	// Groups are what it planned for each group, and Preempted the
+	// allocations of other jobs it preempted. A cluster older than Nomad
+	// 1.11 sends neither.
+	Groups    []PlanGroup
+	Preempted []Alloc
 }
 
 // PlacementFailure is why the allocations of one task group were not placed:
@@ -78,6 +84,11 @@ func newEvaluationDetail(eval *api.Evaluation) EvaluationDetail {
 		Failures:          newPlacementFailures(eval.FailedTGAllocs),
 	}
 
+	if eval.PlanAnnotations != nil {
+		detail.Groups = planGroups(eval.PlanAnnotations.DesiredTGUpdates)
+		detail.Preempted = allocsOf(eval.PlanAnnotations.PreemptedAllocs)
+	}
+
 	for _, stub := range eval.RelatedEvals {
 		if stub == nil {
 			continue

internal/nomad/evaluation_test.go

@@ -123,6 +123,49 @@ func TestFailedPlacement_WhenNoneFailed(t *testing.T) {
 	r.ErrorIs(err, nomad.ErrNoFailures)
 }
 
+func TestEvaluation_WhatItPlannedForEachGroup(t *testing.T) {
+	r := require.New(t)
+
+	client, _ := recorder(t, `{
+		"ID": "e1",
+		"PlanAnnotations": {
+			"DesiredTGUpdates": {
+				"web": {"Place": 2, "Disconnect": 1},
+				"api": {"Reconnect": 1, "RescheduleNow": 1}
+			},
+			"PreemptedAllocs": [
+				{"ID": "p1", "Name": "batch.work[0]", "Namespace": "other", "JobID": "batch", "TaskGroup": "work"}
+			]
+		}
+	}`)
+
+	eval, err := client.Evaluation(context.Background(), "production", "e1")
+	r.NoError(err)
+
+	r.Equal([]nomad.PlanGroup{
+		{Name: "api", Reconnect: 1, RescheduleNow: 1},
+		{Name: "web", Place: 2, Disconnect: 1},
+	}, eval.Groups)
+
+	r.Len(eval.Preempted, 1)
+	r.Equal("p1", eval.Preempted[0].ID)
+	r.Equal("batch.work[0]", eval.Preempted[0].Name)
+	r.Equal("other", eval.Preempted[0].Namespace)
+}
+
+func TestEvaluation_WithoutPlanAnnotations(t *testing.T) {
+	r := require.New(t)
+
+	// A cluster older than Nomad 1.11 sends none.
+	client, _ := recorder(t, `{"ID": "e1"}`)
+
+	eval, err := client.Evaluation(context.Background(), "production", "e1")
+	r.NoError(err)
+
+	r.Empty(eval.Groups)
+	r.Empty(eval.Preempted)
+}
+
 func TestEvaluationAllocations(t *testing.T) {
 	r := require.New(t)
 

internal/nomad/identity_test.go

@@ -0,0 +1,78 @@
+package nomad_test
+
+import (
+	"context"
+	"net/http"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+
+	"github.com/ingvarch/urga/internal/nomad"
+)
+
+const identityClaims = `{"Claims": {
+	"exp": 1792454400, "iat": 1792368000, "iss": "nomad",
+	"nomad_node_id": "node-1", "nomad_node_pool": "default"
+}}`
+
+func TestNodeIdentity_ReadsTheExpiryAndKeepsTheClaims(t *testing.T) {
+	r := require.New(t)
+
+	client, _, asked := metaServer(t, identityClaims)
+
+	identity, err := client.NodeIdentity(context.Background(), "node-1")
+	r.NoError(err)
+
+	r.Equal("/v1/client/identity", asked.URL.Path)
+	r.Equal("node-1", asked.URL.Query().Get("node_id"))
+	r.Empty(asked.URL.Query().Get("namespace"))
+
+	r.True(time.Unix(1792454400, 0).Equal(identity.Expires))
+	r.Contains(identity.Claims, `"nomad_node_id": "node-1"`)
+}
+
+func TestNodeIdentity_WithoutAnExpiryHasNone(t *testing.T) {
+	for _, body := range []string{
+		`{"Claims": {"iss": "nomad"}}`,
+		`{"Claims": {"exp": "soon"}}`,
+		`{"Claims": null}`,
+	} {
+		client, _, _ := metaServer(t, body)
+
+		identity, err := client.NodeIdentity(context.Background(), "node-1")
+		require.NoError(t, err, body)
+		require.True(t, identity.Expires.IsZero(), body)
+	}
+}
+
+func TestNodeIdentity_ANotFoundIsNoIdentity(t *testing.T) {
+	client := refusing(t, http.StatusNotFound, "")
+
+	_, err := client.NodeIdentity(context.Background(), "node-1")
+
+	require.ErrorIs(t, err, nomad.ErrNoIdentity)
+}
+
+func TestNodeIdentity_AnotherErrorIsPassedOn(t *testing.T) {
+	r := require.New(t)
+
+	client := refusing(t, http.StatusInternalServerError, "rpc error")
+
+	_, err := client.NodeIdentity(context.Background(), "node-1")
+
+	r.Error(err)
+	r.NotErrorIs(err, nomad.ErrNoIdentity)
+}
+
+func TestRenewNodeIdentity_PostsTheNode(t *testing.T) {
+	r := require.New(t)
+
+	client, sent, asked := metaServer(t, `{}`)
+
+	r.NoError(client.RenewNodeIdentity(context.Background(), "node-1"))
+
+	r.Equal(http.MethodPost, asked.Method)
+	r.Equal("/v1/client/identity/renew", asked.URL.Path)
+	r.JSONEq(`{"NodeID": "node-1"}`, string(*sent))
+}