diff --git a/README.md b/README.md index 25e7eb5..bcaac5e 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,8 @@ One binary, no config to write, no browser. - Watches the deploy of a job with `w`: the newest evaluation, the deployment and its allocations, until it ends. - Updates the screen when the cluster changes, using the Nomad event stream, - and shows the stream itself on `:events`. + and shows the stream itself on `:events`, for the namespace, or for one job + or allocation. - Counts on one screen what needs attention: dead jobs, failed allocations, tasks that restart or run out of memory, clients that are down or draining, blocked evaluations and failed deployments, and opens the list of each with diff --git a/docs/allocations.md b/docs/allocations.md index 596520c..9c059ea 100644 --- a/docs/allocations.md +++ b/docs/allocations.md @@ -28,6 +28,8 @@ A running allocation whose task was restarted or killed for its memory in the last hour is painted in the attention color, and `!` keeps it: that is a crash loop going on now. A restart of a month ago leaves the row as it is. +`o` opens the events of the allocation under the cursor. + `r` restarts the tasks of an allocation that run, `ctrl-r` restarts all of its tasks, the prestart and sidecar ones too, and `ctrl-k` stops it, after you confirm. With several allocations marked, they act on all of them. When you stop an @@ -61,7 +63,8 @@ From here: - `c` opens the client the allocation runs on; - `p` opens the previous allocation, `n` the next one; -- `f` opens the follow-up evaluation. +- `f` opens the follow-up evaluation; +- `o` opens the events of the allocation. ## Checks diff --git a/docs/configuration.md b/docs/configuration.md index 3188a5e..d674645 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -149,8 +149,10 @@ without saving. Most screens, such as jobs, allocations and clients, follow the Nomad event stream: urga reloads the screen when something on it changes. It also reloads every 30 seconds in case an event is missed. Screens that the event stream -does not cover, such as namespaces, variables and servers, reload every 2 -seconds. +does not cover, such as namespaces and servers, reload every 2 seconds. +Volumes and CSI plugins follow the stream on Nomad 1.10 and later, variables +on Nomad 2.0 and later. On an older cluster, or when urga cannot read the +version of the cluster, they are read every 2 seconds. If the event stream is not available, which usually means the ACL token does not allow it, urga reloads every screen every 2 seconds instead. The status diff --git a/docs/jobs.md b/docs/jobs.md index c777744..d3a8ef4 100644 --- a/docs/jobs.md +++ b/docs/jobs.md @@ -19,7 +19,8 @@ From the list: - `t` opens its task groups; - `d` shows the job as the cluster describes it; - `h` shows the job file it was submitted with; -- `v` opens its versions. +- `v` opens its versions; +- `o` shows the events of the job. ## Periodic and parameterized jobs diff --git a/docs/keys.md b/docs/keys.md index 4a70c09..a83137b 100644 --- a/docs/keys.md +++ b/docs/keys.md @@ -147,13 +147,19 @@ the events that report a failure. With the cursor on the top row, it stays there and shows each event as it arrives. Moved down, it stays on its event. `enter` opens what the event is -about: a job, an allocation, a deployment, an evaluation, a client or a -service. `esc` comes back to the events as they were, and the ones that -happened meanwhile are added on top. +about: a job, an allocation, a deployment, an evaluation, a client, a +service, a volume, a CSI plugin or a variable. `esc` comes back to the +events as they were, and the ones that happened meanwhile are added on top. -The list follows the namespace of the session. Events of clients belong to no -namespace and show in every one. In another namespace it starts again from -what the cluster keeps. +The list follows the namespace of the session. Events of clients and CSI +plugins belong to no namespace and show in every one. In another namespace it +starts again from what the cluster keeps. + +`o` on a job or an allocation opens the same screen with only the events of +it. The cluster sends only those, from what it still keeps. The title names +the job or the allocation, and the screen stays on it when you switch +namespace. These need only a token that can read the job, while the events +of the whole namespace need a management token. ## Sort @@ -216,6 +222,7 @@ start, stop, restart or drain then applies to every marked row. | `v` | Versions | | | `l` | Logs of a task in every allocation of the job. Not shown for a periodic or parameterized job. | | | `w` | Watch the deploy of the job at the version it runs. Not shown for a periodic or parameterized job. | | +| `o` | Events of the job, its allocations, evaluations and deployments. Not shown for a periodic or parameterized job. | | | `e` | Edit the job in your editor | yes | | `n` | New job: opens a job template in your editor | yes | | `ctrl-s` | Start a stopped job, or stop a running one | yes | @@ -281,6 +288,7 @@ On what was done to the count of a group, `d` describes its policy. | `ctrl-r` | Restart every task, the prestart and sidecar ones too | yes | | `ctrl-k` | Stop | yes | | `l` | Logs of a task in every allocation of the list | | +| `o` | Events of the allocation | | | `space`, `ctrl-a` | Mark | | ### Tasks @@ -290,6 +298,7 @@ On what was done to the count of a group, `d` describes its policy. | `enter` | Logs, stdout | | | `ctrl-e` | Logs, stderr | | | `e` | Events of the task | | +| `o` | Events of the allocation | | | `b` | Files of the task | | | `s` | Shell in the task | yes | | `r` | Restart the task. Shown when it is running. | yes | diff --git a/docs/storage.md b/docs/storage.md index ea66e25..c3a67a5 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -29,8 +29,9 @@ still being made (`pending`) is yellow. A CSI volume whose plugin has fewer healthy controllers or node plugins than expected has the attention color: attaching it may fail. `!` shows only those volumes. -The cluster reports no change of a CSI volume in its event stream, so the -storage screens are polled, not watched. +The storage screens follow the event stream on Nomad 1.10 and later. On an +older cluster, or when urga cannot read the version of the cluster, they are +read every 2 seconds. ## A volume diff --git a/internal/nomad/events.go b/internal/nomad/events.go index 7853ffa..05cc08c 100644 --- a/internal/nomad/events.go +++ b/internal/nomad/events.go @@ -2,6 +2,9 @@ package nomad import ( "context" + "encoding/json" + "regexp" + "strconv" "time" "github.com/hashicorp/nomad/api" @@ -16,8 +19,87 @@ const ( TopicNode = "Node" TopicNodePool = "NodePool" TopicService = "Service" + + // Nomad's api module has no constant for these. + TopicCSIVolume = "CSIVolume" + TopicCSIPlugin = "CSIPlugin" + TopicHostVolume = "HostVolume" + TopicVariable = "Variable" ) +// since is the first version of Nomad that sends the events of a topic. A +// topic not in it is older than any cluster urga reads. +var since = map[string]string{ + TopicCSIVolume: "1.10.0", + TopicCSIPlugin: "1.10.0", + TopicHostVolume: "1.10.0", + TopicVariable: "2.0.0", +} + +// Streams says a cluster of that version sends the events of every one of +// the topics. A cluster that does not know a topic opens the stream anyway +// and never sends anything for it, so the version is checked before it is +// asked. +func Streams(version string, topics []string) bool { + for _, topic := range topics { + if minimum, ok := since[topic]; ok && !AtLeast(version, minimum) { + return false + } + } + + return true +} + +var numbers = regexp.MustCompile(`^(\d+)\.(\d+)\.(\d+)`) + +// AtLeast says version reads as major.minor.patch at its start, and is at or +// above the minimum. What follows the patch (-beta.1, +ent) is not compared. +func AtLeast(version, minimum string) bool { + have, ok := triple(version) + if !ok { + return false + } + + need, ok := triple(minimum) + if !ok { + return false + } + + return have.compare(need) >= 0 +} + +type release [3]int + +func (a release) compare(b release) int { + for i := range a { + if a[i] != b[i] { + return a[i] - b[i] + } + } + + return 0 +} + +func triple(version string) (release, bool) { + parts := numbers.FindStringSubmatch(version) + if parts == nil { + return release{}, false + } + + var r release + + for i := range r { + n, err := strconv.Atoi(parts[i+1]) + if err != nil { + return release{}, false + } + + r[i] = n + } + + return r, true +} + // Change reports that something in the cluster changed. It carries no new // state: the screen reads that with its usual request. type Change struct { @@ -86,13 +168,36 @@ type Event struct { // Feed is what happens in the cluster, event by event. type Feed = Stream[Event] -// Feed follows every topic of the cluster in a namespace, from the event of -// an index on: 1 is the oldest the cluster still keeps. The caller closes it -// when it stops reading. -func (c *Client) Feed(ctx context.Context, namespace string, from uint64) (*Feed, error) { +// Subject is what a feed follows: one job, one allocation, or, when it is +// empty, the whole namespace. +type Subject struct{ JobID, AllocID string } + +// topics are what the cluster is asked for. An allocation, an evaluation and +// a deployment carry the ID of their job as a filter key, so the cluster +// sends the work of a job by the job's ID. Services are left out: their +// filter keys hold the service name, which another job's service can share. +func (s Subject) topics() map[api.Topic][]string { + switch { + case s.AllocID != "": + return map[api.Topic][]string{api.TopicAllocation: {s.AllocID}} + case s.JobID != "": + key := []string{s.JobID} + + return map[api.Topic][]string{ + api.TopicJob: key, api.TopicAllocation: key, api.TopicEvaluation: key, api.TopicDeployment: key, + } + } + + return map[api.Topic][]string{api.TopicAll: nil} +} + +// Feed follows the subject in a namespace, from the event of an index on: 1 +// is the oldest the cluster still keeps. The caller closes it when it stops +// reading. +func (c *Client) Feed(ctx context.Context, namespace string, subject Subject, from uint64) (*Feed, error) { // Asked for an index past its newest event, the cluster starts at that // event again: what came before the index was read already. - return follow(ctx, c, namespace, map[api.Topic][]string{api.TopicAll: nil}, from, func(event api.Event) (Event, bool) { + return follow(ctx, c, namespace, subject.topics(), from, func(event api.Event) (Event, bool) { return newEvent(event), event.Index >= from }) } @@ -133,11 +238,64 @@ func newEvent(e api.Event) Event { if sr, err := e.Service(); err == nil && sr != nil { event.Namespace, event.Name = sr.Namespace, sr.ServiceName } + default: + event.readStored(e) } return event } +// readStored names an event of a topic the api module cannot decode. +func (event *Event) readStored(e api.Event) { + switch e.Topic { + case TopicCSIVolume: + if v, ok := payloadOf(e, "Volume"); ok { + event.Namespace, event.Name, event.State, event.At = v.Namespace, v.Name, csiState(v.Schedulable), unixTime(v.ModifyTime) + } + case TopicHostVolume: + if v, ok := payloadOf(e, "Volume"); ok { + event.Namespace, event.Name, event.State, event.At = v.Namespace, v.Name, v.State, unixTime(v.ModifyTime) + + // A deleted volume has no key of its own. + if event.Key == "" { + event.Key = v.ID + } + } + case TopicCSIPlugin: + if v, ok := payloadOf(e, "Plugin"); ok { + event.Name, event.At = v.ID, unixTime(v.ModifyTime) + } + case TopicVariable: + if v, ok := payloadOf(e, "Metadata"); ok { + event.Namespace, event.Name, event.At = v.Namespace, v.Path, unixTime(v.ModifyTime) + } + } +} + +// stored is the fields urga reads of an object the api module cannot decode. +type stored struct { + ID, Name, Namespace, Path, State string + Schedulable bool + ModifyTime int64 +} + +// payloadOf reads the object an event carries under a key of its payload. +func payloadOf(e api.Event, key string) (stored, bool) { + raw, ok := e.Payload[key] + if !ok { + return stored{}, false + } + + data, err := json.Marshal(raw) + if err != nil { + return stored{}, false + } + + var v stored + + return v, json.Unmarshal(data, &v) == nil +} + // streamBuffer is how many events of a stream wait for the reader. const streamBuffer = 256 diff --git a/internal/nomad/events_test.go b/internal/nomad/events_test.go index d64a04e..25ae5eb 100644 --- a/internal/nomad/events_test.go +++ b/internal/nomad/events_test.go @@ -124,7 +124,7 @@ func TestFeed_ReadsWhatHappened(t *testing.T) { ]}`, ) - feed, err := client.Feed(context.Background(), "production", 1) + feed, err := client.Feed(context.Background(), "production", nomad.Subject{}, 1) r.NoError(err) defer feed.Close() @@ -174,7 +174,7 @@ func TestFeed_NamesEachKindOfObject(t *testing.T) { } { client, _ := eventServer(t, `{"Index": 3, "Events": [{"Topic": "`+topic+`", "Type": "Updated", "Key": "k1", "Index": 3, "Payload": `+want.payload+`}]}`) - feed, err := client.Feed(context.Background(), "", 1) + feed, err := client.Feed(context.Background(), "", nomad.Subject{}, 1) r.NoError(err, topic) event := waitForEvent(t, feed) @@ -202,7 +202,7 @@ func TestFeed_GoesOnFromAnIndex(t *testing.T) { client, asked := eventServer(t) - feed, err := client.Feed(context.Background(), "production", 58) + feed, err := client.Feed(context.Background(), "production", nomad.Subject{}, 58) r.NoError(err) defer feed.Close() @@ -211,6 +211,31 @@ func TestFeed_GoesOnFromAnIndex(t *testing.T) { r.Equal("58", asked.URL.Query().Get("index")) } +func TestFeed_AsksForTheTopicsOfItsSubject(t *testing.T) { + r := require.New(t) + + for name, want := range map[string]struct { + subject nomad.Subject + topics []string + }{ + "namespace": {nomad.Subject{}, []string{"*"}}, + "job": {nomad.Subject{JobID: "web"}, []string{"Job:web", "Allocation:web", "Evaluation:web", "Deployment:web"}}, + "allocation": {nomad.Subject{AllocID: "af1f37df"}, []string{"Allocation:af1f37df"}}, + } { + client, asked := eventServer(t) + + feed, err := client.Feed(context.Background(), "ml", want.subject, 7) + r.NoError(err, name) + + feed.Close() + + query := asked.URL.Query() + r.ElementsMatch(want.topics, query["topic"], name) + r.Equal("ml", query.Get("namespace"), name) + r.Equal("7", query.Get("index"), name) + } +} + func TestFeed_KeepsWhatArrivesWhileNobodyReads(t *testing.T) { r := require.New(t) @@ -220,7 +245,7 @@ func TestFeed_KeepsWhatArrivesWhileNobodyReads(t *testing.T) { `{"Index": 9, "Events": [{"Topic": "Job", "Type": "JobRegistered", "Key": "db", "Index": 9}]}`, ) - feed, err := client.Feed(context.Background(), "", 1) + feed, err := client.Feed(context.Background(), "", nomad.Subject{}, 1) r.NoError(err) defer feed.Close() @@ -240,7 +265,7 @@ func TestFeed_LeavesOutWhatCameBeforeTheIndex(t *testing.T) { `{"Index": 19, "Events": [{"Topic": "Job", "Type": "JobRegistered", "Key": "web", "Index": 19}]}`, ) - feed, err := client.Feed(context.Background(), "", 19) + feed, err := client.Feed(context.Background(), "", nomad.Subject{}, 19) r.NoError(err) defer feed.Close() @@ -280,3 +305,98 @@ func waitForChange(t *testing.T, changes *nomad.Changes) nomad.Change { return nomad.Change{} } + +func TestAtLeast(t *testing.T) { + for _, c := range []struct { + version, minimum string + want bool + }{ + {"1.10.0", "1.10.0", true}, + {"1.10.0-beta.1", "1.10.0", true}, + {"2.0.7+ent", "2.0.0", true}, + {"10.0.0", "2.0.0", true}, + {"1.11.3", "1.10.0", true}, + {"1.9.7", "1.10.0", false}, + {"1.11.4+ent", "2.0.0", false}, + {"1.10.0", "1.10.1", false}, + {"", "1.10.0", false}, + {"dev", "1.10.0", false}, + {"1.10", "1.10.0", false}, + {"v1.10.0", "1.10.0", false}, + } { + require.Equal(t, c.want, nomad.AtLeast(c.version, c.minimum), "%q >= %q", c.version, c.minimum) + } +} + +func TestStreams(t *testing.T) { + for _, c := range []struct { + version string + topics []string + want bool + }{ + {"", []string{nomad.TopicJob, nomad.TopicAllocation}, true}, + {"1.9.7", []string{nomad.TopicNode}, true}, + {"", []string{nomad.TopicCSIVolume}, false}, + {"dev", []string{nomad.TopicCSIVolume}, false}, + {"1.9.7", []string{nomad.TopicCSIVolume}, false}, + {"1.10.0", []string{nomad.TopicCSIVolume}, true}, + {"1.10.0-beta.1", []string{nomad.TopicHostVolume}, true}, + {"1.11.3", []string{nomad.TopicCSIVolume, nomad.TopicHostVolume, nomad.TopicCSIPlugin}, true}, + {"1.11.4+ent", []string{nomad.TopicVariable}, false}, + {"2.0.0", []string{nomad.TopicVariable}, true}, + {"10.0.0", []string{nomad.TopicVariable}, true}, + {"1.11.3", []string{nomad.TopicCSIPlugin, nomad.TopicVariable}, false}, + {"1.11.3", []string{nomad.TopicAllocation, nomad.TopicCSIVolume}, true}, + } { + require.Equal(t, c.want, nomad.Streams(c.version, c.topics), "%q %v", c.version, c.topics) + } +} + +func TestFeed_NamesVolumesPluginsAndVariables(t *testing.T) { + r := require.New(t) + + for _, c := range []struct { + name, topic, payload, key string + want nomad.Event + }{ + { + "csi volume", "CSIVolume", + `{"Volume": {"ID": "v1", "Name": "data", "Namespace": "ml", "Schedulable": false, "ModifyTime": 1790671900000000000}}`, "v1", + nomad.Event{Key: "v1", Name: "data", Namespace: "ml", State: "unschedulable"}, + }, + { + "host volume", "HostVolume", + `{"Volume": {"ID": "h1", "Name": "cache", "Namespace": "ml", "State": "ready", "ModifyTime": 1790671900000000000}}`, "h1", + nomad.Event{Key: "h1", Name: "cache", Namespace: "ml", State: "ready"}, + }, + { + "deleted host volume", "HostVolume", + `{"Volume": {"ID": "h2", "Name": "cache", "Namespace": "ml", "State": "ready", "ModifyTime": 1790671900000000000}}`, "", + nomad.Event{Key: "h2", Name: "cache", Namespace: "ml", State: "ready"}, + }, + { + "plugin", "CSIPlugin", + `{"Plugin": {"ID": "ebs", "ModifyTime": 1790671900000000000}}`, "ebs", + nomad.Event{Key: "ebs", Name: "ebs"}, + }, + { + "variable", "Variable", + `{"Metadata": {"Path": "nomad/jobs/web", "Namespace": "ml", "ModifyTime": 1790671900000000000}}`, "nomad/jobs/web", + nomad.Event{Key: "nomad/jobs/web", Name: "nomad/jobs/web", Namespace: "ml"}, + }, + } { + client, _ := eventServer(t, `{"Index": 3, "Events": [{"Topic": "`+c.topic+`", "Type": "Updated", "Key": "`+c.key+`", "Index": 3, "Payload": `+c.payload+`}]}`) + + feed, err := client.Feed(context.Background(), "", nomad.Subject{}, 1) + r.NoError(err, c.name) + + event := waitForEvent(t, feed) + feed.Close() + + r.Equal(c.want.Key, event.Key, c.name) + r.Equal(c.want.Name, event.Name, c.name) + r.Equal(c.want.Namespace, event.Namespace, c.name) + r.Equal(c.want.State, event.State, c.name) + r.WithinDuration(time.Unix(0, 1790671900000000000), event.At, time.Microsecond, c.name) + } +} diff --git a/internal/ui/alloc.go b/internal/ui/alloc.go index 97d788f..3cb9a50 100644 --- a/internal/ui/alloc.go +++ b/internal/ui/alloc.go @@ -150,6 +150,7 @@ func allocKeys[P allocLister]() []pageKey[P] { {press: "ctrl+r", label: "Restart All", do: restartAllTasks[P], writes: true}, {press: "ctrl+k", label: "Stop", do: stopAllocs[P], writes: true}, {press: "l", label: "Logs", do: allocLogs[P]}, + {press: "o", label: "Alloc Events", do: openAllocEvents[P]}, markKey[P](), markAllKey[P](), } @@ -166,6 +167,16 @@ func openAllocTasks[P allocLister](p P, e env) (P, outcome) { return p, then(openMsg{listedTasks(alloc)}) } +// openAllocEvents opens the events of the allocation under the cursor. +func openAllocEvents[P allocLister](p P, e env) (P, outcome) { + alloc, ok := pickedFrom(e, p.visible(e)) + if !ok { + return p, outcome{} + } + + return p, then(openMsg{subjectEvents(alloc.Namespace, nomad.Subject{AllocID: alloc.ID})}) +} + // describeAlloc requests the allocation under the cursor, as the cluster // describes it. func describeAlloc[P allocLister](p P, e env) (P, outcome) { diff --git a/internal/ui/app_test.go b/internal/ui/app_test.go index 192068f..2960440 100644 --- a/internal/ui/app_test.go +++ b/internal/ui/app_test.go @@ -262,6 +262,7 @@ type fakeClient struct { // asked for last. feed *nomad.Feed feedNamespace string + feedSubject nomad.Subject feedFrom uint64 // scalingPolicies are the policies of the list, groupScaling how the @@ -701,8 +702,8 @@ func (f *fakeClient) DeleteNamespace(_ context.Context, name string) error { return f.actionErr } -func (f *fakeClient) Feed(_ context.Context, namespace string, from uint64) (*nomad.Feed, error) { - f.feedNamespace, f.feedFrom = namespace, from +func (f *fakeClient) Feed(_ context.Context, namespace string, subject nomad.Subject, from uint64) (*nomad.Feed, error) { + f.feedNamespace, f.feedSubject, f.feedFrom = namespace, subject, from return f.feed, f.err } diff --git a/internal/ui/eventsof_test.go b/internal/ui/eventsof_test.go new file mode 100644 index 0000000..76beb3c --- /dev/null +++ b/internal/ui/eventsof_test.go @@ -0,0 +1,176 @@ +package ui + +import ( + "testing" + + "github.com/stretchr/testify/require" + + "github.com/ingvarch/urga/internal/nomad" +) + +const mlAlloc = "af1f37df-7b19-6b1c-da67-5e8f482b5a15" + +// inML is a job and an allocation of the namespace ml, seen from a session +// that looks at every namespace. +func inML() ([]nomad.Job, []nomad.Alloc) { + jobs := []nomad.Job{{ID: "web", Name: "web", Namespace: "ml", Type: "service", Status: "running", Running: 1, Desired: 1}} + allocs := []nomad.Alloc{{ + ID: mlAlloc, Namespace: "ml", JobID: "web", TaskGroup: "frontend", Status: "running", + Tasks: []nomad.Task{{Name: "server", State: "running"}}, + }} + + return jobs, allocs +} + +// onEventsOfAJob is the job list of every namespace, and the events of the +// job under the cursor opened with o. +func onEventsOfAJob(t *testing.T) (Model, *fakeClient, *bool) { + t.Helper() + + jobs, allocs := inML() + feed, closed := feedOf(someEvents()...) + client := &fakeClient{jobs: jobs, allocs: allocs, alloc: allocs[0], feed: feed} + + m := newTestModel(client) + m.namespace = nomad.AllNamespaces + m, _ = m.update(jobsMsg(jobs)) + m, cmd := m.update(key('o')) + + return playOut(m, cmd), client, closed +} + +func TestEventsOf_AJobKeepsToItsOwnNamespace(t *testing.T) { + r := require.New(t) + + m, client, _ := onEventsOfAJob(t) + + r.Equal("Events (Job: web) [2]", m.title()) + r.Equal("ml", client.feedNamespace) + r.Equal(nomad.Subject{JobID: "web"}, client.feedSubject) + r.Equal(uint64(1), client.feedFrom) +} + +func TestEventsOf_AnAllocationOfEveryList(t *testing.T) { + r := require.New(t) + + _, allocs := inML() + allocs[0].NodeID = "node-1" + + opens := map[string]func(*fakeClient) Model{ + "allocations": func(c *fakeClient) Model { + m := newTestModel(c) + m.namespace = nomad.AllNamespaces + m, _ = m.update(openMsg{allocationsPage{allocs: allocs}}) + + return m + }, + "tasks": func(c *fakeClient) Model { + m := newTestModel(c) + m.namespace = nomad.AllNamespaces + m, _ = m.update(openMsg{listedTasks(allocs[0])}) + + return m + }, + "client": func(c *fakeClient) Model { + c.nodes, c.nodeAllocs = busyClient(), allocs + m, _ := nodeModelOf(c) + m, cmd := m.update(enter()) + + return drain(m, cmd) + }, + } + + for name, open := range opens { + feed, _ := feedOf(someEvents()...) + client := &fakeClient{allocs: allocs, alloc: allocs[0], feed: feed} + + m := open(client) + m, cmd := m.update(key('o')) + m = playOut(m, cmd) + + r.Equal("Events (Allocation: af1f37df) [2]", m.title(), name) + r.Equal("ml", client.feedNamespace, name) + r.Equal(nomad.Subject{AllocID: mlAlloc}, client.feedSubject, name) + } +} + +func TestEventsOf_SwitchingNamespaceLeavesThemAlone(t *testing.T) { + r := require.New(t) + + m, client, closed := onEventsOfAJob(t) + m.namespaceOrder = []string{"production", "ml"} + feed := m.screen.page.(feedPage).feed + client.feedNamespace, client.feedFrom = "", 0 + + m, cmd := m.update(key('1')) + m = playOut(m, cmd) + + r.Equal("Events (Job: web) [2]", m.title()) + r.False(m.screen.followsSession()) + r.False(*closed) + r.Empty(client.feedNamespace) + r.Zero(client.feedFrom) + r.Same(feed, m.screen.page.(feedPage).feed) +} + +func TestEventsOf_ComingBackGoesOnInTheSameSubject(t *testing.T) { + r := require.New(t) + + m, client, _ := onEventsOfAJob(t) + + // Into the allocation of the newest event, and back. + client.feed, _ = feedOf() + + m, cmd := m.update(enter()) + m = playOut(m, cmd) + r.IsType(tasksPage{}, m.screen.page) + + m, cmd = m.update(escape()) + m = playOut(m, cmd) + + r.Equal("Events (Job: web) [2]", m.title()) + r.Equal("ml", client.feedNamespace) + r.Equal(nomad.Subject{JobID: "web"}, client.feedSubject) + r.Equal(uint64(12), client.feedFrom) +} + +func TestEventsOf_ByNameStillFollowTheSession(t *testing.T) { + r := require.New(t) + + feed, _ := feedOf(someEvents()...) + client := &fakeClient{feed: feed} + m := onEvents(t, client) + + r.Equal(nomad.Subject{}, client.feedSubject) + r.True(m.screen.followsSession()) +} + +func TestEventsOf_AJobThatLaunchesOthersHasNone(t *testing.T) { + r := require.New(t) + + events := hint{Key: "", Description: "Events"} + + m := newTestModel(&fakeClient{jobs: periodicJobs()}) + m, _ = m.update(jobsMsg(periodicJobs())) + r.Contains(m.hints(), events) + + // A periodic job has almost no events of its own: its launches do. + m, _ = m.update(key('j')) + r.NotContains(m.hints(), events) + + m, cmd := m.update(key('o')) + r.Nil(cmd) + r.IsType(jobsPage{}, m.screen.page) +} + +func TestEventsOf_ReadOnlyOffersThem(t *testing.T) { + r := require.New(t) + + m := newTestModel(&fakeClient{jobs: twoJobs(), allocs: twoAllocs()}) + m.opts.ReadOnly = true + m, _ = m.update(jobsMsg(twoJobs())) + r.Contains(m.hints(), hint{Key: "", Description: "Events"}) + + m, _ = m.update(openMsg{allocationsPage{allocs: twoAllocs()}}) + r.Contains(m.hints(), hint{Key: "", Description: "Alloc Events"}) +} diff --git a/internal/ui/feed.go b/internal/ui/feed.go index ea9236d..6da96c3 100644 --- a/internal/ui/feed.go +++ b/internal/ui/feed.go @@ -46,7 +46,9 @@ type feedEvent struct { // feedPage is what happens in the cluster, event by event, newest first. It // shows the events as they came; no other screen is built out of them. type feedPage struct { - ofTheSession + // subject is what the page follows; the zero subject is the namespace + // of the session. + subject nomad.Subject feed *nomad.Feed events []feedEvent @@ -54,17 +56,40 @@ type feedPage struct { // namespace is the one the events were read in, and last the index of // the newest of them: back on the same namespace, the page goes on after - // it. + // it. A page with a subject stays in the namespace of the job or + // allocation. namespace string last uint64 } var feedTitles = []string{"Topic", "Type", "Namespace", "Name", "State", "Age"} -func (feedPage) title(e env, count int) string { +// subjectEvents is the events of a job or an allocation of a namespace. +func subjectEvents(namespace string, subject nomad.Subject) feedPage { + return feedPage{namespace: namespace, subject: subject} +} + +func (p feedPage) title(e env, count int) string { + switch { + case p.subject.AllocID != "": + return sprintf("Events (Allocation: %s) [%d]", shortID(p.subject.AllocID), count) + case p.subject.JobID != "": + return sprintf("Events (Job: %s) [%d]", p.subject.JobID, count) + } + return sprintf("Events (%s) [%d]", namespaceLabel(e.namespace), count) } +func (p feedPage) followsSession() bool { return p.subject == nomad.Subject{} } + +func (p feedPage) where(e env) string { + if p.followsSession() { + return e.namespace + } + + return p.namespace +} + func (feedPage) titles() []string { return feedTitles } // topics: none. The page reads the stream itself. @@ -79,19 +104,21 @@ func (feedPage) newestOnTop() {} func (p feedPage) open(e env) (page, tea.Cmd) { p.stop() + namespace := p.where(e) + from := uint64(1) - if p.namespace == e.namespace && p.last > 0 { + if p.namespace == namespace && p.last > 0 { from = p.last + 1 } else { p.events, p.last = nil, 0 } - p.namespace = e.namespace + p.namespace = namespace - client, namespace := e.client, e.namespace + client, subject := e.client, p.subject return p, func() tea.Msg { - feed, err := client.Feed(context.Background(), namespace, from) + feed, err := client.Feed(context.Background(), namespace, subject, from) if err != nil { return errMsg{err: err} } @@ -294,6 +321,14 @@ func matchOf(event nomad.Event) nomad.Match { m.Kind = nomad.MatchNode case nomad.TopicService: m.Kind = nomad.MatchService + case nomad.TopicCSIVolume: + m.Kind = nomad.MatchVolume + case nomad.TopicHostVolume: + m.Kind = nomad.MatchHostVolume + case nomad.TopicCSIPlugin: + m.Kind = nomad.MatchPlugin + case nomad.TopicVariable: + m.Kind = nomad.MatchVariable } return m diff --git a/internal/ui/feed_test.go b/internal/ui/feed_test.go index e398d1b..dea3f60 100644 --- a/internal/ui/feed_test.go +++ b/internal/ui/feed_test.go @@ -164,6 +164,30 @@ func TestEvents_EnterOpensWhatTheEventIsAbout(t *testing.T) { r.NotContains(m.hints(), hint{Key: "", Description: "Open"}) } +func TestEvents_EnterOpensAVariableAndAHostVolume(t *testing.T) { + r := require.New(t) + + feed, _ := feedOf(nomad.Event{Topic: nomad.TopicVariable, Namespace: "staging", Key: "app/db", Name: "app/db"}) + m := onEvents(t, &fakeClient{feed: feed}) + + m, cmd := m.update(enter()) + m = playOut(m, cmd) + + r.IsType(variablePage{}, m.screen.page) + r.Equal("staging", m.screen.page.(variablePage).namespace) + r.Equal("app/db", m.screen.page.(variablePage).path) + + feed, _ = feedOf(nomad.Event{Topic: nomad.TopicHostVolume, Namespace: "staging", Key: "h1", Name: "cache"}) + m = onEvents(t, &fakeClient{feed: feed}) + + m, cmd = m.update(enter()) + m = playOut(m, cmd) + + r.IsType(volumePage{}, m.screen.page) + r.Equal("staging", m.screen.page.(volumePage).namespace) + r.Equal(nomad.VolumeHost, m.screen.page.(volumePage).kind) +} + func TestEvents_TheFeedFollowsTheSession(t *testing.T) { r := require.New(t) @@ -198,6 +222,8 @@ func TestEvents_WhatEachTopicOpens(t *testing.T) { nomad.TopicJob: nomad.MatchJob, nomad.TopicAllocation: nomad.MatchAlloc, nomad.TopicDeployment: nomad.MatchDeployment, nomad.TopicEvaluation: nomad.MatchEval, nomad.TopicNode: nomad.MatchNode, nomad.TopicService: nomad.MatchService, + nomad.TopicCSIVolume: nomad.MatchVolume, nomad.TopicHostVolume: nomad.MatchHostVolume, + nomad.TopicCSIPlugin: nomad.MatchPlugin, nomad.TopicVariable: nomad.MatchVariable, nomad.TopicNodePool: "", "ACLToken": "", } { m := matchOf(nomad.Event{Topic: topic, Key: "k1", Name: "web", Namespace: "production"}) diff --git a/internal/ui/jobs.go b/internal/ui/jobs.go index 7920e73..ff13aac 100644 --- a/internal/ui/jobs.go +++ b/internal/ui/jobs.go @@ -184,11 +184,23 @@ var jobsKeys = []pageKey[jobsPage]{ {press: "v", label: "Versions", do: openVersions}, {press: "l", label: "Logs", do: jobLogs, offered: runsItself}, {press: "w", label: "Watch", do: watchJob, offered: runsItself}, + {press: "o", label: "Events", do: openJobEvents, offered: runsItself}, {press: "e", label: "Edit", do: editJob, writes: true}, {press: "n", label: "New", do: newJob, writes: true}, {press: "p", label: "Placement", do: jobPlacement, offered: jobWaits}, } +// openJobEvents opens the events of the job under the cursor, in the +// namespace of the job. +func openJobEvents(p jobsPage, e env) (jobsPage, outcome) { + job, ok := p.picked(e) + if !ok { + return p, outcome{} + } + + return p, then(openMsg{subjectEvents(job.Namespace, nomad.Subject{JobID: job.ID})}) +} + func (p jobsPage) keys(e env) []keyHint { return hintsOf(p, e, jobsKeys) } func (p jobsPage) press(k string, e env) (page, outcome, bool) { diff --git a/internal/ui/jobs_test.go b/internal/ui/jobs_test.go index 0422a98..6b9978f 100644 --- a/internal/ui/jobs_test.go +++ b/internal/ui/jobs_test.go @@ -90,6 +90,7 @@ func TestJobs_TheKeysOfAJob(t *testing.T) { {Key: "", Description: "Versions"}, {Key: "", Description: "Logs"}, {Key: "", Description: "Watch"}, + {Key: "", Description: "Events"}, {Key: "", Description: "Edit"}, {Key: "", Description: "New"}, } diff --git a/internal/ui/navigation_test.go b/internal/ui/navigation_test.go index 260ddad..ca9273d 100644 --- a/internal/ui/navigation_test.go +++ b/internal/ui/navigation_test.go @@ -279,7 +279,7 @@ func allocationLists(t *testing.T) map[string]Model { func TestAllocations_EachListAnswersItsOwnKeys(t *testing.T) { r := require.New(t) - allocKeys := []string{"enter Tasks", "d Describe", "r Restart", "ctrl+r Restart All", "ctrl+k Stop", "l Logs", "space Mark", "ctrl+a Mark All"} + allocKeys := []string{"enter Tasks", "d Describe", "r Restart", "ctrl+r Restart All", "ctrl+k Stop", "l Logs", "o Alloc Events", "space Mark", "ctrl+a Mark All"} want := map[string][]string{ "job": allocKeys, diff --git a/internal/ui/plugins.go b/internal/ui/plugins.go index 8ad67a7..72d5a87 100644 --- a/internal/ui/plugins.go +++ b/internal/ui/plugins.go @@ -33,9 +33,7 @@ func (pluginsPage) title(_ env, count int) string { return sprintf("Plugins [%d] func (pluginsPage) titles() []string { return pluginsTitles } -// topics: none. The cluster reports no change of a plugin; the list is -// polled. -func (pluginsPage) topics() []string { return nil } +func (pluginsPage) topics() []string { return []string{nomad.TopicCSIPlugin} } func (pluginsPage) fetch(e env) tea.Cmd { return fetchList(e.client.Plugins, func(items []nomad.Plugin) tea.Msg { return pluginsMsg(items) }) @@ -137,8 +135,7 @@ func (p pluginPage) title(_ env, count int) string { return sprintf("Plugin %s [ func (pluginPage) titles() []string { return pluginTitles } -// topics: none, as for the list. -func (pluginPage) topics() []string { return nil } +func (pluginPage) topics() []string { return []string{nomad.TopicCSIPlugin} } func (p pluginPage) fetch(e env) tea.Cmd { client, id := e.client, p.id diff --git a/internal/ui/roles.go b/internal/ui/roles.go index a10b127..92e12e9 100644 --- a/internal/ui/roles.go +++ b/internal/ui/roles.go @@ -49,7 +49,7 @@ type ( // stale, and the feed of what happened. eventsClient interface { Events(ctx context.Context, namespace string, topics []string) (*nomad.Changes, error) - Feed(ctx context.Context, namespace string, from uint64) (*nomad.Feed, error) + Feed(ctx context.Context, namespace string, subject nomad.Subject, from uint64) (*nomad.Feed, error) } jobsClient interface { diff --git a/internal/ui/tasks.go b/internal/ui/tasks.go index d3427ff..5df92fe 100644 --- a/internal/ui/tasks.go +++ b/internal/ui/tasks.go @@ -114,6 +114,7 @@ var tasksKeys = []pageKey[tasksPage]{ // The same key opens the events of a task here and edits elsewhere, // because a screen never offers both. {press: "e", label: "Events", do: openTaskEvents}, + {press: "o", label: "Alloc Events", do: openAllocationEvents}, {press: "ctrl+e", label: "Stderr", do: followStderr}, // The same key opens a shell here and scales a task group elsewhere. {press: "s", label: "Shell", do: shell, writes: true}, @@ -181,6 +182,12 @@ func openTaskEvents(p tasksPage, e env) (tasksPage, outcome) { return p, then(openMsg{taskEventsPage{namespace: p.namespace, allocID: p.allocID, task: task.Name, alloc: p.alloc}}) } +// openAllocationEvents opens the events of the allocation the page is open +// on. +func openAllocationEvents(p tasksPage, _ env) (tasksPage, outcome) { + return p, then(openMsg{subjectEvents(p.namespace, nomad.Subject{AllocID: p.allocID})}) +} + // taskEventsPage is what happened to one task of an allocation. type taskEventsPage struct { noKeys diff --git a/internal/ui/tasks_test.go b/internal/ui/tasks_test.go index 8de25e9..5ec5740 100644 --- a/internal/ui/tasks_test.go +++ b/internal/ui/tasks_test.go @@ -147,6 +147,7 @@ func TestTasks_TheKeysOfATask(t *testing.T) { keys := []hint{ {Key: "", Description: "Logs"}, {Key: "", Description: "Events"}, + {Key: "", Description: "Alloc Events"}, {Key: "", Description: "Stderr"}, {Key: "", Description: "Shell"}, {Key: "", Description: "Restart"}, diff --git a/internal/ui/update.go b/internal/ui/update.go index 5dd1783..2e36553 100644 --- a/internal/ui/update.go +++ b/internal/ui/update.go @@ -182,7 +182,9 @@ func (m Model) fromSession(msg tea.Msg) (Model, tea.Cmd, bool) { return m.keepToken(msg), nil, true case agentMsg: - return m.keepAgent(msg), nil, true + m = m.keepAgent(msg) + + return m, m.watchOnVersion(), true case namespacesMsg: return taken(m.keepNamespaces(msg)) diff --git a/internal/ui/variable.go b/internal/ui/variable.go index d9625bc..c6d7e35 100644 --- a/internal/ui/variable.go +++ b/internal/ui/variable.go @@ -37,7 +37,7 @@ func (variablesPage) title(e env, count int) string { } func (variablesPage) titles() []string { return variableTitles } -func (variablesPage) topics() []string { return nil } +func (variablesPage) topics() []string { return []string{nomad.TopicVariable} } func (variablesPage) fetch(e env) tea.Cmd { client, namespace := e.client, e.namespace @@ -129,7 +129,10 @@ func (p variablePage) title(_ env, count int) string { } func (variablePage) titles() []string { return variableItemTitles } -func (variablePage) topics() []string { return nil } +func (variablePage) topics() []string { return []string{nomad.TopicVariable} } + +// where: a variable lives in its namespace, whatever the session looks at. +func (p variablePage) where(env) string { return p.namespace } // fetch reads the variable the page is open on. func (p variablePage) fetch(e env) tea.Cmd { diff --git a/internal/ui/volumes.go b/internal/ui/volumes.go index f8cafd2..d88da15 100644 --- a/internal/ui/volumes.go +++ b/internal/ui/volumes.go @@ -33,9 +33,11 @@ func (volumesPage) title(e env, count int) string { func (volumesPage) titles() []string { return volumesTitles } -// topics: none. The cluster reports no change of a CSI volume, and a topic -// for half the list would leave the other half stale; the list is polled. -func (volumesPage) topics() []string { return nil } +// topics: the plugin as well, because the colour of a CSI volume follows the +// health of its plugin, which only a plugin event reports. +func (volumesPage) topics() []string { + return []string{nomad.TopicCSIVolume, nomad.TopicHostVolume, nomad.TopicCSIPlugin} +} func (volumesPage) fetch(e env) tea.Cmd { client, namespace := e.client, e.namespace @@ -189,8 +191,14 @@ func (p volumePage) title(_ env, count int) string { func (volumePage) titles() []string { return allocTitles } -// topics: none, as for the list. -func (volumePage) topics() []string { return nil } +// topics: the rows are allocations, as on the deployment screen. +func (p volumePage) topics() []string { + if p.kind == nomad.VolumeHost { + return []string{nomad.TopicHostVolume, nomad.TopicAllocation} + } + + return []string{nomad.TopicCSIVolume, nomad.TopicCSIPlugin, nomad.TopicAllocation} +} // where: a volume lives in its namespace, whatever the session looks at. func (p volumePage) where(env) string { return p.namespace } diff --git a/internal/ui/watch.go b/internal/ui/watch.go index 5c2487c..4a5a07e 100644 --- a/internal/ui/watch.go +++ b/internal/ui/watch.go @@ -70,7 +70,7 @@ type watchState struct { // way it always was. func (m Model) watchScreen() tea.Cmd { topics := m.screen.page.topics() - if len(topics) == 0 { + if len(topics) == 0 || !nomad.Streams(m.nomadVersion, topics) { return nil } @@ -86,6 +86,18 @@ func (m Model) watchScreen() tea.Cmd { } } +// watchOnVersion opens the stream of a screen whose topics need to know the +// version of the cluster, which arrives after the screen asked. A screen of +// older topics has its stream on the way already. +func (m Model) watchOnVersion() tea.Cmd { + topics := m.screen.page.topics() + if m.watch.live() || nomad.Streams("", topics) { + return nil + } + + return m.watchScreen() +} + // waitForChange waits for the next message on the stream. func (w watchState) waitForChange() tea.Cmd { changes := w.changes diff --git a/internal/ui/watch_test.go b/internal/ui/watch_test.go index f8ee62b..066e853 100644 --- a/internal/ui/watch_test.go +++ b/internal/ui/watch_test.go @@ -312,7 +312,7 @@ func TestWatch_EveryListTheClusterTalksAboutIsWatched(t *testing.T) { r := require.New(t) // A list the cluster has a topic for is watched; one it does not, like - // the namespaces or the variables, is asked for on a timer. + // the namespaces or the servers, is asked for on a timer. watched := map[*view][]string{ jobsView: {nomad.TopicJob}, allocationsView: {nomad.TopicAllocation}, @@ -321,6 +321,9 @@ func TestWatch_EveryListTheClusterTalksAboutIsWatched(t *testing.T) { nodesView: {nomad.TopicNode}, servicesView: {nomad.TopicService}, nodePoolsView: {nomad.TopicNodePool}, + volumesView: {nomad.TopicCSIVolume, nomad.TopicHostVolume, nomad.TopicCSIPlugin}, + pluginsView: {nomad.TopicCSIPlugin}, + variablesView: {nomad.TopicVariable}, } // Each is checked the way the program opens it, page and all. @@ -328,11 +331,86 @@ func TestWatch_EveryListTheClusterTalksAboutIsWatched(t *testing.T) { r.Equal(topics, v.open().topics(), nameOf(v)) } - for _, v := range []*view{namespacesView, variablesView, serversView} { + for _, v := range []*view{namespacesView, serversView} { r.Empty(v.open().topics(), nameOf(v)) } } +func TestWatch_ThePagesOfStorageAndVariablesWatchTheirOwn(t *testing.T) { + r := require.New(t) + + for _, c := range []struct { + page page + topics []string + }{ + {volumePage{kind: nomad.VolumeCSI}, []string{nomad.TopicCSIVolume, nomad.TopicCSIPlugin, nomad.TopicAllocation}}, + {volumePage{kind: nomad.VolumeHost}, []string{nomad.TopicHostVolume, nomad.TopicAllocation}}, + {pluginPage{id: "ebs"}, []string{nomad.TopicCSIPlugin}}, + {variablePage{path: "app/db"}, []string{nomad.TopicVariable}}, + } { + r.Equal(c.topics, c.page.topics(), "%T", c.page) + } + + // A variable is watched in its own namespace, whatever the session looks at. + for _, session := range []string{"production", nomad.AllNamespaces} { + client := &fakeClient{changes: newChanges()} + m := newTestModel(client) + m.nomadVersion = "2.0.7" + m.namespace = session + m.screen.page = variablePage{namespace: "staging", path: "app/db"} + + m.update(m.watchScreen()()) + r.Equal("staging", client.watchedNamespace, session) + } +} + +func TestWatch_AClusterOlderThanTheTopicIsPolled(t *testing.T) { + r := require.New(t) + + for version, topics := range map[string][]string{ + "1.11.3": nil, + "": nil, + "2.0.7": {nomad.TopicVariable}, + } { + client := &fakeClient{changes: newChanges()} + m := newTestModel(client) + m.nomadVersion = version + m = typeCommand(m, "variables") + + r.Equal(topics, client.watchedTopics, version) + + if topics == nil { + r.Equal(time.Millisecond, m.pollEvery(), version) + } + } +} + +func TestWatch_TheVersionArrivingAfterTheScreenOpensItsStream(t *testing.T) { + r := require.New(t) + + client := &fakeClient{changes: newChanges()} + m := typeCommand(newTestModel(client), "variables") + r.Empty(client.watchedTopics) + + m, cmd := m.update(agentMsg{Version: "2.0.7"}) + drain(m, cmd) + r.Equal([]string{nomad.TopicVariable}, client.watchedTopics) + + // The stream is open: another answer about the version does not open a second. + m, _ = m.update(m.watchScreen()()) + r.True(m.watch.live()) + + _, cmd = m.update(agentMsg{Version: "2.0.7"}) + r.Nil(cmd) + + // A screen of topics every cluster streams already asked on its own. + client = &fakeClient{jobs: twoJobs(), changes: newChanges()} + m = newTestModel(client) + + _, cmd = m.update(agentMsg{Version: "2.0.7"}) + r.Nil(cmd) +} + func TestWatch_AClusterThatWillNotStreamSaysSo(t *testing.T) { r := require.New(t)