diff --git a/README.md b/README.md index d45551d..b4a7700 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ Let's ask for the report: opa eval -d policy -i deployments.json --format=pretty 'data.deploy.report' ``` -The report records every check ergo ran, including the ones that passed. It comes in three parts, after the policy's own `description` and `meta`, which this one doesn't have: +The report records every check ergo ran, including the ones that passed. It comes in four parts, after the policy's own `description` and `meta`, which this one doesn't have: ```json { @@ -107,13 +107,26 @@ The report records every check ergo ran, including the ones that passed. It come "description": "", "meta": {}, "requirements": { ... }, - "results": [ ... ] + "results": [ ... ], + "subjects": { ... } } ``` - `compliant` is the overall answer. When you need a plain yes or no to allow or block something, this is the one to use: `allow := report.compliant`. - `requirements` has an entry for each requirement. It repeats the requirement's `description`, and its `status` is `met`, `not_met`, or `not_applicable` when no subject was left to check. When it's `not_met`, `causes` says why, worst first, with how many rows each: here `[{"cause": "absent", "rows": 1}]`. It also says how many subjects were found and kept, and lists every check with a readable `expression`, which is the sentence as you wrote it. - `results` has one row per subject and check, with the value ergo read, whether the check passed, and a `cause`. +- `subjects` says where each subject the requirements use comes from. Here it holds one entry, and the requirement's own entry has `"subject": "deployment"` to point at it: + + ```json + "subjects": { + "deployment": { + "description": "", + "from": "deployments whose environment is \"prod\"", + "identified_by": "id", + "parts": {} + } + } + ``` Here's what those rows look like for our example: diff --git a/REFERENCE.md b/REFERENCE.md index a57a4f2..539670b 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -130,11 +130,11 @@ deployment: identified_by: id ``` -| Field | Meaning | Default | -| --------------- | ----------------------------------------------------------------------------------------------------------- | -------------------------------------- | -| `description` | What the subject is, in words anyone can read. ergo doesn't copy it into the report. | `""` | -| `from` | Where the subjects are in the input, and which of them are in scope, written as a [from line](#from-lines). | the input field named like the subject | -| `identified_by` | The path, inside one subject, to the value that identifies it. | the whole subject | +| Field | Meaning | Default | +| --------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | +| `description` | What the subject is, in words anyone can read. The report copies it into its [`subjects`](#subjects-in-the-report). | `""` | +| `from` | Where the subjects are in the input, and which of them are in scope, written as a [from line](#from-lines). | the input field named like the subject | +| `identified_by` | The path, inside one subject, to the value that identifies it. | the whole subject | `applies_to` and `of`, and a `from` or `identified_by` written as a list, are the [old syntax](#subjects-in-the-old-syntax). @@ -849,7 +849,8 @@ Three rules: "description": "", "meta": {}, "requirements": { ... }, - "results": [ ... ] + "results": [ ... ], + "subjects": { ... } } ``` @@ -857,6 +858,8 @@ Three rules: `description` and `meta` are the [policy's](#describing-a-policy), or `""` and `{}` when it has none. +`subjects` says where the subjects come from, as [Subjects in the report](#subjects-in-the-report) shows. + `requirements` has an entry for each requirement: ```json @@ -893,6 +896,7 @@ Three rules: "require": "every", "status": "not_met", "causes": [{ "cause": "absent", "rows": 1 }], + "subject": "deployment", "subjects": { "matching": 2, "total": 3 } } ``` @@ -935,7 +939,7 @@ The three `ill_formed` rows are the misspelt check on `d-1`, `d-2` and `d-4`. Th `description` and `meta` are the requirement's, or `""` and `{}` when it has none. They're always there, so you can read `meta.control` without first checking that `meta` exists. -`subjects.total` counts every subject found at `from`, and `subjects.matching` counts the ones left after the subject's conditions. `checks` holds each check as you wrote it, plus the `expression` ergo rendered from it. Each check has a `description` and a `meta` too, `""` and `{}` when you didn't write them. A filter in `applies_to`, in the [old syntax](#subjects-in-the-old-syntax), can have them as well, and they're checked the same way, but the report doesn't copy them: its filters all share the one `$applies` entry, with ergo's own description. A `description` or `meta` of the wrong type makes the requirement or the check [written wrong](#basic-operators), and the report shows `""` or `{}` in its place, so a tool always finds a string and an object there. If you write your own `expression`, yours is used, as long as it's a string. Any other value is ignored, and ergo renders the expression as if it weren't there. A check that uses a [`ref`](#reading-from-the-input) also gets `$refs`: the name and value of each one, as read for this report. The `$` marks it as ergo's, so it can't be mixed up with a field of your own. It also holds the [checks ergo adds](#checks-ergo-adds), each with a `description`, an `expression` and an empty `meta`. +`subject` is the name of the subject the requirement checks, which is also a key of the report's `subjects`. It's `null` when the requirement names no subject, or one that isn't in the policy's `subjects`. `subjects.total` counts every subject found at `from`, and `subjects.matching` counts the ones left after the subject's conditions. `checks` holds each check as you wrote it, plus the `expression` ergo rendered from it. Each check has a `description` and a `meta` too, `""` and `{}` when you didn't write them. A filter in `applies_to`, in the [old syntax](#subjects-in-the-old-syntax), can have them as well, and they're checked the same way, but the report doesn't copy them: its filters all share the one `$applies` entry, with ergo's own description. A `description` or `meta` of the wrong type makes the requirement or the check [written wrong](#basic-operators), and the report shows `""` or `{}` in its place, so a tool always finds a string and an object there. If you write your own `expression`, yours is used, as long as it's a string. Any other value is ignored, and ergo renders the expression as if it weren't there. A check that uses a [`ref`](#reading-from-the-input) also gets `$refs`: the name and value of each one, as read for this report. The `$` marks it as ergo's, so it can't be mixed up with a field of your own. It also holds the [checks ergo adds](#checks-ergo-adds), each with a `description`, an `expression` and an empty `meta`. In an expression, a value written in the policy is shown as JSON, written the same way whatever the policy looked like, so anyone can produce the same text: @@ -964,6 +968,113 @@ Several of these rules depend on how a number was written, not only on its value Passing and failing rows have the same fields. A row of an `all` or `any` check also has [`failed_items`](#which-items-failed). To find a row's description and expression, look up `requirements[row.requirement].checks[row.check]`. Look it up through the requirement, because two requirements can use the same check name for different things. +### Subjects in the report + +`subjects` has an entry for each subject the requirements use, by name. That's the subject a requirement names, the subject it narrows or reads from, and so on all the way back. A subject no requirement uses isn't listed. + +Take this policy: + +```yaml +subjects: + deployment: + description: Something we ship + from: deployments + identified_by: id + prod deployment: + from: deployment whose environment is "prod" + release: + from: releases +requirements: + approved: + subject: prod deployment + checks: + approved: approved_by is a non-empty string +``` + +and this input: + +```json +{ + "deployments": [ + { "id": "d-1", "environment": "prod", "approved_by": "alice" }, + { "id": "d-2", "environment": "staging" } + ] +} +``` + +The requirement's entry has `"subject": "prod deployment"`, and `subjects` is: + +```json +{ + "deployment": { + "description": "Something we ship", + "from": "deployments", + "identified_by": "id", + "parts": {} + }, + "prod deployment": { + "description": "", + "from": "deployment whose environment is \"prod\"", + "identified_by": "id", + "parts": {} + } +} +``` + +`release` isn't there, because no requirement uses it. `deployment` is, because `prod deployment` narrows it. + +Each entry has: + +- `description` is the subject's, or `""` when it has none. A `description` that isn't a string is [written wrong](#subjects) and shows as `""`. +- `from` says how the subject is found, as a [from line](#from-lines) is written. A `from` written as a line is printed the same way whatever spaces it had. One written as a list, in the [old syntax](#subjects-in-the-old-syntax), is printed as a sentence too: its path, then ` whose ` and its `applies_to` filters, sorted by name and joined with `and`. A [naming step](#naming-subjects) at the end of the path shows as `named by each of $params.suites` when its keys come from a ref, as `` when they're listed, and as `` when it has none, since a `from` line has no way to write those two. A filter that can't be written as a sentence, like one with a custom operator or a substitute, shows as `` so it's never left out, and an `applies_to` that isn't an object shows as ``. A subject built on another with `of` starts with that subject's name. A subject without `from` reads the input field named like it. A `from` that is written wrong shows as ``. +- `identified_by` is the path of the subject's id, or `null` when it has none, and the whole subject is its id. A subject that narrows another has the id of the one it narrows. An id that is written wrong shows as ``. +- `parts` is always `{}` for now. + +A subject that is written wrong still has its entry, so nothing disappears from the report. With this policy and the same input: + +```yaml +subjects: + deployment: + from: [deployments] + identified_by: [id] + applies_to: + is_prod: {op: equals, path: [environment], value: prod} + note: {} +requirements: + approved: + subject: deployment + checks: + approved: approved_by is a non-empty string + noted: + subject: note + checks: + text: text is a non-empty string + other: + checks: + x: x exists +``` + +`subjects` is: + +```json +{ + "deployment": { + "description": "", + "from": "deployments whose environment is \"prod\"", + "identified_by": "id", + "parts": {} + }, + "note": { + "description": "", + "from": "note", + "identified_by": null, + "parts": {} + } +} +``` + +and the three requirements' `subject` are `"deployment"`, `"note"` and `null`, because `other` names no subject. + ### Checks ergo adds ergo adds four checks of its own. They start with `$`, so they can't clash with yours. diff --git a/ergo.rego b/ergo.rego index ef55bb7..2984ca9 100644 --- a/ergo.rego +++ b/ergo.rego @@ -4423,7 +4423,7 @@ report(doc, policy, params) := r if { sentences := _s_sentences(raw, reqs) unreadable := _unreadable_inputs(doc, params) resolved := _resolved_subjects with data.ergo_subjects as subjects with data.ergo_subject_sentences as subject_sentences - written := object.union(_report_of(doc, reqs), {"description": _reported_description(policy), "meta": _reported_meta(policy)}) with data.ergo_policy_problems as _policy_problem_inputs(policy) with data.ergo_document as doc with data.ergo_params as params with data.ergo_subjects as subjects with data.ergo_resolved as resolved with data.ergo_unreadable as unreadable with data.ergo_sentences as sentences with data.ergo_subject_sentences as subject_sentences with input as {"ergo/names": {}} + written := object.union(_report_of(doc, reqs), {"description": _reported_description(policy), "meta": _reported_meta(policy), "subjects": _reported_subjects(written_subjects, reqs)}) with data.ergo_policy_problems as _policy_problem_inputs(policy) with data.ergo_document as doc with data.ergo_params as params with data.ergo_subjects as subjects with data.ergo_resolved as resolved with data.ergo_unreadable as unreadable with data.ergo_sentences as sentences with data.ergo_subject_sentences as subject_sentences with input as {"ergo/names": {}} used := _used_subjects(reqs, subjects, resolved) r := _reported(written, unreadable, [raw, used]) } @@ -4434,6 +4434,188 @@ _used_subjects(reqs, subjects, resolved) := {name: subjects[name] | some name in resolved[req.subject].members } +_reported_subject_name(req) := req.subject if { + _names_a_subject(req) + is_object(data.ergo_subjects) + req.subject in object.keys(data.ergo_subjects) +} else := null + +_listed_subject_graph := {name: array.concat(_of_edges(def, data.ergo_subjects), [chains[name] | name in object.keys(chains)]) | + some name, def in data.ergo_subjects + chains := object.get(data.ergo_subject_sentences, "chains", {}) +} + +_listed_subject_names(reqs) := graph.reachable(_listed_subject_graph, {req.subject | + some req in reqs + _reported_subject_name(req) != null +}) + +_reported_subjects(written, reqs) := {name: _reported_subject(name, written[name]) | some name in _listed_subject_names(reqs)} if is_object(written) + +_reported_subjects(written, _) := {} if not is_object(written) + +_reported_subject(name, def) := { + "description": _reported_description(def), + "from": _subject_from_text(name, def), + "identified_by": _subject_id_text(name), + "parts": {}, +} + +_subject_from_text(name, null) := _path_name([name]) + +_subject_from_text(_, def) := "" if { + def != null + not is_object(def) +} + +_subject_from_text(name, def) := "" if { + is_object(def) + _subject_from_unreadable(name, def) +} + +_subject_from_text(name, def) := _subject_text(name, def) if { + is_object(def) + not _subject_from_unreadable(name, def) +} + +default _subject_from_invalid(_) := false + +_subject_from_invalid(name) if { + some p in data.ergo_resolved[name].problems + some f in ["from", "of"] + p[0] == ["subjects", name, f] +} + +_subject_from_unreadable(name, _) if _subject_from_invalid(name) + +_subject_from_unreadable(name, def) if not _subject_readable(name, def) + +_subject_readable(name, def) if _subject_text(name, def) + +_subject_text(_, def) := _s_from_line_text(_from_line(def.from)) if { + is_string(object.get(def, "from", null)) + object.keys(def) & {"of", "applies_to"} == set() +} + +_subject_text(name, def) := concat("", [_subject_source_text(name, def), _subject_whose_text(def)]) if not is_string(object.get(def, "from", null)) + +_subject_source_text(name, def) := _path_name([def.of]) if { + is_string(object.get(def, "of", null)) + not "from" in object.keys(def) + def.of != name + def.of in object.keys(data.ergo_subjects) +} else := _subject_list_source_text(name, def.from) if { + is_array(object.get(def, "from", null)) + not "of" in object.keys(def) +} else := _path_name([name]) if { + not "from" in object.keys(def) + not "of" in object.keys(def) +} + +_subject_list_source_text(_, []) := "$input" + +_subject_list_source_text(name, from) := _c_from_path(name, from, data.ergo_subjects) if { + count(from) > 0 + _all_string_steps(from) +} + +_subject_list_source_text(name, from) := text if { + count(from) > 0 + not _all_string_steps(from) + text := _c_source(name, from, data.ergo_subjects) +} else := concat("", [_from_text({"subject": name}), _naming_text(from[count(from) - 1])]) if { + count(from) > 0 + not _all_string_steps(from) +} + +_naming_text(step) := "" if not _c_naming(step) + +_naming_text(step) := concat("", [" "]) if { + _c_naming(step) + not "keys" in object.keys(step) +} + +_naming_text(step) := concat("", [" named by each of ", _ref_name(step.keys.ref)]) if { + _c_naming(step) + _is_ref(object.get(step, "keys", null)) +} + +_naming_text(step) := concat("", [" "]) if { + _c_naming(step) + is_array(object.get(step, "keys", null)) +} + +_naming_text(step) := concat("", [" "]) if { + _c_naming(step) + _is_literal(object.get(step, "keys", null)) + is_array(step.keys.literal) +} + +_naming_key_text(k) := _ref_name(k.ref) if _is_ref(k) + +_naming_key_text(k) := _literal_text(k.literal) if _is_literal(k) + +_naming_key_text(k) := _literal_text(k) if { + not _is_ref(k) + not _is_literal(k) +} + +_all_string_steps(from) if { + every step in from { + is_string(step) + } +} + +_subject_whose_text(def) := "" if not "applies_to" in object.keys(def) + +_subject_whose_text(def) := " whose " if { + "applies_to" in object.keys(def) + not is_object(def.applies_to) +} + +_subject_whose_text(def) := "" if { + is_object(object.get(def, "applies_to", null)) + count(def.applies_to) == 0 +} + +_subject_whose_text(def) := concat("", [" whose ", concat(" and ", [_subject_condition_text(n, def.applies_to[n]) | some n in sort(object.keys(def.applies_to))])]) if { + is_object(object.get(def, "applies_to", null)) + count(def.applies_to) > 0 +} + +_subject_condition_text(n, f) := _c_candidates(_c_bare(f))[0] if { + is_object(f) + (object.keys(_c_bare(f)) - {"op"}) - object.get(_plain_condition_fields, object.get(f, "op", null), set()) == set() +} else := concat("", [""]) + +_plain_condition_fields := { + "present": {"path"}, + "missing": {"path"}, + "non_empty_string": {"path"}, + "equals": {"path", "value"}, + "in": {"path", "values"}, + "range": {"path", "min", "max"}, + "matches_any": {"path", "patterns"}, + "not_matches_any": {"path", "patterns"}, + "includes": {"path", "value", "values"}, + "excludes": {"path", "value", "values"}, + "compare": {"left", "right", "cmp", "within"}, + "compare_time": {"left", "right", "cmp"}, +} + +_subject_id_text(name) := _id_text(data.ergo_resolved[name].def.identified_by) if { + "identified_by" in object.keys(data.ergo_resolved[name].def) +} else := null + +_id_text(p) := null if p == [] + +_id_text(p) := _path_name(p) if { + is_array(p) + count(p) > 0 +} + +_id_text(p) := "" if not is_array(p) + _section(policy, name) := object.get(policy, name, {}) if is_object(policy) _section(policy, _) := null if not is_object(policy) @@ -4478,6 +4660,7 @@ _report_of(doc, policy) := { "description": _reported_description(req), "meta": _reported_meta(req), "require": _require_of(req), + "subject": _reported_subject_name(req), "status": status, "causes": _causes(status, evaluations[name]), "subjects": {"total": evaluations[name].total, "matching": evaluations[name].matching}, diff --git a/ergo_test.rego b/ergo_test.rego index b7e932b..8113368 100644 --- a/ergo_test.rego +++ b/ergo_test.rego @@ -4026,11 +4026,11 @@ test_report_has_one_entry_per_declared_requirement if { test_requirement_entry_carries_exactly_the_documented_keys if { rep := solo({"state": "MERGED"}, is_merged) - object.keys(rep.requirements.s) == {"description", "meta", "require", "status", "causes", "subjects", "checks"} + object.keys(rep.requirements.s) == {"description", "meta", "require", "subject", "status", "causes", "subjects", "checks"} } test_report_carries_exactly_the_documented_keys if { - object.keys(solo({}, is_merged)) == {"compliant", "description", "meta", "requirements", "results"} + object.keys(solo({}, is_merged)) == {"compliant", "description", "meta", "requirements", "results", "subjects"} } test_report_is_deterministic if { @@ -6155,7 +6155,7 @@ test_the_report_never_reads_data_params if { [r.cause | some r in rows_for(unset, "s", "c")] == ["absent"] } -empty_report := {"compliant": false, "description": "", "meta": {}, "requirements": {}, "results": []} +empty_report := {"compliant": false, "description": "", "meta": {}, "requirements": {}, "results": [], "subjects": {}} test_a_policy_without_requirements_is_not_compliant if { ergo.report({"items": [{"id": 1}]}, {}, {}) == empty_report @@ -7455,9 +7455,11 @@ test_a_subject_can_have_a_description if { described := object.union(prod_deployment, {"description": "A deployment to production"}) rep := ergo.report(deployments_doc, subject_policy({"deployment": described}, {"a": approved_req("deployment")}), {}) rows_for(rep, "a", "$well_formed")[0].passed == true - rep == ergo.report(deployments_doc, subject_policy({"deployment": prod_deployment}, {"a": approved_req("deployment")}), {}) + without_subjects(rep) == without_subjects(ergo.report(deployments_doc, subject_policy({"deployment": prod_deployment}, {"a": approved_req("deployment")}), {})) } +without_subjects(rep) := object.remove(rep, ["subjects"]) + test_a_subject_can_read_params if { from_params := {"from": [{"ref": ["$$params", "list"]}], "identified_by": ["id"], "applies_to": {"env": {"op": "equals", "path": ["environment"], "value": {"ref": ["$$params", "env"]}}}} rep := ergo.report(deployments_doc, subject_policy({"deployment": from_params}, {"a": approved_req("deployment")}), {"list": "deployments", "env": "staging"}) @@ -7510,7 +7512,7 @@ test_a_subject_built_on_another_reports_what_the_same_subject_written_out_report }}} versions := object.union(sbom_reqs.versions, {"subject": "locked"}) rep := ergo.report(sbom_doc, subject_policy(built, {"versions": versions}), {}) - rep == ergo.report(sbom_doc, subject_policy(written_out, {"versions": versions}), {}) + without_subjects(rep) == without_subjects(ergo.report(sbom_doc, subject_policy(written_out, {"versions": versions}), {})) rep.requirements.versions.checks["$applies"].expression == "sbom_release is present and lock_release is present" } @@ -7549,7 +7551,7 @@ test_a_subject_can_build_on_a_chain_of_subjects if { "pinned": {"op": "present", "path": ["lock_release"]}, "not_exempt": {"op": "equals", "path": ["exempt"], "value": false}, }}} - rep == ergo.report(sbom_doc, subject_policy(written_out, {"v": sbom_reqs.versions}), {}) + without_subjects(rep) == without_subjects(ergo.report(sbom_doc, subject_policy(written_out, {"v": sbom_reqs.versions}), {})) [[r.subject.id, r.passed] | some r in rep.results; r.check == "$applies"] == [["a", true], ["b", false], ["c", false]] rows_for(rep, "v", "$well_formed")[0].passed == true } @@ -10160,3 +10162,256 @@ test_elapsed_is_an_invalid_side_in_todays_syntax if { _sentence_problems({"op": "compare", "left": {"add": [side, {"literal": 1}], "text": "t"}, "cmp": "eq", "right": {"literal": 1}}) == ["invalid left"] _sentence_problems({"op": "compare", "left": object.union(side, {"unit": "months"}), "cmp": "eq", "right": {"literal": 1}}) == ["invalid left"] } + +_listed_reqs(name) := {"r": {"subject": name, "checks": {"c": "id exists"}}} + +_listed(subjects, name) := ergo.report({}, {"subjects": subjects, "requirements": _listed_reqs(name)}, {}).subjects + +test_the_report_lists_a_subject_with_its_description_from_and_id if { + subjects := {"deployment": {"from": `deployments whose environment is "prod"`, "identified_by": "id", "description": "A deployment to production"}} + _listed(subjects, "deployment") == {"deployment": {"description": "A deployment to production", "from": `deployments whose environment is "prod"`, "identified_by": "id", "parts": {}}} +} + +test_a_listed_subject_carries_exactly_the_documented_keys if { + rep := ergo.report({}, {"subjects": {"d": {"from": "ds"}}, "requirements": _listed_reqs("d")}, {}) + object.keys(rep.subjects.d) == {"description", "from", "identified_by", "parts"} +} + +test_a_from_line_is_printed_the_canonical_way if { + subjects := {"d": {"from": ` deployments whose environment is "prod" and approved_by exists `}} + _listed(subjects, "d").d.from == `deployments whose environment is "prod" and approved_by exists` +} + +test_a_from_line_that_names_each_key_is_printed_with_it if { + subjects := {"d": {"from": "trail.artifacts named by each of $params.names"}} + _listed(subjects, "d").d.from == "trail.artifacts named by each of $params.names" +} + +test_a_subject_that_narrows_another_is_listed_with_the_one_it_narrows if { + subjects := { + "round": {"from": "rounds", "identified_by": "id", "description": "A round"}, + "done": {"from": `round whose state is "done"`}, + "other": {"from": "others"}, + } + _listed(subjects, "done") == { + "done": {"description": "", "from": `round whose state is "done"`, "identified_by": "id", "parts": {}}, + "round": {"description": "A round", "from": "rounds", "identified_by": "id", "parts": {}}, + } +} + +test_a_chain_of_narrowing_subjects_lists_every_subject_it_builds_on if { + subjects := { + "defect": {"from": "defects", "identified_by": "id"}, + "critical": {"from": `defect whose severity is "critical"`}, + "open critical": {"from": `critical whose state is "open"`}, + } + object.keys(_listed(subjects, "open critical")) == {"defect", "critical", "open critical"} + _listed(subjects, "open critical")["open critical"].identified_by == "id" +} + +test_a_subject_read_from_another_is_listed_with_its_parent if { + rep := _chain_report(_chain_doc, _chain_subjects, {"c": "approved is true"}, {"name": "web"}) + rep.subjects == { + "artifact": {"description": "", "from": "trail.artifacts named by $params.name", "identified_by": "fingerprint", "parts": {}}, + "pull request": {"description": "", "from": "artifact.pull_requests", "identified_by": "url", "parts": {}}, + } +} + +test_a_parent_that_is_not_named_by_a_requirement_is_still_listed if { + rep := _chain_report(_chain_doc, _chain_subjects, {"c": "approved is true"}, {}) + object.keys(rep.subjects) == {"artifact", "pull request"} +} + +test_a_subject_in_the_old_syntax_prints_its_path_and_its_filters_as_sentences if { + subjects := {"d": {"from": ["trail", "deployments"], "identified_by": ["id"], "applies_to": { + "prod": {"op": "equals", "path": ["environment"], "value": "prod"}, + "approved": {"op": "non_empty_string", "path": ["approved_by"], "description": "Someone approved it"}, + }}} + _listed(subjects, "d").d == {"description": "", "from": `trail.deployments whose approved_by is a non-empty string and environment is "prod"`, "identified_by": "id", "parts": {}} +} + +test_an_old_subject_without_filters_prints_just_its_path if { + _listed({"d": {"from": ["deployments"], "applies_to": {}}}, "d").d.from == "deployments" + _listed({"d": {"from": [], "identified_by": ["id"]}}, "d").d.from == "$input" +} + +test_an_old_subject_that_names_each_key_prints_it if { + subjects := {"d": {"from": ["trail", "artifacts", {"each_as": "subject", "keys": {"ref": ["$$params", "names"]}}]}} + _listed(subjects, "d").d.from == "trail.artifacts named by each of $params.names" +} + +test_every_old_naming_step_is_printed_and_one_with_no_sentence_form_shows_in_angle_brackets if { + steps := [ + [{"each_as": "it"}, "items "], + [{"each_as": "it", "keys": ["b", "a"]}, `items `], + [{"each_as": "it", "keys": {"literal": ["x"]}}, `items `], + [{"each_as": "it", "keys": [{"ref": ["$$params", "k"]}, "z"]}, `items `], + [{"each_as": "it", "keys": {"ref": ["$$params", "ks"]}}, "items named by each of $params.ks"], + ] + every pair in steps { + _listed({"d": {"from": ["items", pair[0]]}}, "d").d.from == pair[1] + } +} + +test_an_old_subject_built_on_another_prints_the_other_and_its_own_filters if { + subjects := { + "base": {"from": ["components"], "identified_by": ["name"]}, + "locked": {"of": "base", "applies_to": {"locked": {"op": "present", "path": ["lock_release"]}}}, + } + _listed(subjects, "locked") == { + "base": {"description": "", "from": "components", "identified_by": "name", "parts": {}}, + "locked": {"description": "", "from": "base whose lock_release exists", "identified_by": "name", "parts": {}}, + } +} + +test_a_filter_that_cannot_be_a_sentence_is_shown_by_its_name_and_never_dropped if { + subjects := {"d": {"from": ["ds"], "applies_to": {"a": {"op": "present", "path": ["x"]}, "b": {"op": "even", "path": ["n"]}, "c": {"op": "all", "path": ["xs"], "check": {"op": "present", "path": ["y"]}}}}} + _listed(subjects, "d").d.from == "ds whose x exists and and " +} + +test_an_applies_to_that_is_not_an_object_is_shown_as_invalid if { + every bad in ["x", 1, [], null] { + _listed({"d": {"from": ["ds"], "applies_to": bad}}, "d").d.from == "ds whose " + } +} + +test_a_subject_without_from_reads_the_input_field_named_like_it if { + _listed({"deployments": {}}, "deployments").deployments.from == "deployments" + _listed({"deployments": null}, "deployments").deployments.from == "deployments" + _listed({"my deployments": {"identified_by": "id"}}, "my deployments")["my deployments"].from == `"my deployments"` +} + +test_a_subject_without_identified_by_has_none if { + every subjects in [{"d": {"from": "ds"}}, {"d": {"from": ["ds"]}}, {"d": {"from": ["ds"], "identified_by": []}}, {"d": null}] { + _listed(subjects, "d").d.identified_by == null + } +} + +test_an_id_written_in_the_old_syntax_is_printed_as_a_sentence_path if { + _listed({"d": {"from": ["ds"], "identified_by": ["build", "number"]}}, "d").d.identified_by == "build.number" + _listed({"d": {"from": "ds", "identified_by": "build.number"}}, "d").d.identified_by == "build.number" +} + +test_a_from_written_wrong_still_gets_an_entry_marked_invalid if { + every from in ["???", "ds whose", "", 1, true, null, [true], [["a"]], [1.5]] { + subjects := {"d": {"from": from, "identified_by": "id"}} + _listed(subjects, "d").d.from == "" + } +} + +test_a_from_line_with_applies_to_or_of_is_marked_invalid if { + _listed({"d": {"from": "ds", "applies_to": {"a": {"op": "present", "path": ["x"]}}}}, "d").d.from == "" + _listed({"b": {"from": "bs"}, "d": {"from": "ds", "of": "b"}}, "d").d.from == "" +} + +test_a_subject_built_on_one_that_is_missing_or_loops_is_marked_invalid if { + _listed({"d": {"of": "nope"}}, "d").d.from == "" + _listed({"d": {"of": "d"}}, "d").d.from == "" + every name in ["a", "b"] { + _listed({"a": {"of": "b"}, "b": {"of": "a"}}, name)[name].from == "" + } +} + +test_an_id_written_wrong_is_marked_invalid if { + every id in ["???", 1, true] { + _listed({"d": {"from": ["ds"], "identified_by": id}}, "d").d.identified_by == "" + } +} + +test_an_id_with_a_step_that_cannot_be_a_key_shows_the_step_as_invalid if { + _listed({"d": {"from": ["ds"], "identified_by": [[1]]}}, "d").d.identified_by == "" +} + +test_a_subject_that_is_not_an_object_still_gets_an_entry if { + every def in ["ds", 1, true, ["ds"]] { + _listed({"d": def}, "d").d == {"description": "", "from": "", "identified_by": null, "parts": {}} + } +} + +test_a_subject_description_that_is_null_or_the_wrong_type_is_empty if { + every description in [null, 1, true, [], {}] { + _listed({"d": {"from": "ds", "description": description}}, "d").d.description == "" + } + _listed({"d": {"from": "ds"}}, "d").d.description == "" +} + +test_a_subject_no_requirement_uses_is_not_listed if { + subjects := {"used": {"from": "us"}, "unused": {"from": "others"}, "broken": {"from": "???"}} + object.keys(_listed(subjects, "used")) == {"used"} +} + +test_a_subject_nothing_builds_on_or_reads_from_is_not_listed_even_when_it_looks_related if { + subjects := {"round": {"from": "rounds"}, "done": {"from": "round whose state is \"done\""}} + object.keys(_listed(subjects, "round")) == {"round"} +} + +test_no_subject_is_listed_when_no_requirement_names_one if { + ergo.report({}, {"subjects": {"d": {"from": "ds"}}, "requirements": {"r": {"checks": {"c": "id exists"}}}}, {}).subjects == {} + ergo.report({}, {"subjects": {"d": {"from": "ds"}}}, {}).subjects == {} + ergo.report({}, {"subjects": {"d": {"from": "ds"}}, "requirements": {"r": {"subject": "nope", "checks": {"c": "id exists"}}}}, {}).subjects == {} +} + +test_a_subjects_section_that_is_not_an_object_lists_nothing if { + every subjects in [null, true, 1, "d", []] { + ergo.report({}, {"subjects": subjects, "requirements": _listed_reqs("d")}, {}).subjects == {} + } +} + +test_a_requirement_reports_the_name_of_its_subject if { + rep := ergo.report({}, {"subjects": {"d": {"from": "ds"}, "e": {"from": "es"}}, "requirements": {"a": {"subject": "d", "checks": {"c": "id exists"}}, "b": {"subject": "e", "checks": {"c": "id exists"}}}}, {}) + {name: entry.subject | some name, entry in rep.requirements} == {"a": "d", "b": "e"} +} + +test_a_requirement_reports_no_subject_when_it_names_none_or_one_that_is_not_there if { + every req in [{"checks": {"c": "id exists"}}, {"subject": "nope", "checks": {"c": "id exists"}}, {"subject": 1, "checks": {"c": "id exists"}}, {"subject": null, "checks": {"c": "id exists"}}, {"subject": ["d"], "checks": {"c": "id exists"}}] { + ergo.report({}, {"subjects": {"d": {"from": "ds"}}, "requirements": {"r": req}}, {}).requirements.r.subject == null + } + ergo.report({}, {"subjects": [], "requirements": _listed_reqs("d")}, {}).requirements.r.subject == null +} + +test_a_requirement_that_names_a_subject_written_wrong_still_reports_its_name if { + ergo.report({}, {"subjects": {"d": 1}, "requirements": _listed_reqs("d")}, {}).requirements.r.subject == "d" +} + +test_the_subjects_section_is_independent_of_the_order_the_policy_is_written_in if { + first := {"round": {"from": "rounds", "identified_by": "id"}, "done": {"from": `round whose state is "done"`}, "old": {"from": ["ds"], "applies_to": {"b": {"op": "present", "path": ["b"]}, "a": {"op": "present", "path": ["a"]}}}} + reqs := {"x": {"subject": "done", "checks": {"c": "id exists"}}, "y": {"subject": "old", "checks": {"c": "id exists"}}} + one := ergo.report({}, {"subjects": first, "requirements": reqs}, {}) + two := ergo.report({}, {"requirements": {"y": reqs.y, "x": reqs.x}, "subjects": {"old": {"applies_to": {"a": {"op": "present", "path": ["a"]}, "b": {"op": "present", "path": ["b"]}}, "from": ["ds"]}, "done": first.done, "round": first.round}}, {}) + json.marshal(one) == json.marshal(two) + one.subjects.old.from == "ds whose .a exists and b exists" +} + +test_a_from_that_cannot_be_printed_is_invalid_even_when_nothing_flagged_it if { + every def in [{"from": "ds", "applies_to": {}}, {"from": "ds", "of": "b"}, {"from": 1}, {"of": 1}] { + ergo._subject_from_text("d", def) == "" with data.ergo_resolved as {"d": {"problems": set()}} with data.ergo_subjects as {"d": def, "b": {}} + } +} + +test_old_filters_are_printed_as_the_sentence_each_operator_reads_as if { + filters := { + "a": {"op": "present", "path": ["x"]}, + "b": {"op": "missing", "path": ["x"]}, + "c": {"op": "in", "path": ["x"], "values": ["p", "q"]}, + "d": {"op": "range", "path": ["x"], "min": 1, "max": 5}, + "e": {"op": "includes", "path": ["x"], "value": "p"}, + "f": {"op": "excludes", "path": ["x"], "values": ["p", "q"]}, + "g": {"op": "compare", "left": ["x"], "right": ["y"], "cmp": "gte"}, + "h": {"op": "compare_time", "left": ["x"], "right": ["y"], "cmp": "lt"}, + "i": {"op": "matches_any", "path": ["x"], "patterns": ["^a"]}, + } + _listed({"d": {"from": ["ds"], "applies_to": filters}}, "d").d.from == `ds whose x exists and x does not exist and x is one of "p", "q" and x is between 1 and 5 and x contains "p" and x contains none of "p", "q" and x is at least y and x is before y and x matches "^a"` +} + +test_a_filter_with_a_field_its_sentence_would_lose_is_shown_by_its_name if { + filters := { + "a": {"op": "present", "path": ["x"], "substitute": {"op": "present", "path": ["y"]}}, + "b": {"op": "present", "path": ["x"], "zz": 1}, + "c": {"op": "present", "path": "x"}, + "d": {"op": "equals", "path": ["x"]}, + "e": {"op": ["present"], "path": ["x"]}, + "f": 1, + "g": {"op": "present", "path": ["x"], "description": "kept out", "meta": {"k": 1}}, + } + _listed({"d": {"from": ["ds"], "applies_to": filters}}, "d").d.from == "ds whose and and and and and and x exists" +} diff --git a/spec/cases/subjects/cases.json b/spec/cases/subjects/cases.json index 1f36e89..b618735 100644 --- a/spec/cases/subjects/cases.json +++ b/spec/cases/subjects/cases.json @@ -3717,5 +3717,392 @@ ] } ] + }, + { + "description": "the report lists the subjects the requirements use", + "policy": { + "subjects": { + "deployment": { + "from": " deployments whose environment is \"prod\" ", + "identified_by": "id", + "description": "A deployment to production" + }, + "release": { + "from": "releases", + "identified_by": "id" + } + }, + "requirements": { + "s": { + "subject": "deployment", + "checks": { + "c": "id exists" + } + } + } + }, + "cases": [ + { + "description": "lists a subject with its description, its from line printed the same way whatever the spaces, and its id", + "rules": [ + "report.subjects" + ], + "input": {}, + "report": { + "subjects": { + "deployment": { + "description": "A deployment to production", + "from": "deployments whose environment is \"prod\"", + "identified_by": "id", + "parts": {} + } + } + }, + "results": [] + }, + { + "description": "names the subject in the requirement entry, and lists no subject that no requirement uses", + "rules": [ + "report.requirements", + "report.subjects" + ], + "input": {}, + "report": { + "subjects": { + "deployment": { + "description": "A deployment to production", + "from": "deployments whose environment is \"prod\"", + "identified_by": "id", + "parts": {} + } + } + }, + "requirements": { + "s": { + "subject": "deployment" + } + }, + "results": [] + } + ] + }, + { + "description": "a subject that narrows another", + "policy": { + "subjects": { + "round": { + "from": "rounds", + "identified_by": "id", + "description": "A round" + }, + "done": { + "from": "round whose state is \"done\"" + }, + "other": { + "from": "others" + } + }, + "requirements": { + "s": { + "subject": "done", + "checks": { + "c": "id exists" + } + } + } + }, + "cases": [ + { + "description": "lists both subjects, and the narrowing one has the id of the one it narrows", + "rules": [ + "report.subjects" + ], + "input": {}, + "report": { + "subjects": { + "done": { + "description": "", + "from": "round whose state is \"done\"", + "identified_by": "id", + "parts": {} + }, + "round": { + "description": "A round", + "from": "rounds", + "identified_by": "id", + "parts": {} + } + } + }, + "results": [] + } + ] + }, + { + "description": "a subject read from another subject", + "policy": { + "subjects": { + "artifact": { + "from": "trail.artifacts", + "identified_by": "fingerprint" + }, + "pull request": { + "from": "artifact.pull_requests", + "identified_by": "url" + } + }, + "requirements": { + "s": { + "subject": "pull request", + "checks": { + "c": "id exists" + } + } + } + }, + "cases": [ + { + "description": "lists the parent it is read from too", + "rules": [ + "report.subjects" + ], + "input": {}, + "report": { + "subjects": { + "artifact": { + "description": "", + "from": "trail.artifacts", + "identified_by": "fingerprint", + "parts": {} + }, + "pull request": { + "description": "", + "from": "artifact.pull_requests", + "identified_by": "url", + "parts": {} + } + } + }, + "results": [] + } + ] + }, + { + "description": "subjects written as lists", + "policy": { + "subjects": { + "base": { + "from": [ + "components" + ], + "identified_by": [ + "name" + ] + }, + "locked": { + "of": "base", + "applies_to": { + "locked": { + "op": "present", + "path": [ + "lock_release" + ] + }, + "exempt": { + "op": "equals", + "path": [ + "exempt" + ], + "value": false + } + } + }, + "odd": { + "from": [ + "things" + ], + "applies_to": { + "b": { + "op": "present", + "path": [ + "b" + ] + }, + "a": { + "op": "even", + "path": [ + "n" + ] + } + } + } + }, + "requirements": { + "s": { + "subject": "locked", + "checks": { + "c": "id exists" + } + }, + "t": { + "subject": "odd", + "checks": { + "c": "id exists" + } + } + } + }, + "cases": [ + { + "description": "prints the path and the conditions sorted by name as a sentence, and shows a condition that is not a sentence by its name", + "rules": [ + "report.subjects" + ], + "input": {}, + "report": { + "subjects": { + "base": { + "description": "", + "from": "components", + "identified_by": "name", + "parts": {} + }, + "locked": { + "description": "", + "from": "base whose exempt is false and lock_release exists", + "identified_by": "name", + "parts": {} + }, + "odd": { + "description": "", + "from": "things whose and b exists", + "identified_by": null, + "parts": {} + } + } + }, + "results": [] + } + ] + }, + { + "description": "a subject with no from and no identified_by", + "policy": { + "subjects": { + "deployments": {} + }, + "requirements": { + "s": { + "subject": "deployments", + "checks": { + "c": "id exists" + } + } + } + }, + "cases": [ + { + "description": "reads the input field named like it, and has no id path", + "rules": [ + "report.subjects" + ], + "input": {}, + "report": { + "subjects": { + "deployments": { + "description": "", + "from": "deployments", + "identified_by": null, + "parts": {} + } + } + }, + "results": [] + } + ] + }, + { + "description": "a subject written wrong", + "policy": { + "subjects": { + "broken": { + "from": "deployments whose", + "description": 3 + } + }, + "requirements": { + "s": { + "subject": "broken", + "checks": { + "c": "id exists" + } + } + } + }, + "cases": [ + { + "description": "still has its entry, with and an empty description", + "rules": [ + "report.subjects" + ], + "input": {}, + "report": { + "subjects": { + "broken": { + "description": "", + "from": "", + "identified_by": null, + "parts": {} + } + } + }, + "results": [] + } + ] + }, + { + "description": "a requirement with no subject, or one that isn't there", + "policy": { + "subjects": { + "deployment": { + "from": "deployments" + } + }, + "requirements": { + "s": { + "checks": { + "c": "id exists" + } + }, + "t": { + "subject": "nope", + "checks": { + "c": "id exists" + } + } + } + }, + "cases": [ + { + "description": "has subject null, and no subject is listed", + "rules": [ + "report.requirements", + "report.subjects" + ], + "input": {}, + "report": { + "subjects": {} + }, + "requirements": { + "s": { + "subject": null + }, + "t": { + "subject": null + } + }, + "results": [] + } + ] } ] \ No newline at end of file diff --git a/spec/report/schema.json b/spec/report/schema.json index 6e962a1..799cbbd 100644 --- a/spec/report/schema.json +++ b/spec/report/schema.json @@ -7,12 +7,28 @@ "description": {"type": "string"}, "meta": {"type": "object"}, "requirements": {"type": "object", "additionalProperties": {"$ref": "#/$defs/requirement"}}, - "results": {"type": "array", "items": {"$ref": "#/$defs/row"}} + "results": {"type": "array", "items": {"$ref": "#/$defs/row"}}, + "subjects": { + "type": "object", + "description": "The subjects the requirements use, directly or through the subjects they narrow or read from, by name.", + "additionalProperties": {"$ref": "#/$defs/subject"} + } }, - "required": ["compliant", "description", "meta", "requirements", "results"], + "required": ["compliant", "description", "meta", "requirements", "results", "subjects"], "additionalProperties": false, "$defs": { "cause": {"enum": ["satisfied", "substituted", "ill_formed", "not_an_object", "ambiguous", "unmatched", "unusable", "absent", "null", "value"]}, + "subject": { + "type": "object", + "properties": { + "description": {"type": "string", "description": "The subject's description, or \"\" when it has none."}, + "from": {"type": "string", "description": "How the subject is found, as a from line is written, or when it is written wrong."}, + "identified_by": {"type": ["string", "null"], "description": "The path of the subject's id, or null when it has none and the whole subject is its id."}, + "parts": {"type": "object", "description": "Always empty for now."} + }, + "required": ["description", "from", "identified_by", "parts"], + "additionalProperties": false + }, "requirement": { "type": "object", "properties": { @@ -47,6 +63,7 @@ "meta": {"type": "object"}, "require": {"description": "The require the requirement was written with, or every by default."}, "status": {"enum": ["met", "not_met", "not_applicable"]}, + "subject": {"type": ["string", "null"], "description": "The name of the subject the requirement checks, or null when it names none or one that isn't in the policy's subjects."}, "subjects": { "type": "object", "properties": { @@ -57,7 +74,7 @@ "additionalProperties": false } }, - "required": ["causes", "checks", "description", "meta", "require", "status", "subjects"], + "required": ["causes", "checks", "description", "meta", "require", "status", "subject", "subjects"], "additionalProperties": false }, "input": { diff --git a/spec/semantics.md b/spec/semantics.md index e8912f2..9ee8371 100644 --- a/spec/semantics.md +++ b/spec/semantics.md @@ -407,10 +407,19 @@ A policy has two sections. `subjects` defines each kind of subject once, and `re **[compliant]** The report is `compliant` when it has at least one requirement and none is `not_met`. A policy with no requirements is never compliant, because it doesn't check anything. -**[report.requirements]** The report has an entry for each requirement, with its `description` and `meta`, or `""` and `{}` when it has none, its `require`, its `status`, `subjects` with `total`, the number found by `from`, and `matching`, the number in scope, `checks`, each check as written with its `description`, `meta` and `expression`, plus the checks ergo adds, and `causes`. All the filters share one `$applies` entry. +**[report.requirements]** The report has an entry for each requirement, with its `description` and `meta`, or `""` and `{}` when it has none, its `require`, `subject`, the name of the subject it checks, or `null` when it names none or one that isn't in the policy's subjects, its `status`, `subjects` with `total`, the number found by `from`, and `matching`, the number in scope, `checks`, each check as written with its `description`, `meta` and `expression`, plus the checks ergo adds, and `causes`. All the filters share one `$applies` entry. **[report.policy]** The report has the policy's `description` and `meta` at its top, next to `compliant`, or `""` and `{}` when the policy has none. `null` counts as none. A `description` that isn't a string, or a `meta` that isn't an object or holds something JSON can't, makes the `$well_formed` row of every requirement fail, because the policy belongs to all of them, and the report shows `""` or `{}` in its place. +**[report.subjects]** The report has `subjects` at its top, with an entry for each subject a requirement uses, by name. That's the subject it names, the subjects that one narrows or reads from, and so on all the way back. A subject no requirement uses isn't listed. An entry has: + +- `description`, the subject's description, or `""` when it has none or it isn't a string +- `from`, how the subject is found, written as a `from` line: the line itself, printed the same way whatever spaces it was written with, or for a subject written as a list, its path followed by `whose` and its conditions, which are sorted by name. A naming step at the end of the path is shown as `named by each of ` when its keys come from a ref, as `` when they're listed, and as `` when it has none. Anything with no sentence form is shown in angle brackets, never left out: a condition that can't be written as a sentence is shown as ``, and `applies_to` that isn't an object as ``. A subject without `from` reads the input field named like it. A `from` that is written wrong is ``. +- `identified_by`, the path of the subject's id, or `null` when it has none, because the whole subject is its id. A subject that narrows another has the id of the one it narrows. An id that is written wrong is ``. +- `parts`, always `{}` for now + +A subject that is written wrong still has its entry. + **[report.causes]** `causes` lists the causes of the rows that would be [violations](#violations) of the requirement, each with `rows`, the number of rows that have it. It counts rows, not subjects, and lists the causes in the order of [causes](#causes), worst first. A requirement that's `met` or `not_applicable` has an empty list. ## Row order