diff --git a/SECURITY.md b/SECURITY.md index 78792ab8f..1db8ac6b4 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -19,6 +19,28 @@ RustGIS is released often and fixes go into the next release; older releases don patches. Check that the problem still happens with the [latest release](https://github.com/opengeos/RustGIS/releases/latest) or `main`. +## Sharing projects and generated items + +Saving removes credentials from supported address fields and recognized executable parameters, +and reports counted removals. Feature data, user notes, foreign-language notebook code and opaque +plugin content can still contain sensitive information. Data-valued command parameters, such as +imported feature text and expressions, are preserved too. Supplied service documents are kept +opaque, with an unchecked-content notice when recognizable credential material is found. +An unchecked-content notice means that content was preserved, not verified safe. Review it +before sharing; saving is not a general secret scanner. + +For an older file or log that exposed a key, revoke or rotate it with its service. Saving again +cleans only recognized fields and cannot recall shared copies. For a Jupyter notebook, also clear +saved widget state and save again; printed cell output and user data require your own review. + +## Active-login output redaction + +While a service login is active, generated output masks its known secrets throughout the +process. A very short password can therefore also mask ordinary words, names or addresses, +including output from another session. This changes displayed output, not the stored data, +requests or permission checks. RustGIS does not retain revoked secrets just to recognize them +in later output; review that output before sharing it. + ## What counts as a security problem RustGIS makes promises about what untrusted plugins, files, agents and web pages can do. A way diff --git a/apps/rustgis-python/python/rustgis/_embed.py b/apps/rustgis-python/python/rustgis/_embed.py index 5f79c7af1..d2c04867d 100644 --- a/apps/rustgis-python/python/rustgis/_embed.py +++ b/apps/rustgis-python/python/rustgis/_embed.py @@ -67,6 +67,8 @@ def _start( # Notices waiting to be read. The app's messages may arrive on another thread (Jupyter's). self._held: List[Tuple[str, bool]] = [] self._held_lock = threading.Lock() + self._sent_keys = 0 + self._sent_notices: List[str] = [] if sample: self._session.run("project.sample") if data is not None: @@ -201,7 +203,14 @@ def to_png( def _push(self) -> None: """Sends the session's project to the app, when it changed.""" try: - text = self._session.project_json() + text, keys, notices = self._session._engine._peer_text() + # A new loss may leave the public project text unchanged. + if text != self._project or notices != self._sent_notices: + self._hold([(notice, False) for notice in notices]) + self._sent_notices = notices + if keys and (text != self._project or keys > self._sent_keys): + self._hold([(f"{keys} layer address(es) went to the map widget without their keys, tokens, passwords or URL fragments; restore missing values if needed", False)]) + self._sent_keys = keys except RustGISError as e: self._hold([(f"couldn't show the project in the app: {e}", True)]) return @@ -232,16 +241,15 @@ def _from_app(self, content: Any) -> None: text = content.get("text") if not isinstance(text, str) or text == self._project: return - # Opening it reports "Opened ...": not news here, so it is dropped, and what was - # waiting to be read is kept. + # Peer input is an import: keep its containment notices and invalidate old trust. self._hold(self._session.notices()) try: - self._session.run("project.openText", text=text) + self._session._engine._open_peer_text(text) except RustGISError as e: self._hold([(f"the app's project couldn't be read: {e}", True)]) return finally: - self._session.notices() + self._hold(self._session.notices()) # The app's own text, not the session's re-serialised copy: the app then sees its # project and doesn't open it again. self._project = text diff --git a/apps/rustgis-python/src/lib.rs b/apps/rustgis-python/src/lib.rs index a05dd9b17..f21afdaaf 100644 --- a/apps/rustgis-python/src/lib.rs +++ b/apps/rustgis-python/src/lib.rs @@ -180,9 +180,41 @@ impl Engine { } /// The project as `.rgis` JSON text, as `project.saveAs` writes it (what `project.openText` - /// opens). + /// opens). This save-equivalent output retains a bounded same-process restore snapshot; + /// repeated identical text replaces that snapshot. Unfinished imports are refused. fn project_json(&self, py: Python<'_>) -> PyResult { - let result = self.with_session(py, |session| serde_json::to_string(&session.project).map_err(|e| e.to_string())); + let result = self.with_session(py, |session| { + let (text, report) = session.project.saved_text(None)?; + for notice in report.notices() { + session.notify(notice, false); + } + Ok::<_, String>(text) + }); + self.flush(py); + result.map_err(RustGISError::new_err) + } + + /// Private bridge output: never remembered, and never includes keys in widget state. + fn _peer_text(&self, py: Python<'_>) -> PyResult<(String, u32, Vec)> { + let result = self.with_session(py, |session| { + let (text, report) = session.peer_text(rustgis_engine::Peer::Bridge)?; + let addresses = report.layer_addresses(); + let notices = report.notices_without_layer_addresses(); + Ok::<_, String>((text, addresses, notices)) + }); + self.flush(py); + result.map_err(RustGISError::new_err) + } + + /// Private bridge ingress bypasses the saved-text cache and creates no User authority. + fn _open_peer_text(&self, py: Python<'_>, text: String) -> PyResult<()> { + let result = self.with_session(py, |session| { + let report = session.open_peer_text(rustgis_engine::Peer::Bridge, &text)?; + for notice in report.notices() { + session.notify(notice, false); + } + Ok::<_, String>(()) + }); self.flush(py); result.map_err(RustGISError::new_err) } diff --git a/apps/rustgis-python/tests/test_widget.py b/apps/rustgis-python/tests/test_widget.py index 888ef3e42..f8c608a88 100644 --- a/apps/rustgis-python/tests/test_widget.py +++ b/apps/rustgis-python/tests/test_widget.py @@ -98,3 +98,90 @@ def test_layers_of_a_map_change_the_app(): sites.rename("Places") assert m._project != before and "Places" in m._project assert m.layer_names() == ["Places"] + + +def test_widget_exchange_never_sends_keys_and_checks_the_current_slot(): + """The real receiving method keeps only keys still present on the last pushed slot.""" + secret = "RGSYNTH_widget_20261103" + s = rustgis.Session() + s.add_geojson(POINT, name="Sites") + project = json.loads(s.project_json()) + project["maps"][0]["layers"][0]["dataset"]["source"] = f"https://h.invalid/a?token={secret}" + s.run("project.openText", text=json.dumps(project)) + saved = s.project_json() + assert secret not in saved + other = rustgis.Session() + other.run("project.openText", text=saved) + assert other._engine._peer_text()[1] == 1 + m = rustgis.Map(session=s) + assert secret not in m._project + assert any("without their keys" in notice for notice, _ in m.notices()) + before = m._project + m.run("info.project") + assert m._project == before + assert m.notices() == [] + # A different protected source still has one omitted address and needs a fresh warning. + project = json.loads(m._project) + project["maps"][0]["layers"][0]["dataset"]["source"] = f"https://h.invalid/b?token={secret}" + s.run("project.openText", text=json.dumps(project)) + m.run("info.project") + assert any("without their keys" in notice for notice, _ in m.notices()) + m.run("info.project") + assert m.notices() == [] + incoming = json.loads(m._project) + incoming["maps"][0]["layers"][0]["name"] = "Styled by app" + from_app(m, json.dumps(incoming)) + assert m.session._engine._peer_text()[1] == 1 + assert not any("lost their keys" in notice for notice, _ in m.notices()) + # Direct session changes do not call the widget's push. Its stale record cannot resurrect + # the removed source when the app sends the old slot again. + m.session.run("layer.remove", layer="Styled by app") + incoming["maps"][0]["layers"][0]["name"] = "Stale app slot" + from_app(m, json.dumps(incoming)) + assert m.session._engine._peer_text()[1] == 0 + + for command, params, kind in [ + ("connection.login", {"apiKey": "demo"}, "executable step(s)"), + ("plugin.unknown", {"nested": {"token": "demo"}}, "opaque parameters"), + ]: + s = rustgis.Session() + project = json.loads(s.project_json()) + clean = {"apiKey": ""} if command == "connection.login" else {"nested": {}} + project["tasks"] = [{"name": "Notice fixture", "steps": [{"title": "Stored only", "command": command, "params": clean}]}] + s.run("project.openText", text=json.dumps(project)) + s.notices() + m = rustgis.Map(session=s) + assert m.notices() == [], "clean bridge output needs no loss notice" + before = m._project + project["tasks"][0]["steps"][0]["params"] = params + s.run("project.openText", text=json.dumps(project)) + s.notices() + m.run("info.project") + if command == "connection.login": + assert m._project == before, "a newly removed credential need not change public project text" + notices = m.notices() + assert len(notices) == 1 and kind in notices[0][0] and not notices[0][1] + before = m._project + m.run("info.project") + m.run("info.project") + assert m._project == before + assert m.notices() == [], "unchanged internal pushes must not repeat bridge notices" + m.run("layer.template", template="points", name="Changed output") + assert m._project != before + assert m.notices() == notices, "changed outgoing content reports its losses again" + m.run("info.project") + assert m.notices() == [] + + +def test_widget_repeated_table_id_cannot_gain_a_retained_key(): + """The receiving census sees duplicate table ids before project normalization renumbers them.""" + s = rustgis.Session() + project = json.loads(s.project_json()) + project["tables"] = [{"id":7, "name":"Private table", "geometry_type":"None", "fields":[], "features":[], "source":"https://h.invalid/table?token=RGSYNTH_table_20261103"}] + s.run("project.openText", text=json.dumps(project)) + m = rustgis.Map(session=s) + incoming = json.loads(m._project) + incoming["tables"].append(dict(incoming["tables"][0])) + from_app(m, json.dumps(incoming)) + assert s._engine._peer_text()[1] == 0 + assert any("lost their keys" in notice for notice, _ in m.notices()) diff --git a/apps/rustgis-web/src/embed.rs b/apps/rustgis-web/src/embed.rs index dbe30f7c4..1adf6c692 100644 --- a/apps/rustgis-web/src/embed.rs +++ b/apps/rustgis-web/src/embed.rs @@ -17,6 +17,12 @@ //! | `{"rustgis": "project", "text": }` | the project changed here (an edit, a pan) | //! | `{"rustgis": "result", "id", "ok": true, "value"}` or `{…, "ok": false, "error"}` | a `run`'s outcome | //! +//! Exchanged projects remove recognized credentials from supported configuration fields. +//! Preserved user data, foreign notebook source and opaque parameters may still contain secrets. +//! Incoming projects use peer/import semantics: no identical-save restore, item trust or earlier +//! approvals. Parent origin checks identify the peer; they grant no user authority. The existing +//! `ui.*` command policy is separate from this project-exchange contract. +//! //! Only messages from the parent window are taken, and the app answers only the origin that sent //! the first of them. Any page may frame the app, so `run` takes only the interface's commands //! (`ui.*`: panes, the language, the theme); everything else changes the project, which the page diff --git a/apps/rustgis-web/src/web.rs b/apps/rustgis-web/src/web.rs index e80740dae..7d639e163 100644 --- a/apps/rustgis-web/src/web.rs +++ b/apps/rustgis-web/src/web.rs @@ -395,7 +395,7 @@ impl Embed { let queued = std::mem::take(&mut *self.queue.borrow_mut()); for value in queued { match parse(&value) { - Some(Msg::Project(text)) => match app.run("project.openText", serde_json::json!({"text": text})) { + Some(Msg::Project(text)) => match app.open_peer_text(&text) { Ok(_) => self.changes.seen(key(&app.session)), Err(e) => app.session.notify(e, true), }, @@ -411,9 +411,14 @@ impl Embed { } } match self.changes.due(key(&app.session), web_time::Instant::now()) { - (true, _) => match serde_json::to_string(&app.session.project) { - Ok(text) => self.answer(&crate::embed::project(&text)), - Err(e) => log::error!("couldn't send the project: {e}"), + (true, _) => match app.session.peer_text(rustgis_engine::Peer::Embedder) { + Ok((text, report)) => { + self.answer(&crate::embed::project(&text)); + for notice in report.notices() { + app.session.notify(notice, false); + } + } + Err(e) => app.session.notify(e, true), }, (false, Some(wait)) => ctx.request_repaint_after(wait), (false, None) => {} @@ -751,7 +756,7 @@ async fn fetch_bytes(url: &str, capture: Capture, omit: &'static [u16]) -> Resul } .await; logins::check(bound).map_err(str::to_string)?; - result.map_err(|error| if login.is_some() { logins::scrub(&format!("{url}: {error}")) } else { error }) + result.map_err(|error| if login.is_some() { logins::scrub_error(&format!("{url}: {error}")) } else { error }) } .await; diagnostics.record(answer.map_err(|e| Failure::new(url, e))).map_err(|failure| failure.message) @@ -815,7 +820,7 @@ async fn post_json(url: &str, body: &str, capture: Capture) -> Result, S } .await; logins::check(url).map_err(str::to_string)?; - result.map_err(|error| if login.is_some() { logins::scrub(&format!("{url}: {error}")) } else { error }) + result.map_err(|error| if login.is_some() { logins::scrub_error(&format!("{url}: {error}")) } else { error }) } .await; diagnostics.record(answer.map_err(|e| Failure::new(url, e))).map_err(|failure| failure.message) @@ -884,7 +889,7 @@ async fn head(url: &str, capture: Capture) -> Result, took: Duration, .filter(|m| m.get("method").and_then(Value::as_str) == Some("tools/call") && m.get("id").is_some()) .map(|m| { let params = m.get("params").cloned().unwrap_or(Value::Null); - let args = redact(params.get("arguments").cloned().unwrap_or(Value::Null), 0); + let args = + normalized_args(params.get("name").and_then(Value::as_str).unwrap_or(""), params.get("arguments").unwrap_or(&Value::Null)); let r = replies.iter().find(|r| r.get("id") == m.get("id")); let failed = r.is_none_or(|r| r.get("error").is_some() || r.pointer("/result/isError") == Some(&Value::Bool(true))); let error = failed.then(|| { @@ -138,56 +139,28 @@ pub fn entries(source: &str, request: &str, reply: Option<&str>, took: Duration, .collect() } -/// The words that name a secret in a key's name, whole words and their plurals. -const SECRET_WORDS: &[&str] = &[ - "passphrase", - "jwt", - "privatekey", - "password", - "passwd", - "pwd", - "secret", - "token", - "apikey", - "auth", - "authorization", - "bearer", - "credential", - "signature", - "cookie", - "sas", - "dsn", -]; -/// Words before "key" that make it a secret key (`api_key`, `accessKey`), not a sort or join key. -const KEY_KINDS: &[&str] = &["api", "access", "private", "secret", "account", "sas", "subscription", "client"]; - -/// Whether a key's name names a secret: its words (split at `_`, `-`, `.` and camelCase), one of -/// [`SECRET_WORDS`] or its plural, or "key" after one of [`KEY_KINDS`] (or alone). -fn secret_key(name: &str) -> bool { - let mut words: Vec = Vec::new(); - let mut cur = String::new(); - let mut prev_lower = false; - for c in name.chars() { - if c == '_' || c == '-' || c == '.' || c == ' ' { - words.push(std::mem::take(&mut cur)); - prev_lower = false; - continue; - } - if c.is_ascii_uppercase() && prev_lower { - words.push(std::mem::take(&mut cur)); +/// Arguments as the executor reads them. Only this output copy is redacted; it is never +/// supplied to the executor, the permission planner or approval equality. +fn normalized_args(tool: &str, arguments: &Value) -> Value { + match tool { + "run_command" | "run_tool" => { + let params = match rustgis_engine::params_envelope(arguments.get("params")) { + Ok(params) => rustgis_engine::log_params(¶ms).into_log_value(), + Err(_) => json!("[parameters not JSON; withheld]"), + }; + let key = if tool == "run_command" { "command" } else { "tool" }; + let mut out = serde_json::Map::new(); + let id = match arguments.get(key) { + Some(Value::String(id)) => json!(rustgis_engine::log_identifier(id)), + other => redact(other.cloned().unwrap_or(Value::Null), 0), + }; + out.insert(key.into(), id); + out.insert("params".into(), params); + Value::Object(out) } - prev_lower = c.is_ascii_lowercase() || c.is_ascii_digit(); - cur.push(c.to_ascii_lowercase()); + "run" => json!({"script": redact_script(arguments.get("script").and_then(Value::as_str).unwrap_or(""))}), + _ => redact(arguments.clone(), 0), } - words.push(cur); - let words: Vec<&str> = words.iter().map(String::as_str).filter(|w| !w.is_empty()).collect(); - let is = |w: &str, s: &str| w == s || w.strip_suffix('s') == Some(s); - words.iter().enumerate().any(|(i, w)| { - SECRET_WORDS.iter().any(|s| is(w, s)) - // Run together without a separator (`accesstoken`, `clientsecret`, `dbpassword`). - || ["token", "secret", "password", "passphrase"].iter().any(|s| w.len() > s.len() && (w.ends_with(s) || w.ends_with(&format!("{s}s")))) - || (is(w, "key") && (words.len() == 1 || i.checked_sub(1).and_then(|j| words.get(j)).is_some_and(|p| KEY_KINDS.contains(p)))) - }) } /// A string argument without its secrets: an address (one word with a scheme, or a database @@ -215,7 +188,15 @@ fn redact_string(s: &str) -> String { // Text with an address in it (a query, a description): each address word that carries // credentials (a user and password, or a key or token in its query) loses them; the rest, // and the spacing, stay as they were. - let carries = |w: &str| address(w) && (w.contains('@') || mentions_secret(w) || w.to_ascii_lowercase().contains("key=")); + let carries = |w: &str| { + let q = w.split_once('?').map_or("", |(_, q)| q.split('#').next().unwrap_or(q)); + let names = rustgis_engine::data::address::QueryNames::of(q); + let secret_pair = q.split(['&', ';']).any(|pair| { + let name = pair.split_once('=').map_or(pair, |(k, _)| k); + rustgis_engine::data::address::name_class(name, &names) != rustgis_engine::data::address::NameClass::Plain + }); + address(w) && (w.contains('@') || mentions_secret(w) || secret_pair) + }; if t.split_whitespace().any(carries) { let mut out = String::with_capacity(s.len()); let mut word = String::new(); @@ -239,7 +220,7 @@ fn redact_string(s: &str) -> String { s.to_string() } -/// Whether free text names a secret. Narrower than [`SECRET_WORDS`]: a plain "key" in an error +/// Whether free text names a secret. Narrower than the structured-name list: a plain "key" in an error /// ("primary key") isn't one. fn mentions_secret(text: &str) -> bool { let lower = text.to_ascii_lowercase(); @@ -252,24 +233,31 @@ fn mentions_secret(text: &str) -> bool { /// a call's are, line by line, so one line that doesn't parse leaves the others redacted. A line /// whose parameters aren't JSON keeps only its command id. fn redact_script(script: &str) -> String { - script + let omitted = script.lines().skip(rustgis_engine::cmd::workflows::MAX_SCRIPT_LINES).count(); + let output = script .lines() + .take(rustgis_engine::cmd::workflows::MAX_SCRIPT_LINES) .map(|line| { let line = line.trim(); - if line.is_empty() || line.starts_with('#') { - return line.to_string(); - } - let (id, rest) = line.split_once(char::is_whitespace).unwrap_or((line, "")); - if rest.trim().is_empty() { - return id.to_string(); - } - match serde_json::from_str::(rest.trim()) { - Ok(params) => format!("{id} {}", redact(params, 0)), - Err(_) => format!("{id} [parameters not JSON; withheld]"), + // Display only: a malformed identifier may carry JSON glued after `=`. It must + // not be copied as an id; execution still uses the shared parser unchanged. + let head = line.split(|c: char| c.is_whitespace() || c == '=').next().unwrap_or(line); + let glued = line + .get(head.len()..) + .and_then(|r| r.strip_prefix('=')) + .is_some_and(|r| !r.is_empty() && !r.starts_with(char::is_whitespace)); + match rustgis_engine::script_line(line) { + None => redact_string(line), + Some(Ok((id, params))) if !glued => { + let id = rustgis_engine::log_identifier(&id); + if params.is_null() { id } else { format!("{id} {}", rustgis_engine::log_params(¶ms).into_log_value()) } + } + Some(_) => format!("{} [parameters not JSON; withheld]", rustgis_engine::log_identifier(head)), } }) .collect::>() - .join("\n") + .join("\n"); + if omitted > 0 { format!("# [{omitted} further script line(s) not recorded]\n{output}") } else { output } } /// `v` without what would give someone else the user's access: values under keys that name a @@ -283,7 +271,7 @@ fn redact(v: Value, depth: usize) -> Value { Value::Array(a) if depth < 16 => Value::Array(a.into_iter().map(|x| redact(x, depth + 1)).collect()), Value::Object(o) if depth < 16 => Value::Object( o.into_iter() - .map(|(k, x)| match (secret_key(&k), k.as_str(), x) { + .map(|(k, x)| match (rustgis_engine::data::address::log_sensitive(&k), k.as_str(), x) { (true, _, _) => (k, json!("[redacted]")), (false, "script", Value::String(s)) => (k, json!(redact_script(&s))), (false, _, x) => (k, redact(x, depth + 1)), @@ -329,6 +317,72 @@ fn prune(dir: &Path, secs: i64) { mod tests { use super::*; + #[test] + fn audit_logs_the_executed_envelope_and_shared_script_grammar() { + for line in [ + "sql.connect={\"password\":\"RGSYNTH_glued_audit\"}", + "sql.connect={\"password\":\"RGSYNTH_glued_audit\"} junk", + "sql.connect={\"password\":\"RGSYNTH_glued_audit\"", + ] { + let output = redact_script(line); + assert!(!output.contains("RGSYNTH_glued_audit"), "{output}"); + } + for id in [ + "sql.connect={\"password\":\"RGSYNTH_glued_audit\"}", + "sql.connect {\"password\":\"RGSYNTH_glued_audit\"}", + "sql.connect:{\"password\":\"RGSYNTH_glued_audit\"}", + "sql.connect:password:RGSYNTH_glued_audit", + "sql.connect:token:RGSYNTH_glued_audit", + "password:RGSYNTH_glued_audit", + ] { + for tool in ["run_command", "run_tool"] { + let key = if tool == "run_command" { "command" } else { "tool" }; + let mut args = json!({"params":null}); + args[key] = json!(id); + let logged = normalized_args(tool, &args); + assert!(!logged.to_string().contains("RGSYNTH_glued_audit"), "{tool}: {logged}"); + } + } + let params = + json!({"url": "https://h.invalid/a?%74oken=RGSYNTH_audit_20261103", "pass%77ord": "RGSYNTH_option_20261103", "layer": "roads"}); + for tool in ["run_command", "run_tool"] { + let object = json!({"command": "layer.addUrl", "tool": "buffer", "params": params}); + let mut string = object.clone(); + string["params"] = json!(params.to_string()); + let logged = normalized_args(tool, &object); + assert_eq!(logged, normalized_args(tool, &string)); + assert_eq!(logged["params"], rustgis_engine::log_params(¶ms).into_log_value()); + assert!(!logged.to_string().contains("RGSYNTH_")); + for bad in [json!("{broken RGSYNTH_bad_20261103"), json!("[1]"), json!(42)] { + let mut bad_args = object.clone(); + bad_args["params"] = bad; + assert_eq!(normalized_args(tool, &bad_args)["params"], "[parameters not JSON; withheld]"); + } + } + assert!(!redact_string("failed https://h.invalid/a?%74oken=RGSYNTH_error_20261103").contains("RGSYNTH_")); + let script = "map.zoomIn= {}\nlayer.addUrl {broken RGSYNTH_bad_20261103\nmap.zoomOut {}"; + let logged = redact_script(script); + assert!(logged.starts_with("map.zoomIn {}\nlayer.addUrl [parameters not JSON; withheld]\nmap.zoomOut {}")); + assert!(!logged.contains("RGSYNTH_")); + let comments = "# clean comment\n# https://h.invalid/a?token=RGSYNTH_comment_20261106"; + let args = normalized_args("run", &json!({"script": comments})); + assert!(!args.to_string().contains("RGSYNTH_comment")); + assert!(!redact_script(comments).contains("RGSYNTH_comment")); + assert!(redact_script(comments).starts_with("# clean comment\n")); + for url in + ["https://h.invalid/data", "pg:host=h dbname=gis", "mariadb://h/gis", "pmtiles:https://h.invalid/a", "cog:https://h.invalid/a"] + { + let nested = json!({"key":"abc", "source":{"url":url}}); + assert_eq!(normalized_args("run_command", &json!({"command":"plugin.unknown", "params":nested}))["params"]["key"], "", "{url}"); + } + let document = + json!({"type": "FeatureCollection", "features": [], "padding": "x".repeat(250), "token": "RGSYNTH_data_20261103"}).to_string(); + let arguments = json!({"command": "layer.addDataText", "params": {"text": document}}); + let logged = normalized_args("run_command", &arguments); + assert!(logged["params"]["text"].as_str().is_some_and(|s| s.starts_with("(a document of "))); + assert_eq!(arguments["params"]["text"], document, "the executed data document stays intact"); + } + /// A line for each tool call: what ran, whether it worked, how long it took; secrets in its /// arguments are not written. #[test] @@ -389,10 +443,10 @@ mod tests { "passphrase", "jwt", ] { - assert!(secret_key(k), "{k}"); + assert!(rustgis_engine::data::address::log_sensitive(k), "{k}"); } for k in ["primary_key", "sort_key", "keys_column", "texas", "name"] { - assert!(!secret_key(k), "{k}"); + assert!(!rustgis_engine::data::address::log_sensitive(k), "{k}"); } // Free text keeps its shape; a keyword connection string loses its password. assert_eq!(redact_string("SELECT * FROM t WHERE url = 'http://x?a=1#b'"), "SELECT * FROM t WHERE url = 'http://x?a=1#b'"); @@ -446,4 +500,15 @@ mod tests { assert!(dir.path().join("0000000000.jsonl").exists(), "only day files are pruned"); assert!(is_day_file("2026-10-18.jsonl") && !is_day_file("2026-13-01.jsonl") && !is_day_file("2026-10-18.json")); } + + /// The bounded audit copy labels omitted lines before the display's byte clipping. + #[test] + fn audit_script_line_limit_counts_unrecorded_lines() { + let limit = rustgis_engine::cmd::workflows::MAX_SCRIPT_LINES; + let clean = "# safe\n".repeat(limit); + assert!(!redact_script(&clean).contains("not recorded")); + let output = redact_script(&(clean + "layer.rename {\"layer\":\"RGSYNTH_omitted_line\"}")); + assert!(output.starts_with("# [1 further script line(s) not recorded]\n")); + assert!(!output.contains("RGSYNTH_omitted_line")); + } } diff --git a/crates/assistant/src/mcp.rs b/crates/assistant/src/mcp.rs index 9f9583ccb..31cb390a5 100644 --- a/crates/assistant/src/mcp.rs +++ b/crates/assistant/src/mcp.rs @@ -503,7 +503,9 @@ pub fn expects_reply(message: &str) -> bool { } } +/// Projects generated RPC error text while retaining the exact correlation id and code. fn error_value(id: &Value, code: i64, message: &str) -> Value { + let message = rustgis_engine::render::rustgis_tiles::logins::scrub_error(message); json!({"jsonrpc": "2.0", "id": id, "error": {"code": code, "message": message}}) } diff --git a/crates/assistant/src/tests.rs b/crates/assistant/src/tests.rs index e9e3cc5b9..80cddbf73 100644 --- a/crates/assistant/src/tests.rs +++ b/crates/assistant/src/tests.rs @@ -1224,6 +1224,18 @@ fn mcp_protocol_errors() { assert_eq!(rpc(&server, &mut s, "tools/call", json!({"name": "no_such_tool"}))["error"]["code"], -32602); assert_eq!(rpc(&server, &mut s, "tools/call", json!({"name": "list_layers", "arguments": [1]}))["error"]["code"], -32602); assert_eq!(rpc(&server, &mut s, "resources/read", json!({"uri": "rustgis://docs/nope.md"}))["error"]["code"], -32002); + // Protocol validation fails before tool dispatch; its generated text is public too. + let private = "https://rpc-error.invalid/a?token=RGSYNTH_rpc_error&layer=roads"; + for (method, params, code) in [ + ("tools/call", json!({"name":private,"arguments":{}}), -32602), + ("resources/read", json!({"uri":private}), -32002), + (private, json!({}), -32601), + ] { + let reply = rpc(&server, &mut s, method, params); + assert_eq!(reply["error"]["code"], code); + assert!(!reply["error"]["message"].as_str().unwrap().contains("RGSYNTH_rpc_error")); + } + assert_eq!(rpc(&server, &mut s, "ping", json!({})), json!({})); // Batches: one reply per request, none for notifications only. let batch = r#"[{"jsonrpc":"2.0","id":1,"method":"ping"},{"jsonrpc":"2.0","method":"notifications/initialized"},{"jsonrpc":"2.0","id":2,"method":"nope"}]"#; let r: Value = serde_json::from_str(&server.handle(&mut s, batch).unwrap()).unwrap(); @@ -4328,3 +4340,28 @@ fn geolibre_download_callbacks_require_import_approval() { assert!(tools::gate(&s, &call).is_some(), "{key}"); } } + +/// A peer project invalidates queued work and replies while keeping the readable conversation. +#[test] +fn peer_replacement_cancels_approvals_and_stale_replies() { + let mut s = sample(); + let mut agent = Agent::new(cfg("openai"), &s); + assert!(matches!(agent.ask("export the map", &s), Step::Send(_))); + let pending = + call("peer-old", "run_command", json!({"command":"share.exportMap", "params":{"path":"RGSYNTH_never_written_20261103.png"}})); + assert_eq!(agent.receive(agent.generation(), ok(pending), &mut s), Step::Confirm); + let generation = agent.generation(); + let transcript = agent.transcript.len(); + agent.project_replaced(); + assert!(agent.generation() != generation && !agent.busy() && agent.pending(&s).is_none()); + assert!(agent.transcript.len() >= transcript); + let _ = agent.receive(generation, ok(say("stale")), &mut s); + assert!(!agent.busy() && agent.pending(&s).is_none()); + assert!(matches!(agent.ask("start a new request", &s), Step::Send(_))); + let current = agent.generation(); + let transcript = agent.transcript.len(); + let _ = agent.receive(generation, ok(say("stale while waiting")), &mut s); + assert_eq!(agent.generation(), current); + assert!(agent.waiting() && agent.pending(&s).is_none()); + assert_eq!(agent.transcript.len(), transcript); +} diff --git a/crates/assistant/src/tools.rs b/crates/assistant/src/tools.rs index 8b6fe528c..0dc2aa58a 100644 --- a/crates/assistant/src/tools.rs +++ b/crates/assistant/src/tools.rs @@ -259,6 +259,15 @@ pub fn run(s: &mut Session, call: &ToolCall) -> Result { /// Runs a call with a per-result token estimate, capped by the absolute ceiling. pub fn run_with_budget(s: &mut Session, call: &ToolCall, tokens: usize) -> Result { + // Script failures are structured JSON: clean their error field before serialization. + if call.name == "run" { + return run_script(s, &call.args, tokens); + } + run_with_budget_inner(s, call, tokens).map_err(|e| public_error(&e)) +} + +/// Assembles ordinary tool results; the public boundary cleans every early error too. +fn run_with_budget_inner(s: &mut Session, call: &ToolCall, tokens: usize) -> Result { let max_bytes = output_bytes(tokens); let a = &call.args; let v = match call.name.as_str() { @@ -278,20 +287,18 @@ pub fn run_with_budget(s: &mut Session, call: &ToolCall, tokens: usize) -> Resul return Err(format!("unknown command '{id}'; find ids with search_commands")); } let (before, mark) = (layer_snap(s), s.notice_mark()); - let r = s.run(&id, ¶ms_arg(a)?).map_err(|e| public_text(error_with_notices(e, s, mark)))?; - with_notices(with_new_layers(r, s, &before), s, mark) + let r = s.run(&id, ¶ms_arg(a)?).map_err(|e| error_with_notices(e, s, mark))?; + return Ok(command_result(s, r, &before, mark, tokens)); } "run_tool" => { let tool = str_arg(a, "tool").ok_or("tool is required")?; let (before, mark) = (layer_snap(s), s.notice_mark()); let r = s .run("gp.run", &json!({"tool": tool, "params": params_arg(a)?})) - .map_err(|e| public_text(error_with_notices(e, s, mark)))?; - with_notices(with_new_layers(r, s, &before), s, mark) + .map_err(|e| error_with_notices(e, s, mark))?; + return Ok(command_result(s, r, &before, mark, tokens)); } "read_features" => read_features(s, a)?, - // MCP's `run` (not offered to the in-app assistant): a script in one call. - "run" => return run_script(s, a, tokens), "search_docs" => crate::docs::search(&str_arg(a, "query").unwrap_or_default(), limit_arg(a, 8, 30))?, "read_doc" => { return crate::docs::read_with_budget( @@ -312,7 +319,13 @@ pub fn run_with_budget(s: &mut Session, call: &ToolCall, tokens: usize) -> Resul "test_panic" => panic!("a test tool panicked"), other => return Err(format!("unknown tool '{other}'")), }; - Ok(public_result(s, v, tokens)) + if call.name == "read_features" { + Ok(public_envelope(s, v, tokens)) + } else { + let mut withheld = Withheld::default(); + let v = public_metadata(v, 0, &mut withheld); + Ok(finish_result(s, v, tokens, withheld)) + } } /// How deep a result is searched for runtime login addresses; what is deeper is withheld. @@ -322,19 +335,68 @@ const MAX_DEPTH: usize = 16; /// `pmtiles:…@`) replaced by its plain address. Commands return the bound addresses the app needs /// to use a connection's login; the model and MCP clients get only the public ones. fn public_text(text: String) -> String { - if text.contains(rustgis_engine::data::login_address::PREFIX) { - rustgis_engine::render::rustgis_tiles::logins::scrub(&text) - } else { - text - } + rustgis_engine::render::rustgis_tiles::logins::scrub(&text) } +/// Generated errors may quote arbitrary addresses; successful values remain data. +fn public_error(text: &str) -> String { + rustgis_engine::render::rustgis_tiles::logins::scrub_error(text) +} + +/// Publishes a completed result, reporting withheld subtrees before enforcing the response budget. +#[cfg(test)] fn public_result(s: &mut Session, value: Value, tokens: usize) -> String { - let mut withheld = 0usize; - let mut value = public_addresses(value, 0, &mut withheld); - if withheld > 0 { + let mut withheld = Withheld::default(); + let value = public_addresses(value, 0, &mut withheld); + finish_result(s, value, tokens, withheld) +} + +/// These fields belong to the adapter's response schema; only their values contain data. +fn public_envelope(s: &mut Session, value: Value, tokens: usize) -> String { + let mut withheld = Withheld::default(); + let value = match value { + Value::Object(o) => Value::Object(o.into_iter().map(|(key, value)| (key, public_addresses(value, 1, &mut withheld))).collect()), + value => public_addresses(value, 0, &mut withheld), + }; + finish_result(s, value, tokens, withheld) +} + +/// Generated search, tool and project metadata has a fixed schema at every level. Its +/// strings may hold private data, but its keys cannot be login values. Row/property maps +/// and arbitrary command results use `public_addresses` instead. +fn public_metadata(value: Value, depth: usize, withheld: &mut Withheld) -> Value { + match value { + Value::String(text) => Value::String(public_text(text)), + Value::Array(a) if depth < MAX_DEPTH => Value::Array(a.into_iter().map(|v| public_metadata(v, depth + 1, withheld)).collect()), + Value::Object(o) if depth < MAX_DEPTH => { + Value::Object(o.into_iter().map(|(key, v)| (key, public_metadata(v, depth + 1, withheld))).collect()) + } + Value::Array(_) | Value::Object(_) => { + withheld.depth = withheld.depth.saturating_add(1); + json!("[too deep]") + } + other => other, + } +} + +/// Cleans the arbitrary command payload before adding adapter-owned notices/layer metadata. +fn command_result(s: &mut Session, value: Value, before: &[(String, Vec)], mark: u64, tokens: usize) -> String { + let mut withheld = Withheld::default(); + let value = public_addresses(value, 0, &mut withheld); + let value = with_notices(with_new_layers(value, s, before), s, mark); + finish_result(s, value, tokens, withheld) +} + +/// Publishes counted projection losses without sending the completed envelope through a second redactor. +fn finish_result(s: &mut Session, mut value: Value, tokens: usize, withheld: Withheld) -> String { + if withheld.depth > 0 || withheld.keys > 0 { let mark = s.notice_mark(); - s.notify(format!("Tool result: {withheld} subtrees at nesting depth {MAX_DEPTH} were withheld."), false); + if withheld.depth > 0 { + s.notify(format!("Tool result: {} subtrees at nesting depth {MAX_DEPTH} were withheld.", withheld.depth), false); + } + if withheld.keys > 0 { + s.notify(format!("Tool result: {} object(s) were withheld because redacted field names coincide.", withheld.keys), false); + } if let Some(notices) = value.get_mut("notices").and_then(Value::as_array_mut) { notices.extend(s.notices_since(mark).iter().map(|n| json!(n.text))); } else { @@ -344,16 +406,33 @@ fn public_result(s: &mut Session, value: Value, tokens: usize) -> String { bounded_json(&value, tokens) } -/// `v` with its strings made public ([`public_text`]), to a depth of [`MAX_DEPTH`]. -fn public_addresses(v: Value, depth: usize, withheld: &mut usize) -> Value { +/// Counts output withheld for depth limits or field names that collide after redaction. +#[derive(Default)] +struct Withheld { + depth: usize, + keys: usize, +} + +/// Projects keys and values through active-login redaction; ambiguous objects and deep subtrees +/// are withheld and counted without changing the completed command's success. +fn public_addresses(v: Value, depth: usize, withheld: &mut Withheld) -> Value { match v { Value::String(s) => Value::String(public_text(s)), Value::Array(a) if depth < MAX_DEPTH => Value::Array(a.into_iter().map(|x| public_addresses(x, depth + 1, withheld)).collect()), Value::Object(o) if depth < MAX_DEPTH => { - Value::Object(o.into_iter().map(|(k, x)| (k, public_addresses(x, depth + 1, withheld))).collect()) + let mut public = serde_json::Map::new(); + for (key, value) in o { + let key = public_text(key); + if public.contains_key(&key) { + withheld.keys = withheld.keys.saturating_add(1); + return json!("[redacted field names coincide]"); + } + public.insert(key, value); + } + Value::Object(public.into_iter().map(|(key, value)| (key, public_addresses(value, depth + 1, withheld))).collect()) } Value::Array(_) | Value::Object(_) => { - *withheld = withheld.saturating_add(1); + withheld.depth = withheld.depth.saturating_add(1); json!("[too deep]") } other => other, @@ -434,20 +513,10 @@ fn str_arg(a: &Value, k: &str) -> Option { /// The `params` a command or tool runs with: an object, or JSON text of one (some models send /// that). Anything else is an error: running with defaults instead could act on the wrong layer. fn params_arg(a: &Value) -> Result { - match a.get("params") { - None | Some(Value::Null) => Ok(json!({})), - Some(v @ Value::Object(_)) => Ok(v.clone()), - Some(Value::String(t)) if t.trim().is_empty() => Ok(json!({})), - Some(Value::String(t)) => match serde_json::from_str::(t) { - Ok(v @ Value::Object(_)) => Ok(v), - Ok(_) => Err("params must be a JSON object".into()), - Err(e) => Err(format!("params is not valid JSON: {e}")), - }, - Some(_) => Err("params must be a JSON object".into()), - } + rustgis_engine::params_envelope(a.get("params")) } -/// An object parameter, which some models send as JSON text (leniently, for display and approval). +/// An object parameter, which some models send as JSON text (leniently, for display only). fn obj_arg(a: &Value, k: &str) -> Value { match a.get(k) { Some(Value::String(t)) if !t.trim().is_empty() => serde_json::from_str(t).unwrap_or(Value::Null), @@ -479,7 +548,7 @@ fn with_new_layers(r: Value, s: &Session, before: &[(String, Vec)]) -> Valu let same = before.iter().find(|(name, _)| name.eq_ignore_ascii_case(&m.name)).or_else(|| if renamed { before.get(active) } else { None }); let had = same.map_or(&[][..], |(_, ids)| ids.as_slice()); - let added: Vec = m.layers.iter().filter(|l| !had.contains(&l.id)).map(|l| l.name.clone()).collect(); + let added: Vec = m.layers.iter().filter(|l| !had.contains(&l.id)).map(|l| public_text(l.name.clone())).collect(); if added.is_empty() { return if r.is_null() { json!({"ok": true}) } else { r }; } @@ -510,16 +579,17 @@ fn run_script(s: &mut Session, a: &Value, tokens: usize) -> Result { + let e = public_error(&e); // Nothing was kept, so no line ran and no layer was added. // The notice of the failure is the error redacted: compared as such, it isn't repeated. let told = rustgis_engine::data::diagnostics::redact(&e); let said: Vec<&&str> = notices.iter().filter(|t| **t != e && **t != told).collect(); let out = json!({"success": false, "result": null, "lines": 0, "layersAdded": [], "notices": said, "error": clip(&e, 8000), "duration_s": secs}); - Err(public_result(s, out, tokens)) + Err(public_envelope(s, out, tokens)) } } } @@ -527,7 +597,7 @@ fn run_script(s: &mut Session, a: &Value, tokens: usize) -> Result Value { - let said: Vec<&str> = s.notices_since(mark).iter().map(|n| n.text.as_str()).collect(); + let said: Vec = s.notices_since(mark).iter().map(|n| public_text(n.text.clone())).collect(); if said.is_empty() { return r; } @@ -864,6 +934,69 @@ pub fn context(s: &Session) -> String { #[cfg(test)] #[allow(clippy::unwrap_used)] mod tests { + /// Returned tool and fetch errors are cleaned before active patterns can damage URLs; + /// successful values and the original call remain outside the generated-error projection. + #[cfg(not(target_arch = "wasm32"))] + #[test] + fn generated_tool_and_fetch_errors_clean_addresses_before_active_patterns() { + const TEST: &str = "tools::tests::generated_tool_and_fetch_errors_clean_addresses_before_active_patterns"; + if std::env::var("RUSTGIS_ISOLATED_TOOL_ERRORS").as_deref() != Ok(TEST) { + let output = std::process::Command::new(std::env::current_exe().unwrap()) + .args(["--exact", TEST]) + .env("RUSTGIS_ISOLATED_TOOL_ERRORS", TEST) + .output() + .unwrap(); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!( + output.status.success() && stdout.contains("1 passed; 0 failed"), + "{stdout}\n{}", + String::from_utf8_lossy(&output.stderr) + ); + return; + } + use rustgis_engine::render::rustgis_tiles::{failure::Failure, logins}; + let mut s = Session::new(rustgis_engine::Project::default()); + let url = "https://returned-error.invalid/a?token=RGSYNTH_returned_error&layer=roads"; + for active in [false, true] { + let binding = active.then(|| { + logins::activate( + "https://returned-error.invalid", + "synthetic-returned-error", + logins::Method::Header("X-Key".into()), + "https", + false, + ) + .unwrap() + }); + for (name, args) in [ + ("read_features", json!({"layer":url})), + ("describe_tool", json!({"tool":url})), + (url, json!({})), + ("run", json!({"script":url})), + ] { + let call = ToolCall { id: "error".into(), name: name.into(), args }; + let original = call.clone(); + let error = run_with_budget(&mut s, &call, DEFAULT_OUTPUT_TOKENS).unwrap_err(); + assert!(!error.contains("RGSYNTH_returned_error"), "{name}, active={active}: {error}"); + assert_eq!(call, original); + if name == "run" { + let reply: Value = serde_json::from_str(&error).unwrap(); + assert_eq!(reply["success"], false); + assert!(reply["error"].as_str().is_some()); + } + } + let address = binding.as_ref().map_or_else(|| url.to_string(), |binding| logins::bind(binding, url).unwrap()); + let failure = Failure::new(&address, format!("failed to fetch {address}")); + assert!(!failure.message.contains("RGSYNTH_returned_error")); + if let Some(binding) = binding { + assert!(logins::deactivate(&binding)); + } + } + let clean = ToolCall { id: "clean".into(), name: "read_features".into(), args: json!({"layer":"ordinary missing layer"}) }; + assert!(run_with_budget(&mut s, &clean, DEFAULT_OUTPUT_TOKENS).unwrap_err().contains("ordinary missing layer")); + assert_eq!(Failure::new("https://clean.invalid/a?layer=roads", "HTTP 403").message, "HTTP 403"); + assert_eq!(logins::scrub_error("failed https://clean.invalid/a?layer=roads"), "failed https://clean.invalid/a?layer=roads"); + } use super::*; use rustgis_engine::render::rustgis_tiles::logins; @@ -892,13 +1025,61 @@ mod tests { assert!(listed.contains(&plain), "the model still learns the layer's address: {listed}"); } + /// The known response schema survives a secret that happens to name one of its fields. + #[cfg(not(target_arch = "wasm32"))] + #[test] + fn run_envelope_keys_are_not_data_derived_login_values() { + if std::env::var_os("RUSTGIS_TEST_RUN_ENVELOPE").is_none() { + let output = std::process::Command::new(std::env::current_exe().unwrap()) + .args(["--exact", "tools::tests::run_envelope_keys_are_not_data_derived_login_values"]) + .env("RUSTGIS_TEST_RUN_ENVELOPE", "1") + .output() + .unwrap(); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!( + output.status.success() && stdout.contains("1 passed; 0 failed"), + "{stdout}\n{}", + String::from_utf8_lossy(&output.stderr) + ); + return; + } + let mut s = Session::new(rustgis_engine::Project::default()); + let id = s + .run("connection.new", &json!({"kind":"wms", "address":"https://envelope.invalid/wms", "auth":"basic", "user":"reader"})) + .unwrap()["id"] + .clone(); + s.run("connection.login", &json!({"id":id, "password":"success"})).unwrap(); + for (script, success) in [("map.zoomIn {}", true), ("not.a.command {}", false)] { + let call = ToolCall { id: "envelope".into(), name: "run".into(), args: json!({"script":script}) }; + let text = run(&mut s, &call).unwrap_or_else(|e| e); + let out: Value = serde_json::from_str(&text).unwrap(); + assert_eq!(out.get("success"), Some(&json!(success)), "{text}"); + assert!(out.get("result").is_some() && out.get("notices").is_some() && out.get("duration_s").is_some()); + } + let data = public_addresses(json!({"success":"private field"}), 0, &mut Withheld::default()); + assert_eq!(data["[redacted]"], "private field", "the same spelling in dynamic data is still scrubbed"); + s.run("connection.login", &json!({"id":id, "password":"id"})).unwrap(); + for (name, args, array) in + [("search_commands", json!({"query":"zoom"}), "commands"), ("search_tools", json!({"query":"buffer"}), "tools")] + { + let call = ToolCall { id: "metadata".into(), name: name.into(), args }; + let text = run(&mut s, &call).unwrap(); + let out: Value = serde_json::from_str(&text).unwrap(); + let entries = out.get(array).and_then(Value::as_array).unwrap(); + assert!(!entries.is_empty(), "{text}"); + assert!(entries.iter().all(|entry| entry.get("id").is_some()), "fixed nested schema keys survive: {text}"); + } + let data = public_addresses(json!({"id":"private field"}), 0, &mut Withheld::default()); + assert_eq!(data["[redacted]"], "private field"); + } + #[test] fn wrappers_after_a_prefix_are_made_plain_too() { let origin = "https://assistant-login.example"; let binding = logins::activate(origin, "assistant", logins::Method::Header("X-Key".into()), "assistant-secret", false).unwrap(); let archive = logins::bind(&binding, &format!("{origin}/a.pmtiles")).unwrap(); let result = json!({"layers": [{"next": {"request": {"url": format!("pmtiles:x@{archive}")}}}], "count": 2}); - let v = public_addresses(result, 0, &mut 0); + let v = public_addresses(result, 0, &mut Withheld::default()); assert_eq!(v["layers"][0]["next"]["request"]["url"], format!("pmtiles:x@{origin}/a.pmtiles")); assert_eq!(v["count"], 2); assert!(logins::deactivate(&binding)); @@ -930,4 +1111,52 @@ mod tests { assert_eq!(out, result); assert!(s.notices_since(0).is_empty()); } + + #[test] + fn successful_results_scrub_active_raw_service_keys_without_a_wrapper() { + let mut s = Session::new(rustgis_engine::Project::default()); + let id = s + .run( + "connection.new", + &json!({"kind":"wms","address":"https://echo-r15.example.invalid/wms","auth":"header","keyName":"X-Key"}), + ) + .unwrap()["id"] + .as_str() + .unwrap() + .to_string(); + s.run("connection.login", &json!({"id":id,"apiKey":"fake001"})).unwrap(); + let result = json!({"success":true,"echo":"server said fake001","clean":"ordinary café text"}); + let out: Value = serde_json::from_str(&public_result(&mut s, result, DEFAULT_OUTPUT_TOKENS)).unwrap(); + assert_eq!(out["success"], true); + assert_eq!(out["echo"], "server said [redacted]"); + assert_eq!(out["clean"], "ordinary café text"); + assert!(s.notices_since(0).is_empty()); + // A generated row uses imported field names as object keys. Keep the stored data, + // but never send a field repeating an active key to the model in plain text. + let source = json!({"type":"FeatureCollection","features":[{ + "type":"Feature","geometry":{"type":"Point","coordinates":[1,2]}, + "properties":{"fake001":"retained value","clean":"ordinary café text"} + }]}); + s.run("layer.addDataText", &json!({"fileName":"echo-fields.geojson","text":source.to_string()})).unwrap(); + let name = s.map().layers.last().unwrap().name.clone(); + let call = ToolCall { id: "echo-fields".into(), name: "read_features".into(), args: json!({"layer":name,"limit":1}) }; + let mark = s.notice_mark(); + let text = run_with_budget(&mut s, &call, DEFAULT_OUTPUT_TOKENS).unwrap(); + assert!(!text.contains("fake001"), "generated object keys must be scrubbed too: {text}"); + let out: Value = serde_json::from_str(&text).unwrap(); + assert_eq!(out["rows"][0]["[redacted]"], "retained value"); + assert_eq!(out["rows"][0]["clean"], "ordinary café text"); + assert!(s.map().layers.last().unwrap().dataset.fields.iter().any(|f| f.name == "fake001"), "stored data stays unchanged"); + assert!(s.notices_since(mark).is_empty()); + // Two keys that become equal must not silently overwrite one value with the other. + let mark = s.notice_mark(); + let collision = json!({"success":true,"result":{"fake001":"first value","[redacted]":"second value"}}); + let text = public_result(&mut s, collision, DEFAULT_OUTPUT_TOKENS); + assert!(!text.contains("fake001")); + let out: Value = serde_json::from_str(&text).unwrap(); + assert_eq!(out["success"], true); + assert!(out["result"].is_string(), "ambiguous redacted objects are explicitly withheld"); + assert_eq!(s.notices_since(mark).len(), 1); + assert!(s.notices_since(mark)[0].text.contains("field names")); + } } diff --git a/crates/data/src/address.rs b/crates/data/src/address.rs new file mode 100644 index 000000000..c85d7ea1e --- /dev/null +++ b/crates/data/src/address.rs @@ -0,0 +1,768 @@ +//! Credential names and public address text, shared by sources, diagnostics and writers. +//! +//! A plain name is not a promise about its value. In particular it does not authorize sending +//! that value to an address advertised by a service. These helpers never inspect dataset text. + +use std::borrow::Cow; + +/// Maximum decoding work for one name; anything still encoded is withheld. +pub const MAX_DECODE_PASSES: usize = 4; +pub const MAX_NAME_BYTES: usize = 16 * 1024; + +/// Classification after bounded decoding, without decoding a pair's value. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum NameClass { + Credential, + /// Not recognized; this alone never authorizes forwarding. + Plain, + /// Too long, or still decodable after the budget. Never forwarded, even under a scope. + Unresolved, +} + +/// Parses only a complete hexadecimal escape supplied by the byte scanner. +fn hex(bytes: &[u8]) -> Option { + bytes.iter().try_fold(0, |n, b| Some(n * 16 + char::from(*b).to_digit(16)?)) +} + +/// One byte-preserving pass. Lossy UTF-8 is applied only after all passes, so percent-encoded +/// bytes and literal Unicode take the same route. Invalid escapes remain literal. +fn decode_pass(bytes: &[u8]) -> Vec { + let mut out = Vec::with_capacity(bytes.len()); + let mut i = 0; + while let Some(&b) = bytes.get(i) { + if b == b'%' + && let Some(n) = bytes.get(i + 1..i + 3).and_then(hex) + { + out.push(n as u8); + i += 3; + } else if b == b'\\' + && bytes.get(i + 1) == Some(&b'u') + && let Some(c) = bytes.get(i + 2..i + 6).and_then(hex).and_then(char::from_u32) + { + out.extend_from_slice(c.encode_utf8(&mut [0; 4]).as_bytes()); + i += 6; + } else { + out.push(if b == b'+' { b' ' } else { b }); + i += 1; + } + } + out +} + +/// Repeated decoding stops at a fixed point; a fifth changing pass is unresolved. +fn decoded_name(raw: &str) -> Option> { + if raw.len() > MAX_NAME_BYTES { + return None; + } + let mut name = raw.as_bytes().to_vec(); + for _ in 0..MAX_DECODE_PASSES { + let next = decode_pass(&name); + if next == name { + return Some(name); + } + name = next; + } + (decode_pass(&name) == name).then_some(name) +} + +/// Canonical comparison spelling; invalid UTF-8 contributes no ASCII name characters. +fn letters(bytes: &[u8]) -> String { + String::from_utf8_lossy(bytes).chars().filter(char::is_ascii_alphanumeric).map(|c| c.to_ascii_lowercase()).collect() +} + +/// ASCII letters and digits after bounded decoding, lower-cased; `None` means Unresolved. +pub fn normalized_name(raw: &str) -> Option { + decoded_name(raw).map(|s| letters(&s)) +} + +/// Context is collected from decoded query names, including legacy semicolon-separated pairs. +#[derive(Default)] +pub struct QueryNames { + sig: bool, + sv: bool, + state: bool, +} + +impl QueryNames { + /// Collects context flags without decoding values or retaining query payloads. + pub fn of(query: &str) -> Self { + let mut names = Self::default(); + for pair in query.split(['&', ';']) { + if let Some(decoded) = decoded_name(pair_name(pair)) { + let k = letters(decoded.split(|b| *b == b'=').next().unwrap_or(&decoded)); + names.sig |= k == "sig"; + names.sv |= k == "sv"; + names.state |= k == "state"; + } + } + names + } +} + +const SAS_KEYS: &[&str] = + &["sv", "se", "sp", "sr", "st", "spr", "sip", "si", "ss", "srt", "skoid", "sktid", "skt", "ske", "sks", "skv", "sdd"]; + +/// Exact and containment rules over canonical names, plus query-wide context. +fn credential(k: &str, query: &QueryNames) -> bool { + matches!(k, "key" | "appid" | "subscriptionkey" | "clientkey" | "saskey" | "pwd" | "sig" | "signature" | "sas" | "authorization" | "proxyauthorization" | "cookie" | "setcookie" | "bearer") + // These public refusal and tracing headers share the signing family's prefix. + || (k.starts_with("xamz") && !matches!(k, "xamznrequestid" | "xamznwafaction" | "xamzcfid")) + || k.starts_with("xgoog") + || [ + "password", + "passwd", + "secret", + "token", + "credential", + "apikey", + "accesskey", + "privatekey", + "accountkey", + "sharedaccess", + "connectionstring", + ] + .iter() + .any(|w| k.contains(w)) + || (query.sig && query.sv && SAS_KEYS.contains(&k)) + || (k == "code" && query.state) +} + +/// Classifies a raw name. An encoded equals sign is checked both as a name boundary and as +/// part of the whole name; values after a literal equals sign are never decoded here. +pub fn name_class(raw: &str, query: &QueryNames) -> NameClass { + let Some(decoded) = decoded_name(raw) else { return NameClass::Unresolved }; + let before = decoded.split(|b| *b == b'=').next().unwrap_or(&decoded); + if credential(&letters(&decoded), query) || credential(&letters(before), query) { NameClass::Credential } else { NameClass::Plain } +} + +/// True for recognized credentials and unresolved names; no query context is assumed. +pub fn secret_key(raw: &str) -> bool { + name_class(raw, &QueryNames::default()) != NameClass::Plain +} + +/// Names logs also omit, beyond credential-bearing address options. Logs are output only. +pub fn log_sensitive(raw: &str) -> bool { + if secret_key(raw) { + return true; + } + let Some(decoded) = decoded_name(raw) else { return true }; + let name = String::from_utf8_lossy(&decoded); + let k = letters(&decoded); + if k.contains("body") + || matches!(k.as_str(), "basic" | "authentication" | "wwwauthenticate" | "proxyauthenticate" | "userid" | "username") + { + return true; + } + // The audit's existing word policy, including plurals and camelCase, is shared here with + // diagnostics. It deliberately omits more than replayable state does. + let mut words = Vec::new(); + let mut word = String::new(); + let mut prev_lower = false; + for c in name.chars() { + if (!c.is_ascii_alphanumeric() || (c.is_ascii_uppercase() && prev_lower)) && !word.is_empty() { + words.push(std::mem::take(&mut word)); + } + prev_lower = c.is_ascii_lowercase() || c.is_ascii_digit(); + if c.is_ascii_alphanumeric() { + word.push(c.to_ascii_lowercase()); + } + } + if !word.is_empty() { + words.push(word); + } + words.iter().enumerate().any(|(i, word)| { + let w = word.strip_suffix('s').unwrap_or(word); + matches!( + w, + "passphrase" + | "jwt" + | "privatekey" + | "password" + | "passwd" + | "pwd" + | "secret" + | "token" + | "apikey" + | "auth" + | "authorization" + | "bearer" + | "credential" + | "signature" + | "cookie" + | "sas" + | "dsn" + ) || ["token", "secret", "password", "passphrase"].iter().any(|s| w.ends_with(s)) + || (w == "key" + && (words.len() == 1 + || i.checked_sub(1).and_then(|j| words.get(j)).is_some_and(|p| { + ["api", "access", "private", "secret", "account", "sas", "subscription", "client"].contains(&p.as_str()) + }))) + }) +} + +/// Forwarding options compare their complete decoded spelling, including punctuation. The +/// broader credential normalization must not make an unknown name into an allowlisted one. +fn option_name(raw: &str) -> Option { + decoded_name(raw).map(|s| String::from_utf8_lossy(&s).to_ascii_lowercase()) +} + +/// Only a literal equals sign separates a raw name from its value. +fn pair_name(pair: &str) -> &str { + pair.split_once('=').map_or(pair, |(k, _)| k) +} + +/// A legacy semicolon credential taints the whole ampersand pair. +fn pair_class(pair: &str, names: &QueryNames) -> NameClass { + let mut class = NameClass::Plain; + for sub in pair.split(';') { + match name_class(pair_name(sub), names) { + NameClass::Unresolved => return NameClass::Unresolved, + NameClass::Credential => class = NameClass::Credential, + NameClass::Plain => {} + } + } + class +} + +/// Query text ends at the fragment, which is never interpreted as another query. +fn query(address: &str) -> &str { + address.split('#').next().and_then(|s| s.split_once('?')).map_or("", |(_, q)| q) +} + +/// Removes credential and unresolved pairs; retained pairs keep their original bytes. The +/// caller handles user-info and fragments. A legacy `;token=…` removes its entire `&` pair. +pub fn redact_query(address: &mut String) { + let Some(at) = address.find('?').filter(|at| address.find('#').is_none_or(|end| *at < end)) else { return }; + let q = query(address); + let names = QueryNames::of(q); + if q.split('&').all(|p| pair_class(p, &names) == NameClass::Plain) { + return; + } + let kept = q.split('&').filter(|p| !p.is_empty() && pair_class(p, &names) == NameClass::Plain).collect::>().join("&"); + let fragment = address.find('#').and_then(|i| address.get(i..)).unwrap_or("").to_string(); + address.truncate(at); + if !kept.is_empty() { + address.push('?'); + address.push_str(&kept); + } + address.push_str(&fragment); +} + +/// Public form of one web or S3 address. Local paths and other unrecognized text are untouched. +#[derive(Debug, PartialEq, Eq)] +pub enum Public { + Unchanged, + Changed { + address: String, + /// Omitted user-info, query pairs and fragment components; not a credential count. + removed: u16, + }, + /// A recognized address whose authority cannot safely be delimited; retain only its prefix. + Withheld, +} + +/// Masks supported tile placeholders in a validation copy of the authority. Equal byte lengths +/// preserve user-info offsets; the public address always retains the original template spelling. +/// Switch choices may name host labels, but cannot introduce URL delimiters or credentials. +fn template_authority(rest: &str) -> Cow<'_, str> { + let end = rest.find(['/', '?', '#']).unwrap_or(rest.len()); + let Some(authority) = rest.get(..end) else { return Cow::Borrowed(rest) }; + let mut masked = None; + let mut copied = 0; + let mut scan = 0; + while let Some(open) = authority.get(scan..).and_then(|s| s.find('{')).map(|at| scan + at) { + let Some(close) = authority.get(open + 1..).and_then(|s| s.find('}')).map(|at| open + 1 + at) else { break }; + let token = authority.get(open + 1..close).unwrap_or(""); + let known = matches!( + token, + "z" | "zoom" | "x" | "y" | "-y" | "q" | "quadkey" | "s" | "bbox-epsg-3857" | "width" | "height" | "r" | "ratio" + ) || token.strip_prefix("switch:").is_some_and(|choices| { + choices.split(',').all(|choice| { + let choice = choice.trim(); + !choice.is_empty() && choice.bytes().all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_')) + }) + }); + if known { + let out = masked.get_or_insert_with(|| String::with_capacity(rest.len())); + out.push_str(authority.get(copied..open).unwrap_or("")); + out.extend(std::iter::repeat_n('x', close + 1 - open)); + copied = close + 1; + } + scan = close + 1; + } + match masked { + Some(mut out) => { + out.push_str(rest.get(copied..).unwrap_or("")); + Cow::Owned(out) + } + None => Cow::Borrowed(rest), + } +} + +/// Removes user-info, fragments and secret query pairs, or withholds an unreadable authority. +pub fn public_web_address(address: &str) -> Public { + let text = address.trim(); + let Some((protocol, _)) = text.split_once(':') else { return Public::Unchanged }; + if !["http", "https", "s3"].iter().any(|p| protocol.eq_ignore_ascii_case(p)) { + return Public::Unchanged; + } + let Some((scheme, rest)) = text.split_once("://") else { return Public::Withheld }; + let at = crate::source::pgconn::userinfo_end(&template_authority(rest)); + let rest = at.and_then(|i| rest.get(i + 1..)).unwrap_or(rest); + let no_fragment = rest.split('#').next().unwrap_or(rest); + let candidate = format!("{scheme}://{no_fragment}"); + if no_fragment.split(['/', '?']).next().is_none_or(str::is_empty) + // Backslashes can change authority/path parsing, but query backslashes are data: + // escaped credential names still need the bounded query decoder below. + || candidate.split('?').next().is_some_and(|head| head.contains('\\')) + || candidate.chars().any(char::is_control) + || url::Url::parse(&format!("{scheme}://{}", template_authority(no_fragment))) + .ok() + .is_none_or(|u| u.host_str().is_none() || !u.username().is_empty() || u.password().is_some()) + { + return Public::Withheld; + } + let names = QueryNames::of(query(&candidate)); + let removed = query(&candidate).split('&').filter(|p| !p.is_empty() && pair_class(p, &names) != NameClass::Plain).count(); + let removed = + u16::try_from(removed).unwrap_or(u16::MAX).saturating_add(u16::from(at.is_some())).saturating_add(u16::from(rest.contains('#'))); + let mut public = candidate; + redact_query(&mut public); + if public == text { Public::Unchanged } else { Public::Changed { address: public, removed } } +} + +/// A bare supported scheme is a prose example with no authority or credential to remove. +pub(crate) fn scheme_example(text: &str) -> bool { + ["http://", "https://", "s3://"].iter().any(|scheme| text.eq_ignore_ascii_case(scheme)) +} + +/// Cleans addresses in generated text and counts changed addresses. All spacing and all other +/// text stay byte-for-byte. Never apply this to user-authored text or dataset values. +pub fn scrub_addresses(text: &str) -> (Cow<'_, str>, u32) { + let mut changed = 0u32; + let mut out = String::new(); + let mut start = 0; + for word in text.split_inclusive(char::is_whitespace) { + let unspaced = word.trim_end_matches(char::is_whitespace); + // Recognize supported prefixes before parsing: malformed delimiters must not turn a + // recognized address into retained text. The preceding scheme characters establish the + // same word boundary for a bare address, a quoted one and an address inside a wrapper. + let mut schemes = unspaced + .match_indices(':') + .filter_map(|(at, _)| { + let begin = unspaced + .get(..at)? + .char_indices() + .rev() + .find(|(_, c)| !c.is_ascii_alphanumeric() && !"+.-".contains(*c)) + .map_or(0, |(i, c)| i + c.len_utf8()); + let scheme = unspaced.get(begin..at)?; + ["http", "https", "s3"].iter().any(|s| scheme.eq_ignore_ascii_case(s)).then_some((begin, at)) + }) + .peekable(); + let mut copied = 0; + while let Some((begin, scheme_at)) = schemes.next() { + let boundary = schemes.peek().map_or(unspaced.len(), |(begin, _)| *begin); + let end = unspaced + .get(..boundary) + .unwrap_or("") + .trim_end_matches([':', ',', ';', '.', ')', ']', '>', '"', '\'', '»', '›', '”', '’']) + .len() + .max(scheme_at + 1); + let address = word.get(begin..end).unwrap_or(""); + // A bare scheme in diagnostic prose has no address or credentials to remove. + if scheme_example(address) { + continue; + } + let public = public_web_address(address); + if matches!(public, Public::Unchanged) { + continue; + } + if changed == 0 { + out.push_str(text.get(..start).unwrap_or("")); + } + changed = changed.saturating_add(1); + out.push_str(word.get(copied..begin).unwrap_or("")); + match public { + Public::Unchanged => out.push_str(address), + Public::Changed { address, .. } => out.push_str(&address), + Public::Withheld => { + out.push_str(address.split(':').next().unwrap_or("https")); + out.push(':'); + } + } + copied = end; + } + if changed > 0 { + out.push_str(word.get(copied..).unwrap_or("")); + } + start += word.len(); + } + (if changed == 0 { Cow::Borrowed(text) } else { Cow::Owned(out) }, changed) +} + +/// Mechanical path containment for the exact address fetched. Containment alone grants no +/// authority: advertised destinations still use F3 unless a deployment was explicitly reviewed. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Scope { + scheme: String, + host: String, + port: u16, + segments: Vec, +} + +impl Scope { + /// Reads authority with the fetch parser, and path segments from the original bytes. + pub fn of(address: &str) -> Option { + if address.trim() != address || address.chars().any(char::is_control) || address.contains('\\') { + return None; + } + let parsed = url::Url::parse(address).ok()?; + if !matches!(parsed.scheme(), "http" | "https") || !parsed.username().is_empty() || parsed.password().is_some() { + return None; + } + let (_, rest) = address.split_once("://")?; + let end = rest.find(['/', '?', '#']).unwrap_or(rest.len()); + if end == 0 || rest.get(..end)?.contains('@') { + return None; + } + let path = rest.get(end..)?.split(['?', '#']).next()?; + let path = path.strip_prefix('/').unwrap_or(path); + let path = path.strip_suffix('/').unwrap_or(path); + let mut segments = Vec::new(); + if !path.is_empty() { + for segment in path.split('/') { + let lower = segment.to_ascii_lowercase(); + if segment.is_empty() || matches!(segment, "." | "..") || ["%2e", "%2f", "%5c", "%25"].iter().any(|s| lower.contains(s)) { + return None; + } + segments.push(segment.to_string()); + } + } else if rest.get(end..)?.starts_with("//") { + return None; + } + Some(Self { + scheme: parsed.scheme().to_string(), + host: parsed.host_str()?.trim_end_matches('.').to_ascii_lowercase(), + port: parsed.port_or_known_default()?, + segments, + }) + } + + /// Whole raw segments must match; ambiguous paths and user-info are outside every scope. + pub fn covers_below(&self, to: &str) -> bool { + Self::of(to).is_some_and(|other| { + self.scheme == other.scheme && self.host == other.host && self.port == other.port && other.segments.starts_with(&self.segments) + }) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Relation { + Constructed, + Advertised, +} + +pub const SAFE_OPTIONS: &[&str] = &["collections", "datetime", "subset", "subset-crs", "crs", "bgcolor", "transparent"]; + +/// Only a reviewed deployment may authorize full-query inheritance for advertised links. Root +/// entries require separate whole-origin agreement, including every tenant on that origin. +pub struct ReviewedScope { + pub directory: &'static str, + pub whole_origin: bool, +} +/// No deployment has been authorized for full-query inheritance. Keep this list empty until +/// that review exists; the explicit-list tests exercise the same deny boundary. +pub const F2_SCOPES: &[ReviewedScope] = &[]; + +#[derive(Debug, PartialEq, Eq)] +pub struct Inherited { + pub url: String, + pub withheld: bool, +} + +/// Adds missing pairs, preserving their bytes. Unresolved pairs never follow; F3 sends only +/// SAFE_OPTIONS to advertised addresses. Only checked constructed sub-addresses get other pairs. +pub fn inherit_query(from: &str, to: &str, relation: Relation, skip: &[&str]) -> Inherited { + inherit_reviewed(from, to, relation, skip, F2_SCOPES) +} + +/// Applies the fixed policy with an explicit reviewed list; production passes the empty list. +fn inherit_reviewed(from: &str, to: &str, relation: Relation, skip: &[&str], reviewed: &[ReviewedScope]) -> Inherited { + let full = match relation { + Relation::Constructed => Scope::of(from).is_some_and(|s| s.covers_below(to)), + Relation::Advertised => reviewed.iter().any(|r| { + Scope::of(r.directory).is_some_and(|s| (!s.segments.is_empty() || r.whole_origin) && s.covers_below(from) && s.covers_below(to)) + }), + }; + let q = query(from); + let names = QueryNames::of(q); + let mut out = Inherited { url: to.to_string(), withheld: false }; + let mut present: Vec = query(to).split(['&', ';']).filter_map(|p| option_name(pair_name(p))).collect(); + for pair in q.split('&').filter(|p| !p.is_empty()) { + let class = pair_class(pair, &names); + if class == NameClass::Unresolved { + out.withheld = true; + continue; + } + let keys: Vec = pair.split(';').filter_map(|p| option_name(pair_name(p))).collect(); + let skipped = |k: &String| skip.iter().any(|s| option_name(s).as_ref() == Some(k)); + if keys.iter().any(skipped) { + out.withheld |= keys.iter().any(|k| !skipped(k)); + continue; + } + if keys.iter().any(|k| present.contains(k)) { + // Keep destination and earlier pairs unchanged. Only an identical raw pair is + // already represented; other skipped values must be reported. + out.withheld |= !query(&out.url).split('&').any(|kept| kept == pair); + continue; + } + let safe = class == NameClass::Plain && keys.iter().all(|k| SAFE_OPTIONS.contains(&k.as_str())); + if pair.contains(['{', '}']) || !(full || safe) { + out.withheld = true; + continue; + } + let fragment = out.url.find('#').unwrap_or(out.url.len()); + let separator = if out.url.get(..fragment).is_some_and(|u| u.contains('?')) { '&' } else { '?' }; + out.url.insert(fragment, separator); + out.url.insert_str(fragment + 1, pair); + present.extend(keys); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn canonical_names_remove_encoded_and_unresolved_pairs_without_touching_values() { + let over = "a".repeat(MAX_NAME_BYTES + 1); + let mut unredacted = Vec::new(); + for key in [ + "%74oken", + "%2574oken", + "to%6Ben", + "pass%77ord", + "+token", + "token%3DM", + "%FFtoken", + "ok=1;token", + "X-Amz-%53ignature", + "X-Amz-%53ecurity-Token", + "X-Amz-Credential", + "X-Amzn-RequestId-Token", + "X-Amz-Cf-Id-Secret", + "sig=M&%73v", + "\\u0074oken", + "accessKeyId", + "appid", + "APP_ID", + "%2561ppid", + "subscription-key", + "Subscription_Key", + "%73ubscription%2Dkey", + "client-key", + "clientKey", + "sas-key", + "SAS_Key", + "%2525252574oken", + over.as_str(), + ] { + let address = format!("https://h.invalid/a?{key}=RGSYNTH_name_20261103&layer=a%20b&f=json&%zz=1"); + let Public::Changed { address: public, removed, .. } = public_web_address(&address) else { + unredacted.push(key.to_string()); + continue; + }; + assert!(!public.contains("RGSYNTH_name_20261103"), "{key}: {public}"); + assert!(public.ends_with("layer=a%20b&f=json&%zz=1"), "{public}"); + assert!(removed > 0); + } + assert!(unredacted.is_empty(), "credential names left unchanged: {unredacted:?}"); + for query in [ + "client_id=public&application_id=public&map=1", + "layer=a%20b&f=json&%zz=1", + "layer=token%3DM", + "code=EPSG:4326", + "sv=2026&layer=1", + r"layer=a\b&f=json", + "", + "&", + "layer=1&&f=json", + "layer=1&", + ] { + let address = format!("https://h.invalid/a?{query}"); + assert_eq!(public_web_address(&address), Public::Unchanged); + } + for name in ["X-Amzn-RequestId", "X-Amzn-Waf-Action", "X-Amz-Cf-Id", "X-%2541mzn-RequestId"] { + assert_eq!(name_class(name, &QueryNames::default()), NameClass::Plain, "{name}"); + assert!(!log_sensitive(name), "{name}"); + } + for address in [r"https://h.invalid\a?token=RGSYNTH_boundary", r"https://h.invalid/a\b?token=RGSYNTH_boundary"] { + assert_eq!(public_web_address(address), Public::Withheld); + } + for template in [ + "https://{switch:a,b}.host/{z}/{x}/{y}@2x.png?map=m", + "https://{s}.host/{zoom}/{x}/{y}@2x.png", + " \thttps://h.invalid/a?f=json \n", + ] { + assert_eq!(public_web_address(template), Public::Unchanged); + } + assert_eq!( + public_web_address( + "https://u:RGSYNTH_template_20261104@{switch:a,b}.host/{z}/{x}/{y}@2x.png?token=RGSYNTH_query_20261104&map=m#fragment" + ), + Public::Changed { address: "https://{switch:a,b}.host/{z}/{x}/{y}@2x.png?map=m".into(), removed: 3 } + ); + assert_eq!(name_class("%2525252574oken", &QueryNames::default()), NameClass::Unresolved); + assert_eq!(name_class(&over, &QueryNames::default()), NameClass::Unresolved); + let mut callback = "https://h.invalid/cb?code=RGSYNTH_code_20261103&%73tate=flow".to_string(); + redact_query(&mut callback); + assert_eq!(callback, "https://h.invalid/cb?%73tate=flow"); + let (clean, count) = scrub_addresses( + "failed https://u:RGSYNTH_user_20261103@h.invalid/a?%74oken=RGSYNTH_query_20261103 then https://h.invalid/b?layer=a%20b\n", + ); + assert_eq!(clean, "failed https://h.invalid/a then https://h.invalid/b?layer=a%20b\n"); + assert_eq!(count, 1); + assert_eq!(scrub_addresses("(https://h.invalid/a?token=RGSYNTH_punctuation_20261103), done").0, "(https://h.invalid/a), done"); + for (text, public, count) in [ + ("read https:/h.invalid/x?token=RGSYNTH_0", "read https:", 1), + ("\t(HTTP:/h.invalid/x?token=RGSYNTH_0), done\n", "\t(HTTP:), done\n", 1), + ("[s3:bucket/x?token=RGSYNTH_0]", "[s3:]", 1), + ("https:x", "https:", 1), + ("s3:", "s3:", 1), + ("plain http://, confirm before sending", "plain http://, confirm before sending", 0), + ("use https:// for encryption", "use https:// for encryption", 0), + ("an S3:// address", "an S3:// address", 0), + ("read https:/h.invalid/x then https://h.invalid/b?layer=a%20b", "read https: then https://h.invalid/b?layer=a%20b", 1), + ( + "http://h.invalid/a https://h.invalid/b?layer=a%20b s3://bucket/x", + "http://h.invalid/a https://h.invalid/b?layer=a%20b s3://bucket/x", + 0, + ), + ("myhttps:/h.invalid/x?token=RGSYNTH_unrelated_20261104", "myhttps:/h.invalid/x?token=RGSYNTH_unrelated_20261104", 0), + ("dataset token=RGSYNTH_data_20261104 key password", "dataset token=RGSYNTH_data_20261104 key password", 0), + ] { + let (scrubbed, changed) = scrub_addresses(text); + assert_eq!((scrubbed.as_ref(), changed), (public, count), "{text}"); + if count == 0 { + assert!(matches!(scrubbed, Cow::Borrowed(_))); + } + } + assert!(matches!(scrub_addresses("dataset token key password").0, Cow::Borrowed(_))); + assert_eq!(public_web_address("/data/token?key=password"), Public::Unchanged); + assert_eq!(public_web_address("https:///path"), Public::Withheld); + assert_eq!(public_web_address("https://u:RGSYNTH_malformed_20261103/path"), Public::Withheld); + assert_eq!( + public_web_address("https://u:RGSYNTH_malformed_20261103/part@h.invalid/a"), + Public::Changed { address: "https://h.invalid/a".into(), removed: 1 } + ); + } + + #[test] + fn generated_text_scrubs_every_adjacent_address() { + for (text, expected, count) in [ + ( + "read https://a.invalid/a?layer=roads,https://b.invalid/b?token=demo", + "read https://a.invalid/a?layer=roads,https://b.invalid/b", + 1, + ), + ("([http://a.invalid/a,https://u:demo@b.invalid/b]),", "([http://a.invalid/a,https://b.invalid/b]),", 1), + ("https://a.invalid/a?token=demo;S3://bucket/b?password=demo", "https://a.invalid/a;S3://bucket/b", 2), + ( + "\t(https://a.invalid/a?layer=roads,https://b.invalid/b?layer=plots),\n", + "\t(https://a.invalid/a?layer=roads,https://b.invalid/b?layer=plots),\n", + 0, + ), + ] { + let (public, changed) = scrub_addresses(text); + assert_eq!((public.as_ref(), changed), (expected, count), "{text}"); + if count == 0 { + assert!(matches!(public, Cow::Borrowed(_)), "clean adjacent addresses need no rewrite"); + } + } + } + + #[test] + fn inherited_queries_require_the_reviewed_relationship_and_unambiguous_path() { + let source = "https://h.invalid/svc?token=RGSYNTH_inherit_20261103&k=RGSYNTH_unknown_20261103&collections=roads&f=json"; + let target = "https://h.invalid/svc/tiles/{z}/{y}/{x}"; + let kept = inherit_query(source, target, Relation::Constructed, &["f"]); + assert!(!kept.withheld); + assert!(kept.url.ends_with("?token=RGSYNTH_inherit_20261103&k=RGSYNTH_unknown_20261103&collections=roads")); + for destination in [ + target, + "https://other.invalid/svc/tiles", + "http://h.invalid/svc/tiles", + "https://h.invalid:444/svc/tiles", + "https://u@h.invalid/svc/tiles", + "https://h.invalid/svc-other/tiles", + ] { + let f3 = inherit_query(source, destination, Relation::Advertised, &["f"]); + assert_eq!(f3.url, format!("{destination}?collections=roads")); + assert!(f3.withheld); + if destination != target { + let constructed = inherit_query(source, destination, Relation::Constructed, &["f"]); + assert_eq!(constructed, f3); + } + } + for suffix in + ["../tiles", "./tiles", "%2e%2e/tiles", "%2Ftiles", "%5Ctiles", "\\tiles", "%252e/tiles", "/tiles", "tiles//x", "tiles/\nnext"] + { + let to = format!("https://h.invalid/svc/{suffix}"); + let inherited = inherit_query(source, &to, Relation::Constructed, &["f"]); + assert!(inherited.withheld && !inherited.url.contains("RGSYNTH_"), "{to}"); + } + for name in ["%2525252574oken".to_string(), "a".repeat(MAX_NAME_BYTES + 1)] { + let source = format!("https://h.invalid/svc?{name}=RGSYNTH_unresolved_20261103"); + for relation in [Relation::Constructed, Relation::Advertised] { + assert_eq!(inherit_query(&source, target, relation, &[]), Inherited { url: target.into(), withheld: true }); + } + } + let root = "https://h.invalid/?token=RGSYNTH_root_20261103&collections=x"; + assert_eq!(inherit_query(root, target, Relation::Advertised, &[]).url, format!("{target}?collections=x")); + let reviewed = [ReviewedScope { directory: "https://h.invalid/svc/", whole_origin: false }]; + assert_eq!(inherit_reviewed(source, target, Relation::Advertised, &["f"], &reviewed), kept); + assert!(inherit_reviewed(source, "https://h.invalid/svc-other/a", Relation::Advertised, &["f"], &reviewed).withheld); + let unreviewed_root = [ReviewedScope { directory: "https://h.invalid/", whole_origin: false }]; + assert!(inherit_reviewed(source, target, Relation::Advertised, &["f"], &unreviewed_root).withheld); + assert!(F2_SCOPES.is_empty()); + let to = format!("{target}?%74oken=destination-owned#look"); + let own = inherit_query(source, &to, Relation::Constructed, &["f"]); + assert!(own.url.contains("%74oken=destination-owned") && !own.url.contains("RGSYNTH_inherit")); + assert!(own.url.ends_with("#look")); + assert!(!inherit_query("https://h.invalid/svc?collections=x", target, Relation::Advertised, &[]).withheld); + for key in ["c_r_s", "%FFcollections", "+collections", "f_"] { + let source = format!("https://h.invalid/svc?{key}=RGSYNTH_alias_20261103"); + let inherited = inherit_query(&source, target, Relation::Advertised, &["f"]); + assert_eq!(inherited, Inherited { url: target.into(), withheld: true }); + } + assert_eq!( + inherit_query("https://h.invalid/svc?%63ollections=x&subset-crs=y", target, Relation::Advertised, &[]).url, + format!("{target}?%63ollections=x&subset-crs=y") + ); + let duplicate = inherit_query( + "https://h.invalid/svc?token=source;collections=x", + &format!("{target}?token=destination"), + Relation::Constructed, + &[], + ); + assert_eq!(duplicate.url, format!("{target}?token=destination")); + assert!(duplicate.withheld); + // A different value of an existing option is withheld too, including an earlier + // source option, encoded names and indivisible legacy pairs. Exact duplicates are quiet. + for relation in [Relation::Constructed, Relation::Advertised] { + for (query, destination, expected, withheld) in [ + ("subset=Lat(10,20)&subset=Long(30,40)", "", "?subset=Lat(10,20)", true), + ("%63ollections=roads", "?collections=z", "?collections=z", true), + ("collections=roads;datetime=2020", "?collections=z&datetime=2021", "?collections=z&datetime=2021", true), + ("collections=roads&collections=roads", "", "?collections=roads", false), + ("collections=roads", "?collections=roads", "?collections=roads", false), + ("collections=roads;datetime=2020", "?collections=roads;datetime=2020", "?collections=roads;datetime=2020", false), + ] { + let inherited = inherit_query(&format!("https://h.invalid/svc?{query}"), &format!("{target}{destination}"), relation, &[]); + assert_eq!(inherited, Inherited { url: format!("{target}{expected}"), withheld }, "{relation:?}: {query}"); + } + } + } +} diff --git a/crates/data/src/diagnostics.rs b/crates/data/src/diagnostics.rs index b7577f90d..7da31f998 100644 --- a/crates/data/src/diagnostics.rs +++ b/crates/data/src/diagnostics.rs @@ -491,73 +491,12 @@ fn normalized_key(key: &str) -> impl Iterator + '_ { /// Whether `key` is `target` (lower case, letters and digits) however it is written. fn key_is(key: &str, target: &str) -> bool { - normalized_key(key).eq(target.chars()) -} - -/// A name with its percent and `\u` escapes decoded, to the depth an attacker could nest them. -fn decoded_name(key: &str) -> String { - let mut name = key.to_string(); - for _ in 0..4 { - let bytes = name.as_bytes(); - let mut decoded = String::with_capacity(name.len()); - let mut i = 0; - while let Some(&b) = bytes.get(i) { - if b == b'%' - && let Some(hex) = name.get(i + 1..i + 3) - && let Ok(value) = u8::from_str_radix(hex, 16) - { - decoded.push(char::from(value)); - i += 3; - } else if b == b'\\' - && bytes.get(i + 1) == Some(&b'u') - && let Some(hex) = name.get(i + 2..i + 6) - && let Ok(value) = u32::from_str_radix(hex, 16) - && let Some(c) = char::from_u32(value) - { - decoded.push(c); - i += 6; - } else if let Some(c) = name.get(i..).and_then(|s| s.chars().next()) { - decoded.push(c); - i += c.len_utf8(); - } else { - break; - } - } - if decoded == name { - break; - } - name = decoded; - } - name + crate::address::normalized_name(key).is_some_and(|name| name == target) } +/// Shared bounded name decoding; diagnostics additionally omit log-only fields. fn secret_name(key: &str) -> bool { - // Decode only names, never values: decoding a URL value could introduce whitespace and - // incorrectly end its redacted query. Work and allocation are capped by MAX_INPUT_BYTES, and - // a name with nothing to decode is not copied. - let decoded; - let name = if key.contains(['%', '\\']) { - decoded = decoded_name(key); - decoded.as_str() - } else { - key - }; - let normalized: String = normalized_key(name).collect(); - crate::source::credentials::secret_key(name) - || crate::source::credentials::secret_key(&normalized) - || normalized.contains("credential") - || normalized.contains("connectionstring") - || normalized.contains("privatekey") - || normalized.contains("accountkey") - // A prefixed name is still one: MAPBOX_API_KEY, X_API_KEY, AWS_ACCESS_KEY_ID. - || normalized.contains("apikey") - || normalized.contains("accesskey") - || normalized.contains("sharedaccess") - || normalized.contains("body") - || matches!( - normalized.as_str(), - "basic" | "bearer" | "authentication" | "wwwauthenticate" | "proxyauthenticate" | "userid" | "username" - ) + crate::address::log_sensitive(key) } /// Whether the name is an HTTP credential header: its value is always a scheme and a credential @@ -700,6 +639,12 @@ pub fn redact(text: &str) -> String { if after.starts_with("://") { let url_end = rest.find(|c: char| c.is_whitespace() || matches!(c, '"' | '\'' | '<' | '>' | '`')).unwrap_or(rest.len()); let address = rest.get(..url_end).unwrap_or(""); + if crate::address::scheme_example(address.trim_end_matches([':', ',', ';', '.', ')', ']'])) { + out.push_str(address); + i += url_end; + field_before = false; + continue; + } match url::Url::parse(address) { // Nothing to take out: the address stays as written, its punctuation too // ("https://example.org: timeout"), unless it holds control characters. @@ -971,7 +916,7 @@ mod tests { // swallow the rest of the message. for text in [ "The saved password is missing or doesn't match this connection.", - "4 layer address(es) were saved without their keys, tokens or passwords: after RustGIS closes, add those layers again to give them", + "4 layer address(es) were saved without their keys, tokens, passwords or URL fragments; after RustGIS closes, add those layers again to restore missing values", "3 layer(s) read the database of the connection 'db' and need its password: right-click the connection in Data Sources › Connections, or one of the layers, and choose Log In…", "Set Color's step keeps its key parameter. If that is an access key, delete the step (Edit Steps) before sharing the task.", "Exported /data/a[1].png", @@ -1209,6 +1154,12 @@ mod tests { fn addresses_without_anything_to_take_out_stay_as_written() { assert_eq!(redact("Could not reach https://example.org: timeout"), "Could not reach https://example.org: timeout"); assert_eq!(redact("failed https://example.org/a/b, retrying"), "failed https://example.org/a/b, retrying"); + for text in ["plain http://, confirm before sending", "use https:// for encryption", "an S3:// address"] { + assert_eq!(redact(text), text); + } + for address in ["https://?token=RGSYNTH_example", "s3://#RGSYNTH_example", "https://u:RGSYNTH_example@"] { + assert!(!redact(address).contains("RGSYNTH_example"), "an incomplete address with content is still withheld"); + } } /// Taking user information out rewrites the address, and the punctuation that ended it in the @@ -1246,6 +1197,9 @@ mod tests { assert_eq!(redact("server: B.earer hidden"), "server: [REDACTED]"); assert_eq!(redact("server: Ba-sic dXNlcjpwYXNz"), "server: [REDACTED]"); assert_eq!(redact("pass%77ord=x"), "[REDACTED]"); + for key in ["%2574oken", "%2525252574oken"] { + assert!(!redact(&format!("{key}=RGSYNTH_diagnostic_20261103")).contains("RGSYNTH_")); + } } #[test] diff --git a/crates/data/src/lib.rs b/crates/data/src/lib.rs index 3a06b3062..6b141ff33 100644 --- a/crates/data/src/lib.rs +++ b/crates/data/src/lib.rs @@ -4,6 +4,7 @@ //! bad features are skipped and counted in [`ReadReport`]. #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] +pub mod address; pub mod cityjson; pub mod color; mod dbf; diff --git a/crates/data/src/ogc.rs b/crates/data/src/ogc.rs index d2916e356..fa7684594 100644 --- a/crates/data/src/ogc.rs +++ b/crates/data/src/ogc.rs @@ -442,6 +442,8 @@ pub struct Service { /// Some advertised collections were left out (past the listing limit, or with an items /// address RustGIS doesn't fetch): the listing isn't the whole service. pub incomplete: bool, + /// Destinations that did not inherit one or more query pairs; reported by the caller. + pub withheld_addresses: u32, } /// Most layers listed from one service. @@ -531,7 +533,7 @@ fn is_data_link(link: &serde_json::Value) -> bool { } /// Chooses JSON before an untyped data link, without guessing an endpoint or copying credentials. -fn collections_link(address: &str, links: &[serde_json::Value]) -> Option { +fn collections_link(address: &str, links: &[serde_json::Value]) -> Option<(String, bool)> { let base = url::Url::parse(crate::login_address::inner(address)).ok()?; links .iter() @@ -553,7 +555,12 @@ fn collections_link(address: &str, links: &[serde_json::Value]) -> Option(url: &'a str, key: &str) -> Option<&'a str> { let q = url.split('#').next()?.split_once('?')?.1; q.split('&').find_map(|kv| { let (k, v) = kv.split_once('=').unwrap_or((kv, "")); - k.eq_ignore_ascii_case(key).then_some(v) + (if key == "token" { crate::address::normalized_name(k).as_deref() == Some("token") } else { k.eq_ignore_ascii_case(key) }) + .then_some(v) }) } @@ -2275,7 +2283,8 @@ fn arcgis(url: &str, v: &serde_json::Value) -> Result { max_paged_features() )); } - // A secured service's token goes with every request. + // Offers are constructed under the service. A suffix supplied by its document still has + // to pass the raw-path check before it may receive the service's token. if let Some(token) = query_value(url, "token").filter(|t| !t.is_empty()) { for l in &mut service.layers { match &mut l.offer { @@ -2284,7 +2293,15 @@ fn arcgis(url: &str, v: &serde_json::Value) -> Result { | Offer::Style { url: u } | Offer::VectorTiles { template: u, .. } | Offer::Service { url: u } - | Offer::Coverage(Coverage { endpoint: u, .. }) => *u = with_query(u, &format!("token={token}")), + | Offer::Coverage(Coverage { endpoint: u, .. }) => { + if crate::address::Scope::of(crate::login_address::inner(url)) + .is_some_and(|scope| scope.covers_below(crate::login_address::inner(u))) + { + *u = with_query(u, &format!("token={token}")); + } else { + service.withheld_addresses = service.withheld_addresses.saturating_add(1); + } + } } } } @@ -2395,10 +2412,14 @@ fn ogc_api(url: &str, v: &serde_json::Value) -> Result { let title = v.get("title").and_then(|t| t.as_str()).unwrap_or_default().to_string(); let mut service = Service { kind: "OGC API Tiles".into(), title: title.clone(), ..Service::default() }; let links = v.get("links").and_then(|l| l.as_array()).cloned().unwrap_or_default(); - service.collections_url = collections_link(url, &links); + if let Some((next, withheld)) = collections_link(url, &links) { + service.collections_url = Some(next); + service.withheld_addresses = u32::from(withheld); + } let name = if title.is_empty() { "tiles".to_string() } else { title }; // The dataset's layers aren't listed here, so its vector tiles can't be styled: map tiles only. - if let Some((t, list)) = tileset(url, &links, TileRel::Map) { + if let Some((t, list, withheld)) = tileset(url, &links, TileRel::Map) { + service.withheld_addresses = service.withheld_addresses.saturating_add(u32::from(withheld)); service.layers.push(ServiceLayer { zooms: None, name: "map".into(), @@ -2452,7 +2473,8 @@ fn ogc_api(url: &str, v: &serde_json::Value) -> Result { // tiles' layer is named after the collection, as the standard has it. let map = tileset(url, &links, TileRel::Map); let vector = tileset(url, &links, TileRel::Vector); - if let Some((t, list)) = &map { + if let Some((t, list, withheld)) = &map { + service.withheld_addresses = service.withheld_addresses.saturating_add(u32::from(*withheld)); service.layers.push(ServiceLayer { zooms: None, name: format!("{id} (map tiles)"), @@ -2465,7 +2487,8 @@ fn ogc_api(url: &str, v: &serde_json::Value) -> Result { legends: Vec::new(), }); } - if let Some((t, list)) = &vector { + if let Some((t, list, withheld)) = &vector { + service.withheld_addresses = service.withheld_addresses.saturating_add(u32::from(*withheld)); service.layers.push(ServiceLayer { zooms: None, name: format!("{id} (vector tiles)"), @@ -2578,9 +2601,9 @@ enum TileRel { /// `tilesets-vector` link: the template at the usual path under the tileset list /// (`…/WebMercatorQuad/{tileMatrix}/{tileRow}/{tileCol}`), and the list's address (JSON), whose /// WebMercatorQuad tileset advertises the template [`resolve_tiles`] reads (the usual path is what -/// is used when that can't be read). Both keep the link's query (a token, a collections choice), -/// all but `f`. -fn tileset(from: &str, links: &[serde_json::Value], which: TileRel) -> Option<(String, String)> { +/// is used when that can't be read). The constructed template keeps the link's query except +/// `f` only under its checked source path; the third result counts a withheld destination. +fn tileset(from: &str, links: &[serde_json::Value], which: TileRel) -> Option<(String, String, bool)> { let suffix = match which { TileRel::Map => "tilesets-map", TileRel::Vector => "tilesets-vector", @@ -2604,23 +2627,17 @@ fn tileset(from: &str, links: &[serde_json::Value], which: TileRel) -> Option<(S TileRel::Vector => format!("{at}/WebMercatorQuad/{{z}}/{{y}}/{{x}}?f=mvt"), }; let list = if query_value(&href, "f").is_some() { href.clone() } else { with_query(&href, "f=json") }; - Some((carry_query(&template, &href), list)) + let inherited = crate::address::inherit_query( + crate::login_address::inner(&href), + crate::login_address::inner(&template), + crate::address::Relation::Constructed, + &["f"], + ); + Some((crate::login_address::inherit(&template, &inherited.url), list, inherited.withheld)) } -/// `template` with the query parameters of `link` it doesn't have, but `f` (the representation -/// the link was for). -fn carry_query(template: &str, link: &str) -> String { - let q = link.split('#').next().and_then(|u| u.split_once('?')).map(|(_, q)| q).unwrap_or(""); - let mut out = template.to_string(); - for kv in q.split('&').take(32) { - let k = kv.split_once('=').map(|(k, _)| k).unwrap_or(kv); - if kv.is_empty() || k.eq_ignore_ascii_case("f") || query_value(&out, k).is_some() || kv.contains(['{', '}']) { - continue; - } - out = with_query(&out, kv); - } - out -} +/// Notice template shared by native discovery and the translated asynchronous UI resolver. +pub const QUERY_WITHHELD_NOTICE: &str = "{n} address(es) the service listed didn't get its keys or other settings: RustGIS sends those only to addresses it builds itself from the one given; add such an address with its own key if it needs one"; /// What an OGC API tiles document says on the way to a tile template: the template, or the /// address of the document to read next (a tileset list's WebMercatorQuad tileset). @@ -2630,15 +2647,43 @@ pub enum TilesStep { Next(String), } +/// One discovery result and the number of destinations that lost query settings. Callers must +/// report this count, including when a later fetch fails and falls back to a constructed URL. +#[derive(Clone, Debug, PartialEq)] +pub struct TilesResolution { + pub step: TilesStep, + pub withheld_addresses: u32, +} + +/// Applies the advertised-link policy; the document cannot claim a constructed relationship. +fn advertised_tiles(list: &str, to: &str, next: bool) -> TilesResolution { + let inherited = crate::address::inherit_query( + crate::login_address::inner(list), + crate::login_address::inner(to), + crate::address::Relation::Advertised, + &["f"], + ); + // Preserve only the destination's binding: a foreign document cannot recover a login + // merely by linking back to the original query source's origin. + let url = crate::login_address::inherit(to, &inherited.url); + let step = if next { + TilesStep::Next(if query_value(&url, "f").is_some() { url } else { with_query(&url, "f=json") }) + } else { + TilesStep::Template(url) + }; + TilesResolution { step, withheld_addresses: u32::from(inherited.withheld) } +} + /// One step of reading an OGC API tile template: `body` is the tileset list (or a tileset) -/// fetched from `doc`; `list` is the list's address, whose query the template keeps. `None` when +/// fetched from `doc`; `list` is the query source. Advertised links receive SAFE_OPTIONS only; +/// withheld settings are counted for the caller's notice. `None` when /// the document names no WebMercatorQuad tileset or tile template. -pub fn resolve_tiles(list: &str, doc: &str, body: &[u8], vector: bool) -> Option { +pub fn resolve_tiles(list: &str, doc: &str, body: &[u8], vector: bool) -> Option { let v: serde_json::Value = serde_json::from_slice(body).ok()?; let links = |v: &serde_json::Value| v.get("links").and_then(|l| l.as_array()).cloned().unwrap_or_default(); let Some(sets) = v.get("tilesets").and_then(|t| t.as_array()) else { // A tileset: its tile template. - return item_template(doc, &links(&v), vector).map(|t| TilesStep::Template(carry_query(&t, list))); + return item_template(doc, &links(&v), vector).map(|t| advertised_tiles(list, &t, false)); }; let mercator = |t: &serde_json::Value| { let tiling = links(t); @@ -2662,14 +2707,13 @@ pub fn resolve_tiles(list: &str, doc: &str, body: &[u8], vector: bool) -> Option let t = sets.iter().take(200).find(|t| mercator(t) && kind_fits(t))?; let ls = links(t); if let Some(template) = item_template(doc, &ls, vector) { - return Some(TilesStep::Template(carry_query(&template, list))); + return Some(advertised_tiles(list, &template, false)); } let next = ls .iter() .find(|l| l.get("rel").and_then(|r| r.as_str()).is_some_and(|r| r == "self" || r.ends_with("/tileset"))) .and_then(|l| resolve(doc, l.get("href")?.as_str()?))?; - let next = carry_query(&next, list); - Some(TilesStep::Next(if query_value(&next, "f").is_some() { next } else { with_query(&next, "f=json") })) + Some(advertised_tiles(list, &next, true)) } /// A tileset's advertised tile template (its `item` link, `{tileMatrix}/{tileRow}/{tileCol}`) as @@ -3956,11 +4000,12 @@ mod tests { let address = crate::login_address::bind(binding, origin, "https://maps.example.org/api?key=a%2Fb").unwrap(); for href in ["/collections", "https://MAPS.example.org:443/collections"] { let links = serde_json::json!([{"rel":"data","type":"application/json","href":href}]); - let next = collections_link(&address, links.as_array().unwrap()).unwrap(); + let (next, withheld) = collections_link(&address, links.as_array().unwrap()).unwrap(); assert_eq!(crate::login_address::parse(&next).unwrap().unwrap().binding, binding); + assert!(withheld, "the advertised destination omits the original key"); } let links = serde_json::json!([{"rel":"data","type":"application/json","href":"https://other.example.org/collections"}]); - assert_eq!(collections_link(&address, links.as_array().unwrap()).unwrap(), "https://other.example.org/collections"); + assert_eq!(collections_link(&address, links.as_array().unwrap()).unwrap(), ("https://other.example.org/collections".into(), true)); } #[test] @@ -6190,6 +6235,21 @@ mod tests { let s = parse("https://gis.example.org/arcgis/rest/services/P/FeatureServer?token=abc", fs).unwrap(); let Offer::Features { url } = &s.layers[0].offer else { panic!() }; assert!(url.ends_with("&token=abc"), "{url}"); + // A bound service constructs the same requests while retaining only its own login. + for address in [ + "https://gis.example.org/arcgis/rest/services/P/FeatureServer", + "https://gis.example.org/arcgis/rest/services/P/FeatureServer?token=abc", + ] { + let expected = parse(address, fs).unwrap(); + let bound = crate::login_address::bind("0123456789abcdef", "https://gis.example.org", address).unwrap(); + let service = parse(&bound, fs).unwrap(); + let Offer::Features { url } = &service.layers[0].offer else { panic!("features expected") }; + let offered = crate::login_address::parse(url).unwrap().unwrap(); + assert_eq!((offered.binding, offered.origin), ("0123456789abcdef", "https://gis.example.org")); + let Offer::Features { url: expected } = &expected.layers[0].offer else { panic!("features expected") }; + assert_eq!(offered.url, expected, "{address}"); + assert_eq!(service.withheld_addresses, 0, "{address}"); + } } #[test] @@ -6250,6 +6310,18 @@ mod tests { assert_eq!(chosen.collections_url.as_deref(), Some("https://example.org/json")); } + #[test] + fn collections_links_count_only_settings_the_selected_link_omits() { + let body = br#"{"links":[{"rel":"data","type":"application/json","href":"/catalog?f=json"}]}"#; + for (from, count) in + [("https://collections.invalid/root?f=json", 0), ("https://collections.invalid/root?f=json&token=RGSYNTH_link", 1)] + { + let service = parse(from, body).unwrap(); + assert_eq!(service.collections_url.as_deref(), Some("https://collections.invalid/catalog?f=json")); + assert_eq!(service.withheld_addresses, count); + } + } + #[test] fn ogc_json_format_is_idempotent() { let url = capabilities_url("https://example.org/api?f=html&key=a%2Fb&f=json#fragment", Some("ogcapi")); @@ -6327,6 +6399,25 @@ mod tests { assert_eq!(s.layers[0].tilesets.as_deref(), Some("https://x.org/c/roads/map/tiles?token=abc&f=json")); } + #[test] + fn bound_tiles_keep_query_rules_without_rebinding_after_a_foreign_hop() { + let origin = "https://x.org"; + let bound = crate::login_address::bind("0123456789abcdef", origin, "https://x.org/c").unwrap(); + let document = br#"{"collections":[{"id":"roads","links":[{"rel":"tilesets-map","href":"https://x.org/c/roads/tiles?token=RGSYNTH_tiles_r15&f=json"}]}]}"#; + let service = parse(&bound, document).unwrap(); + let Offer::Tiles { template } = &service.layers[0].offer else { panic!("tiles expected") }; + let template = crate::login_address::parse(template).unwrap().unwrap(); + assert_eq!(template.url, "https://x.org/c/roads/tiles/WebMercatorQuad/{z}/{y}/{x}?token=RGSYNTH_tiles_r15"); + assert_eq!(service.withheld_addresses, 0); + let list = service.layers[0].tilesets.as_deref().unwrap(); + let foreign = br#"{"links":[{"rel":"item","type":"image/png","href":"https://x.org/back/{tileMatrix}/{tileRow}/{tileCol}.png"}]}"#; + let resolution = resolve_tiles(list, "https://foreign-r15.example.invalid/tiles", foreign, false).unwrap(); + let TilesStep::Template(url) = resolution.step else { panic!("template expected") }; + assert_eq!(url, "https://x.org/back/{z}/{y}/{x}.png"); + assert!(crate::login_address::parse(&url).unwrap().is_none()); + assert_eq!(resolution.withheld_addresses, 1); + } + #[test] fn ogc_api_advertised_tile_templates() { let list = "https://x.org/c/roads/tiles?token=abc&f=json"; @@ -6336,35 +6427,65 @@ mod tests { {"tileMatrixSetURI": "http://www.opengis.net/def/tilematrixset/OGC/1.0/WorldCRS84Quad", "links": [{"rel": "self", "href": "crs84"}]}, {"dataType": "vector", "tileMatrixSetURI": "http://www.opengis.net/def/tilematrixset/OGC/1.0/WebMercatorQuad", "links": [{"rel": "self", "href": "https://x.org/c/roads/tiles/smerc"}]}]}"#; assert_eq!( - resolve_tiles(list, list, sets, true), - Some(TilesStep::Next("https://x.org/c/roads/tiles/smerc?token=abc&f=json".into())) + resolve_tiles(list, list, sets, true).map(|r| r.step), + Some(TilesStep::Next("https://x.org/c/roads/tiles/smerc?f=json".into())) ); - assert_eq!(resolve_tiles(list, list, sets, false), None, "no map tileset in WebMercatorQuad"); - // The tileset's item links: MVT for vector tiles, PNG first for map tiles, the token kept. + assert_eq!(resolve_tiles(list, list, sets, false).map(|r| r.step), None, "no map tileset in WebMercatorQuad"); + // The tileset's item links: MVT for vector tiles, PNG first for map tiles, the list's token withheld under F3. let tileset = br#"{"links": [ {"rel": "item", "type": "image/jpeg", "templated": true, "href": "https://x.org/c/roads/tiles/smerc/{tileMatrix}/{tileRow}/{tileCol}.jpg"}, {"rel": "item", "type": "image/png", "templated": true, "href": "https://x.org/c/roads/tiles/smerc/{tileMatrix}/{tileRow}/{tileCol}.png"}, {"rel": "item", "type": "application/vnd.mapbox-vector-tile", "templated": true, "href": "smerc/{tileMatrix}/{tileRow}/{tileCol}.mvt"}]}"#; let doc = "https://x.org/c/roads/tiles/smerc?token=abc&f=json"; assert_eq!( - resolve_tiles(list, doc, tileset, true), - Some(TilesStep::Template("https://x.org/c/roads/tiles/smerc/{z}/{y}/{x}.mvt?token=abc".into())) + resolve_tiles(list, doc, tileset, true).map(|r| r.step), + Some(TilesStep::Template("https://x.org/c/roads/tiles/smerc/{z}/{y}/{x}.mvt".into())) ); assert_eq!( - resolve_tiles(list, doc, tileset, false), - Some(TilesStep::Template("https://x.org/c/roads/tiles/smerc/{z}/{y}/{x}.png?token=abc".into())) + resolve_tiles(list, doc, tileset, false).map(|r| r.step), + Some(TilesStep::Template("https://x.org/c/roads/tiles/smerc/{z}/{y}/{x}.png".into())) ); // A list that carries its tilesets' item links needs no second document. let inline = br#"{"tilesets": [{"tileMatrixSetId": "WebMercatorQuad", "links": [ {"rel": "item", "type": "image/png", "href": "https://x.org/t/{tileMatrix}/{tileRow}/{tileCol}"}]}]}"#; - assert_eq!(resolve_tiles(list, list, inline, false), Some(TilesStep::Template("https://x.org/t/{z}/{y}/{x}?token=abc".into()))); + assert_eq!( + resolve_tiles(list, list, inline, false).map(|r| r.step), + Some(TilesStep::Template("https://x.org/t/{z}/{y}/{x}".into())) + ); // A template with other variables, or no web address, isn't taken. for bad in [ br#"{"links": [{"rel": "item", "href": "https://x.org/{style}/{tileMatrix}/{tileRow}/{tileCol}"}]}"#.as_slice(), br#"{"links": [{"rel": "item", "href": "javascript:{tileMatrix}"}]}"#.as_slice(), b"not json", ] { - assert_eq!(resolve_tiles(list, doc, bad, false), None); + assert_eq!(resolve_tiles(list, doc, bad, false).map(|r| r.step), None); + } + // F3 is independent of origin and directory containment. Destination-owned keys stay; + // only the allowlisted collection choice may follow from the list. + let list = "https://h.invalid/svc?token=RGSYNTH_list_20261103&k=RGSYNTH_unknown_20261103&collections=roads"; + for host in ["h.invalid", "elsewhere.invalid"] { + let doc = format!( + r#"{{"links":[{{"rel":"item","type":"image/png","href":"https://{host}/svc/{{tileMatrix}}/{{tileRow}}/{{tileCol}}?own=destination"}}]}}"# + ); + let resolved = resolve_tiles(list, list, doc.as_bytes(), false).unwrap(); + assert_eq!(resolved.withheld_addresses, 1); + let TilesStep::Template(template) = resolved.step else { panic!("expected template") }; + assert_eq!(template, format!("https://{host}/svc/{{z}}/{{y}}/{{x}}?own=destination&collections=roads")); + } + let clean = "https://h.invalid/svc?collections=roads"; + assert_eq!(resolve_tiles(clean, clean, tileset, false).unwrap().withheld_addresses, 0); + // Even a suffix appended by RustGIS must pass the raw-path check. The normal style + // stays under its service; an advertised '..' style does not inherit its token. + let address = "https://h.invalid/rest/services/a/VectorTileServer?token=RGSYNTH_style_20261103"; + let bound = crate::login_address::bind("0123456789abcdef", "https://h.invalid", address).unwrap(); + for (style, withheld) in [("resources/styles", 0), ("../other/styles", 1), ("%2e%2e/other/styles", 1)] { + let description = serde_json::json!({"defaultStyles": style}); + for address in [address, bound.as_str()] { + let service = parse(address, description.to_string().as_bytes()).unwrap(); + assert_eq!(service.withheld_addresses, withheld, "{address}: {style}"); + let Offer::Style { url } = &service.layers[0].offer else { panic!("expected style") }; + assert_eq!(url.contains("RGSYNTH_style_20261103"), withheld == 0, "{address}: {style}"); + } } } diff --git a/crates/data/src/source/credentials.rs b/crates/data/src/source/credentials.rs index cee9f13b4..e649530f8 100644 --- a/crates/data/src/source/credentials.rs +++ b/crates/data/src/source/credentials.rs @@ -1,97 +1,3 @@ -//! Secret-name and query filtering shared by connection addresses and saved sources. +//! Compatibility path for the shared credential-name and query policy. -/// Whether a key names a secret, wherever it is: an exact list (case aside), and any key -/// naming a password, secret or token. Context-dependent query keys are handled by [`redact_query`]. -pub fn secret_key(k: &str) -> bool { - secret_key_lower(&k.to_ascii_lowercase()) -} - -fn secret_key_lower(k: &str) -> bool { - matches!( - k, - "token" - | "access_token" - | "id_token" - | "refresh_token" - | "api_key" - | "apikey" - | "api-key" - | "x-api-key" - | "key" - | "access_key" - | "accesskey" - | "access_key_id" - | "accesskeyid" - | "secret" - | "client_secret" - | "password" - | "passwd" - | "pwd" - | "sig" - | "signature" - | "sas" - | "authorization" - | "proxy-authorization" - | "cookie" - | "set-cookie" - | "bearer" - ) || k.starts_with("x-amz-") - || k.starts_with("x-goog-") - || k.contains("password") - || k.contains("secret") - || k.contains("token") -} - -/// The query keys of an Azure shared access signature, secret only with both `sig` and `sv`. -const SAS_KEYS: &[&str] = - &["sv", "se", "sp", "sr", "st", "spr", "sip", "si", "ss", "srt", "skoid", "sktid", "skt", "ske", "sks", "skv", "sdd"]; - -fn secret_in(k: &str, keys: &[String]) -> bool { - let has = |x: &str| keys.iter().any(|q| q == x); - secret_key_lower(k) || (has("sig") && has("sv") && SAS_KEYS.contains(&k)) || (k == "code" && has("state")) -} - -/// Removes secret query pairs without changing the address's user-info or connection options. -/// Callers must remove passwords from user-info and fragments before calling this function. -pub fn redact_query(address: &mut String) { - let Some(at) = address.find('?') else { return }; - let query = address.get(at + 1..).unwrap_or(""); - let keys: Vec = query.split('&').map(|kv| kv.split_once('=').map_or(kv, |(k, _)| k).to_ascii_lowercase()).collect(); - if query.split('&').zip(&keys).all(|(kv, key)| !kv.is_empty() && !secret_in(key, &keys)) { - return; - } - let mut kept = String::with_capacity(query.len()); - for (kv, key) in query.split('&').zip(&keys) { - if kv.is_empty() || secret_in(key, &keys) { - continue; - } - if !kept.is_empty() { - kept.push('&'); - } - kept.push_str(kv); - } - address.truncate(at); - if !kept.is_empty() { - address.push('?'); - address.push_str(&kept); - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn access_key_ids_and_their_secrets_are_secret_names() { - for k in ["accessKeyId", "access_key_id", "AccessKeyID", "secretAccessKey", "sessionToken", "X-Amz-Credential"] { - assert!(secret_key(k), "{k}"); - } - for k in ["remember", "region", "buckets", "endpoint"] { - assert!(!secret_key(k), "{k}"); - } - // In a query, the key id goes with the signature. - let mut a = "https://s3.example.org/b/k?accessKeyId=AKIDEXAMPLE&prefix=x".to_string(); - redact_query(&mut a); - assert_eq!(a, "https://s3.example.org/b/k?prefix=x"); - } -} +pub use crate::address::{NameClass, QueryNames, log_sensitive, name_class, normalized_name, redact_query, secret_key}; diff --git a/crates/data/src/source/pgconn.rs b/crates/data/src/source/pgconn.rs index 972b41930..09d2210ea 100644 --- a/crates/data/src/source/pgconn.rs +++ b/crates/data/src/source/pgconn.rs @@ -14,21 +14,28 @@ pub fn public_label(s: &str) -> Option { let t = s.trim(); let lower = t.to_ascii_lowercase(); if lower.starts_with("postgresql://") || lower.starts_with("postgres://") { - let (no_userinfo, _) = split_secrets(t, &["password"]); - // A secret in the query (`?password=…`) goes too. - return Some(match no_userinfo.split_once('?') { - Some((base, q)) => { - let kept: Vec<&str> = q - .split('&') - .filter(|kv| { - let k = kv.split_once('=').map(|(k, _)| k).unwrap_or(kv); - !kv.is_empty() && !SECRET_KEYS.iter().any(|s| k.eq_ignore_ascii_case(s)) - }) - .collect(); - if kept.is_empty() { base.to_string() } else { format!("{base}?{}", kept.join("&")) } - } - None => no_userinfo, - }); + let (_, raw) = t.split_once("://")?; + let authority = raw.split(['/', '?', '#']).next()?; + let path = raw.get(authority.len()..)?.split(['?', '#']).next()?; + // A path `@` after userinfo, or several path `@`s, can be a malformed password tail. + // Refuse ambiguous output rather than choosing another host; percent-encode a + // database name's `@` to keep it. + if (authority.contains('@') && path.contains('@')) || path.matches('@').nth(1).is_some() { + return None; + } + let public = split_secrets(t, SECRET_KEYS).0; + let (_, rest) = public.split_once("://")?; + let authority = rest.split(['/', '?', '#']).next()?; + let hosts = authority.rsplit_once('@').map_or(authority, |(_, hosts)| hosts); + // A non-numeric port with no delimited password is an unreadable authority. Multi-host + // URIs, IPv6 and empty hosts (the local socket) keep libpq's existing spelling. + if hosts.split(',').any(|host| { + let port = if host.starts_with('[') { host.split_once(']').map(|(_, rest)| rest) } else { Some(host) }; + port.and_then(|h| h.split_once(':')).is_some_and(|(_, p)| !p.bytes().all(|b| b.is_ascii_digit())) + }) { + return None; + } + return Some(public); } let rest = ["postgresql:", "postgres:", "pg:"].iter().find(|p| lower.starts_with(**p)).and_then(|p| t.get(p.len()..))?; let (public, _) = split_secrets(rest.trim(), SECRET_KEYS); @@ -122,13 +129,17 @@ fn query_secrets(s: &str, secrets: &[&str]) -> (String, Option) { let mut kept = Vec::new(); for kv in query.split('&').filter(|kv| !kv.is_empty()) { let (k, v) = kv.split_once('=').unwrap_or((kv, "")); - if k.eq_ignore_ascii_case("password") { + if crate::address::normalized_name(k).as_deref() == Some("password") { + // The URI driver decodes the name and password once; keyword strings below are + // intentionally not percent-decoded. password = Some(percent_decode(v)); - } else if !secrets.iter().any(|s| k.eq_ignore_ascii_case(s)) { + } else if !secrets.iter().any(|s| crate::address::normalized_name(k).as_deref() == Some(s)) { kept.push(kv); } } - (if kept.is_empty() { base.to_string() } else { format!("{base}?{}", kept.join("&")) }, password) + let mut public = if kept.is_empty() { base.to_string() } else { format!("{base}?{}", kept.join("&")) }; + crate::address::redact_query(&mut public); + (public, password) } /// A libpq keyword/value connection string's pairs (password included), in order; a value is a @@ -221,5 +232,33 @@ mod tests { assert_eq!(pw.as_deref(), Some("s@c")); let (public, pw) = super::split_secrets("postgresql://h/g?password=x", &["password"]); assert_eq!((public.as_str(), pw.as_deref()), ("postgresql://h/g", Some("x"))); + let uri = "postgresql://u@h.invalid/db?pass%77ord=RGSYNTH_pg_20261103&sslmode=require&pass%66ile=RGSYNTH_file_20261103"; + assert_eq!(super::public_label(uri).as_deref(), Some("postgresql://u@h.invalid/db?sslmode=require")); + let (public, password) = super::query_secrets(uri, super::SECRET_KEYS); + assert_eq!(public, "postgresql://u@h.invalid/db?sslmode=require"); + assert_eq!(password.as_deref(), Some("RGSYNTH_pg_20261103")); + let clean = "postgresql://u@h.invalid/db?sslmode=require"; + assert_eq!(super::public_label(clean).as_deref(), Some(clean)); + // A later path `@` could be a password tail, even after an earlier split removed its + // colon. Withhold the ambiguous label, including an intentional raw `@` database name; + // percent-encoding the database's `@` keeps that name unambiguous. + for ambiguous in [ + "postgresql://user:p@ss/word@host.invalid/db", + "postgresql://user@ss/word@host.invalid/db", + "postgresql://user@host.invalid/db@tag", + "postgresql://user:pass/word@host.invalid/db@tag", + "postgresql://user:pass/word@host.invalid/db@tag@tail", + ] { + assert_eq!(super::public_label(ambiguous), None, "{ambiguous}"); + } + let encoded = "postgresql://user:p%40ss%2Fword@host.invalid/db%40tag"; + assert_eq!(super::public_label(encoded).as_deref(), Some("postgresql://user@host.invalid/db%40tag")); + let query_at = "postgresql://user@host.invalid/db?application_name=x@y"; + assert_eq!(super::public_label(query_at).as_deref(), Some(query_at)); + assert_eq!(super::public_label("postgresql://u:RGSYNTH_unreadable_20261103/path"), None); + let keyword = "host=h.invalid dbname='a%20b' user=u password='RGSYNTH_keyword_20261103'"; + let (public, password) = super::split_secrets(keyword, super::SECRET_KEYS); + assert_eq!(public, "host=h.invalid dbname=a%20b user=u"); + assert_eq!(password.as_deref(), Some("RGSYNTH_keyword_20261103")); } } diff --git a/crates/engine/src/cmd/arcgis.rs b/crates/engine/src/cmd/arcgis.rs index 3d7fc6ab9..c9582dc2e 100644 --- a/crates/engine/src/cmd/arcgis.rs +++ b/crates/engine/src/cmd/arcgis.rs @@ -10,22 +10,22 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("arcgis.signIn", "Sign In to ArcGIS", "Home › Add › Service", sign_in).view().params( + C::new("arcgis.signIn", "Sign In to ArcGIS", "Home › Add › Service", sign_in).roles(&[]).view().params( r#"{"portal"?: "the portal's address: an ArcGIS Online organization's (https://myorg.maps.arcgis.com) or an ArcGIS Enterprise portal's (https://gis.example.org/portal); ArcGIS Online when blank", "clientId"?: "the client id of an OAuth app registered on the portal (Developer credentials) whose redirect URLs list http://127.0.0.1:7290/arcgis/callback; RustGIS's own app when absent on ArcGIS Online (required for ArcGIS Enterprise)", "port"?: "the callback server's port (7290; the redirect URL registered must name it)"} → the sign-in page to open in the browser; the session is kept when the browser comes back (arcgis.status tells). Desktop and command line only"#, ), - C::new("arcgis.token", "Use ArcGIS Token", "Home › Add › Service", token).view().params( + C::new("arcgis.token", "Use ArcGIS Token", "Home › Add › Service", token).roles(&[]).view().params( r#"{"url": "an ArcGIS REST address on https:// (a service's, …/rest/services, or a portal's …/sharing/rest)", "token": "an ArcGIS token for it"} — kept for the session only, never in the project; sent to that ArcGIS Server's (or portal's) requests only. A service address given to service.layers or service.add with ?token=… does the same"#, ), - C::new("arcgis.signOut", "Sign Out of ArcGIS", "Home › Add › Service", sign_out) + C::new("arcgis.signOut", "Sign Out of ArcGIS", "Home › Add › Service", sign_out).roles(&[]) .view() .params(r#"{"portal"?: "a signed-in portal's address, or a server a token was given for (arcgis.status lists them); every one when absent"}"#), C::new("arcgis.cancelSignIn", "Cancel ArcGIS Sign-In", "Home › Add › Service", |_s, _p| { #[cfg(not(target_arch = "wasm32"))] arcgis_auth::cancel_sign_in(); Ok(status()) - }) + }).roles(&[]) .view(), - C::new("arcgis.status", "ArcGIS Sign-In Status", "Home › Add › Service", |_s, _p| Ok(status())).view(), + C::new("arcgis.status", "ArcGIS Sign-In Status", "Home › Add › Service", |_s, _p| Ok(status())).roles(&[]).view(), ] } diff --git a/crates/engine/src/cmd/catalogs.rs b/crates/engine/src/cmd/catalogs.rs index 3e13c7d13..df92b5189 100644 --- a/crates/engine/src/cmd/catalogs.rs +++ b/crates/engine/src/cmd/catalogs.rs @@ -10,13 +10,13 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("overture.add", "Add Overture Maps Data", "Home › Add › Web Data", overture_add).params( + C::new("overture.add", "Add Overture Maps Data", "Home › Add › Web Data", overture_add).roles(&[]).params( r#"{"theme": "buildings"|"places"|"transportation"|"divisions"|"base"|"addresses", "type"?: "the theme's type (building, building_part; place; segment, connector; division_area, division, division_boundary; land_use, land, land_cover, water, infrastructure, bathymetry; address)", "release"?: "such as 2026-09-23.1 (default: the latest)", "bbox"?: [west, south, east, north] | "view" (the map's extent, the default), "tiles"?: bool (the whole world as map tiles, from the theme's PMTiles archive, instead of the features in view), "files"?: [the files' GeoParquet addresses, when already found and opened (the app's background requests)]}"#, ), - C::new("sourcecoop.products", "List Source Cooperative Products", "Home › Add › Web Data", sourcecoop_products) + C::new("sourcecoop.products", "List Source Cooperative Products", "Home › Add › Web Data", sourcecoop_products).roles(&[]) .view() .params(r#"{"account"?: "an account's products (default: the featured ones)"}"#), - C::new("sourcecoop.list", "List Source Cooperative Files", "Home › Add › Web Data", sourcecoop_list).view().params( + C::new("sourcecoop.list", "List Source Cooperative Files", "Home › Add › Web Data", sourcecoop_list).roles(&[]).view().params( r#"{"account": text, "product": text, "path"?: "a folder inside the product, ending in /", "token"?: "the next page's, from a listing"} (a file's "url" is added with layer.addUrl)"#, ), ] diff --git a/crates/engine/src/cmd/clipboard.rs b/crates/engine/src/cmd/clipboard.rs index 5d67bfd75..d9ce18c33 100644 --- a/crates/engine/src/cmd/clipboard.rs +++ b/crates/engine/src/cmd/clipboard.rs @@ -20,15 +20,17 @@ pub struct Clipboard { pub fn commands() -> Vec { vec![ - C::new("edit.copy", "Copy", "Edit › Clipboard", |s, _| copy(s)).view().key("Mod+C").when(need_selection), - C::new("edit.cut", "Cut", "Edit › Clipboard", cut).key("Mod+X").when(super::design::need_editable_selection), + C::new("edit.copy", "Copy", "Edit › Clipboard", |s, _| copy(s)).roles(&[]).view().key("Mod+C").when(need_selection), + C::new("edit.cut", "Cut", "Edit › Clipboard", cut).roles(&[]).key("Mod+X").when(super::design::need_editable_selection), C::new("edit.paste", "Paste", "Edit › Clipboard", paste) + .roles(&[]) .key("Mod+V") .when(super::design::need_editable) .params(r#"{"layer"?, "text"?: GeoJSON text (the copied features when omitted)}"#), C::new("edit.pasteSpecial", "Paste Special", "Edit › Clipboard", paste_special) + .roles(&[]) .params(r#"{"name"?: string, "text"?: GeoJSON text (the copied features when omitted)}"#), - C::new("layer.copyPath", "Copy Path", "Edit › Clipboard", copy_path).view().when(need_layer).params(r#"{"layer"?}"#), + C::new("layer.copyPath", "Copy Path", "Edit › Clipboard", copy_path).roles(&[]).view().when(need_layer).params(r#"{"layer"?}"#), ] } diff --git a/crates/engine/src/cmd/connection_logins.rs b/crates/engine/src/cmd/connection_logins.rs index 71d96aa63..3b778c924 100644 --- a/crates/engine/src/cmd/connection_logins.rs +++ b/crates/engine/src/cmd/connection_logins.rs @@ -23,7 +23,7 @@ pub(crate) fn restore_active(s: &mut Session) { fn groups(s: &Session) -> std::collections::BTreeMap<(&'static str, String), Vec> { let mut groups = std::collections::BTreeMap::<_, Vec<_>>::new(); for l in s.project.maps.iter().flat_map(|m| &m.layers) { - let Some(src) = l.source.as_ref().filter(|src| !src.is_open()) else { continue }; + let Some(src) = l.source.as_ref().filter(|src| !src.is_open() && !src.reconnect_required()) else { continue }; let Some((_, db)) = layer_database(src) else { continue }; let Some(key) = database_key(&crate::sources::redact(db)) else { continue }; groups.entry(key).or_default().push(l.id); @@ -114,7 +114,7 @@ fn resolve(s: &mut Session, only: Option<&[(&'static str, String)]>) { match super::credential(s, &c) { Ok(Some((pw, saved))) => { super::activate(s, &c, pw.expose()); - super::reopen(s, &c, saved, !saved, true); + super::reopen(s, &c, saved, !saved, super::ReopenMode::Automatic); } Ok(None) => { duckdb::set_connection_password(&key, &c.id, None); diff --git a/crates/engine/src/cmd/connections.rs b/crates/engine/src/cmd/connections.rs index 7bb5fb543..ca00b4f04 100644 --- a/crates/engine/src/cmd/connections.rs +++ b/crates/engine/src/cmd/connections.rs @@ -25,34 +25,34 @@ mod connection_use; pub fn commands() -> Vec { vec![ - C::new("connection.list", "List Connections", "Home › Add › Data Sources", list).view().params( + C::new("connection.list", "List Connections", "Home › Add › Data Sources", list).roles(&[]).view().params( r#"{"kind"?: "postgres"|"mysql"|"sqlserver"|"service"|"wms"|"wmts"|"wfs"|"wcs"|"ogcapi"|"arcgis"|"csw"|"stac"|"s3", "group"?: text}"#, ), - C::new("connection.new", "New Connection", "Home › Add › Data Sources", new).view().params( + C::new("connection.new", "New Connection", "Home › Add › Data Sources", new).roles(&[]).view().params( r#"{"kind": "postgres"|"mysql"|"sqlserver"|"service"|"wms"|"wmts"|"wfs"|"wcs"|"ogcapi"|"arcgis"|"csw"|"stac"|"s3", "address"?: "as the kind's add command takes it, without its password or token (postgres:host=… dbname=… user=…, mysql://user@host/db, mssql://user@host/db, a service's or STAC API's https:// address, an S3 endpoint: AWS when absent)", "name"?: "unique (default from the address)", "group"?: text, "options"?: {"region"?: text, "buckets"?: "a,b", "credentials"?: "keys"|"profile"|"environment"|"none", "profile"?: text} (S3 only), "auth"?: "none"|"basic"|"header": how a web-service connection logs in (basic authentication or an API key in a header; the password or key is given to connection.login, never kept here), "user"?: "the basic login's user name", "keyName"?: "the header the API key goes in, e.g. X-API-Key"}"#, ), - C::new("connection.edit", "Edit Connection", "Home › Add › Data Sources", edit).view().params( + C::new("connection.edit", "Edit Connection", "Home › Add › Data Sources", edit).roles(&[]).view().params( r#"{"id": "a connection's id or name", "name"?: text, "group"?: text, "address"?: text, "options"?: "as connection.new (replaces them)", "auth"?: "as connection.new", "user"?: text, "keyName"?: text, "kind"?: "its kind (unchanged)"} (another origin, login method, user or key name ends the web connection's session login)"#, ), - C::new("connection.duplicate", "Duplicate Connection", "Home › Add › Data Sources", duplicate).view().params(r#"{"id": "a connection's id or name", "name"?: text}"#), - C::new("connection.remove", "Remove Connection", "Home › Add › Data Sources", remove).view().params(r#"{"id": "a connection's id or name"} (deletes its saved database password or S3 keys; ends its web-service session login)"#), - C::new("connection.login", "Log In to Connection", "Home › Add › Data Sources", login_command).view().params( + C::new("connection.duplicate", "Duplicate Connection", "Home › Add › Data Sources", duplicate).roles(&[]).view().params(r#"{"id": "a connection's id or name", "name"?: text}"#), + C::new("connection.remove", "Remove Connection", "Home › Add › Data Sources", remove).roles(&[]).view().params(r#"{"id": "a connection's id or name"} (deletes its saved database password or S3 keys; ends its web-service session login)"#), + C::new("connection.login", "Log In to Connection", "Home › Add › Data Sources", login_command).roles(&[]).view().params( r#"{"id": "a connection's id or name; blank credentials explicitly use this ID's session credentials (or saved credentials for database/S3 connections)", "password"?: "a database password, or a web service's basic-authentication password", "apiKey"?: "a web service's API key (auth header)", "allowInsecure"?: "boolean, default false: confirms sending a web service's login to its plain http:// address, unencrypted", "accessKeyId"?: text, "secretAccessKey"?: text, "sessionToken"?: "temporary S3 credentials, never persisted", "remember"?: "boolean, default false: opt into this computer's credential store for a newly typed database password or complete S3 key pair (saved keys mode only). Web-service credentials are session-only and reject remember:true. No credentials are stored in connections or projects; a failed database/S3 save preserves the session login"} (a web service's login is sent only with this session's requests bound to that connection and only to its address's origin: the same scheme, host and port; logging in to one of several connections chooses it for unassigned requests to that address)"#, ), - C::new("connection.logout", "Log Out of Connection", "Home › Add › Data Sources", logout).view().params( + C::new("connection.logout", "Log Out of Connection", "Home › Add › Data Sources", logout).roles(&[]).view().params( r#"{"id": "a web-service connection's id or name"} (clears this session's login and its password or key, which is no longer sent)"#, ), - C::new("connection.forgetSavedKeys", "Forget Saved Keys", "Home › Add › Data Sources", forget_saved_keys).view().params( + C::new("connection.forgetSavedKeys", "Forget Saved Keys", "Home › Add › Data Sources", forget_saved_keys).roles(&[]).view().params( r#"{"id": "a saved S3 connection's id or name"} (deletes stored keys; a login in this session keeps working)"#, ), - C::new("connection.resolveS3", "Choose S3 Connection", "Home › Add › Data Sources", resolve_s3).view().params( + C::new("connection.resolveS3", "Choose S3 Connection", "Home › Add › Data Sources", resolve_s3).roles(&[]).view().params( r#"{"source": "a pending project S3 source address", "id": "a candidate saved connection id"} → {id, reopened, failed, reading}: explicitly grants this bucket and opens only its waiting sources; reading counts the rasters read in place whose overview is still being read (the desktop doesn't wait for it: it lands as the map draws, and one that fails is said then)"#, ), - C::new("connection.forgetPassword", "Forget Saved Password", "Home › Add › Data Sources", forget_password).view().params(r#"{"id": "a database connection's id or name"}"#), - C::new("connection.test", "Test Connection", "Home › Add › Data Sources", test).view().params( + C::new("connection.forgetPassword", "Forget Saved Password", "Home › Add › Data Sources", forget_password).roles(&[]).view().params(r#"{"id": "a database connection's id or name"}"#), + C::new("connection.test", "Test Connection", "Home › Add › Data Sources", test).roles(&[]).view().params( r#"{"id"?: "a connection's id or name", "kind"?: "postgres"|"mysql"|"sqlserver"|"service"|"wms"|"wmts"|"wfs"|"wcs"|"ogcapi"|"arcgis"|"csw"|"stac"|"s3", "name"?: text, "group"?: text, "address"?: text, "options"?: "an unsaved connection, as connection.new (when no id)", "auth"?: text, "user"?: text, "keyName"?: "as connection.new (when no id)", "password"?: "for a named database, authentication refusal restores prior session credentials; a later listing failure keeps the new password; never persisted. A saved web service's: as connection.login (logs in first)", "apiKey"?: "a saved web service's, as connection.login (logs in first)", "allowInsecure"?: "as connection.login", "accessKeyId"?: text, "secretAccessKey"?: text, "sessionToken"?: "as connection.login (logs in first)", "probe"?: "true: only the address the test reads, {fetch} (null for databases, which their driver tests)", "answer"?: "non-database kinds only: what that address returned, when the app fetched it (no network request then); databases ignore answer and test through their driver"}"#, ), - C::new("connection.browse", "Browse Connection", "Home › Add › Data Sources", browse::browse).view().params( + C::new("connection.browse", "Browse Connection", "Home › Add › Data Sources", browse::browse).roles(&[]).view().params( r#"{"id": "a connection's id or name", "node"?: "a child's node, as a listing gives it (the connection's top level when absent)", "probe"?: "true: only the address this level reads, {fetch} (null: read without a fetch, or by the database driver); an S3 connection logs in first, as connection.test does", "answer"?: "what that address returned, when the app fetched it", "documents"?: [{"url": "a further document a listing asked for (next)", "text": "its answer"}]} → {connection, node, children: [{key, name, what, detail?, node? (browse it), children? (listed already), add? {command, params}: the command that adds it}], truncated?, matched?, notes?, service? {url, type}, next? (fetch it, then ask again with answer and documents)}. Database roots list schemas; open a schema node to list its tables."#, ), ] @@ -430,12 +430,26 @@ struct Reopened { } #[cfg(not(target_arch = "wasm32"))] -fn reopen(s: &mut Session, c: &Connection, saved: bool, session: bool, all_unopened: bool) -> Reopened { +#[derive(Clone, Copy)] +enum ReopenMode { + Automatic, + Login { all_unopened: bool }, +} + +/// Reopens this connection's database sources and aggregates refusals. Automatic resolution +/// preserves peer reconnect barriers; only an explicit matching login releases them. +#[cfg(not(target_arch = "wasm32"))] +fn reopen(s: &mut Session, c: &Connection, saved: bool, session: bool, mode: ReopenMode) -> Reopened { let Some(key) = crate::connections::database_key(&c.address) else { return Reopened::default() }; let mut result = Reopened::default(); let mut first_error = None; for layer in s.project.maps.iter_mut().flat_map(|m| m.layers.iter_mut()) { - let Some(src) = layer.source.as_mut().filter(|src| src.login || all_unopened && !src.is_open()) else { continue }; + let Some(src) = layer.source.as_mut().filter(|src| match mode { + ReopenMode::Automatic => !src.reconnect_required() && !src.is_open(), + ReopenMode::Login { all_unopened } => src.login || src.reconnect_required() || all_unopened && !src.is_open(), + }) else { + continue; + }; let Some((kind, database)) = crate::connections::layer_database(src) else { continue }; if kind != key.0 || crate::connections::database_key(&crate::sources::redact(database)).as_ref() != Some(&key) { continue; @@ -448,6 +462,9 @@ fn reopen(s: &mut Session, c: &Connection, saved: bool, session: bool, all_unope result.failed += 1; continue; } + if matches!(mode, ReopenMode::Login { .. }) { + src.allow_reconnect(); + } src.error = None; match src.open() { Ok(()) => result.opened += 1, @@ -606,7 +623,7 @@ fn login(s: &mut Session, c: &Connection, p: &Value) -> Result &key.1, &c.id, pw, - || reopen(s, c, false, false, all_unopened), + || reopen(s, c, false, false, ReopenMode::Login { all_unopened }), |result| result.refused, ) } else { @@ -619,7 +636,7 @@ fn login(s: &mut Session, c: &Connection, p: &Value) -> Result activate(s, c, password.expose()); } let session = password.is_some() && !saved; - reopen(s, c, saved, session, all_unopened) + reopen(s, c, saved, session, ReopenMode::Login { all_unopened }) }; if let Some(pw) = &typed && !result.refused diff --git a/crates/engine/src/cmd/cube.rs b/crates/engine/src/cmd/cube.rs index ee23d189e..4c80027c5 100644 --- a/crates/engine/src/cmd/cube.rs +++ b/crates/engine/src/cmd/cube.rs @@ -25,10 +25,10 @@ use super::ps; use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { - vec![C::new("cube.trend", "Trend Analysis", "Analyze › Image Analysis", trend_cmd).params( + vec![C::new("cube.trend", "Trend Analysis", "Analyze › Image Analysis", trend_cmd).roles(&[]).params( r#"{"cube"?: an entry of cube.catalog (its STAC API, collection, band, cell size and cloud cap, or its Zarr store and variable) | "layers"?: [raster layers, a date each; one on another grid is resampled (nearest) onto the first's] | "layer"?: a Zarr layer (or "path" | "url", "variable", "index", "crs" as for zarr.reduce) or a raster of several bands, a date each (default: the active layer) | "collection": a STAC collection (desktop; read in place over an area, in pieces when large; with url, asset, bbox, maxCloud, cellSize and resampling below, and "start", "end"), "url"?: the collection's STAC API (as for stac.addCube), "asset"?: the collection's band, such as NDVI, "bbox"?: [w, s, e, n] | "view" (the collection's area), "maxCloud"?: a collection's cloud cover cap, "cellSize"?: a collection's cell size in the files' units (the overview level nearest it is read), "resampling"?: "nearest" (default: that overview's pixels as stored) | "average" (each cell the mean of the full images' pixels in it, which reads them), "band"?: 1-based (a stack's band), "dates"?: [dates] (a stack's or the bands' dates, in their order; default: read from the layers' names or files, or the bands' names), "start"?, "end"?: dates (a whole number from 1000 to 9999 is a year; an end given as a year or a month takes all of it), "months"?: [1–12] (one season across years), "test"?: "hamedRao" (default) | "yueWang" | "seasonal" | "original", "seasons"?: parts of a year for the seasonal test (default 12), "alpha"?: 0.05 (above 0, below 0.5), "minValid"?: dates a cell needs (default: half of them, at least 4; the seasonal test also needs 4 in seasons of two or more), "per"?: "year" (default) | "decade", "name"?: text, "ran"?: (internal: a trend the app has worked out)} → four raster layers: Sen's slope, the test's Z and p-value, and the slope where p < alpha"#, ), - C::new("cube.reduce", "Summarise a Data Cube over Time", "Analyze › Image Analysis", reduce_cmd).params( + C::new("cube.reduce", "Summarise a Data Cube over Time", "Analyze › Image Analysis", reduce_cmd).roles(&[]).params( r#"{"cube"?: an entry of cube.catalog (as for cube.trend) | "layers"?: [raster layers, a date each; one on another grid is resampled (nearest) onto the first's] | "layer"?: a Zarr layer or a raster of several bands, a date each (default: the active layer) | "path"?: a Zarr store's folder | "url"?: a Zarr store's address, or a STAC collection's API (as for stac.addCube) | "collection": a STAC collection (desktop; read in place over an area, in pieces when large; with url, asset, bbox, maxCloud, cellSize and resampling below, and "start", "end"), "variable"?, "index"?, "crs"?: a Zarr variable's (as for zarr.reduce), "asset"?: the collection's band, such as NDVI, "bbox"?: [w, s, e, n] | "view" (the collection's area), "maxCloud"?: a collection's cloud cover cap, "cellSize"?: a collection's cell size in the files' units (the overview level nearest it is read), "resampling"?: "nearest" (default) | "average" (as for cube.trend), "band"?: 1-based (a stack's band), "dates"?: [dates] (a stack's or the bands' dates, in their order; default: read from the layers' names or files, or the bands' names), "stat": "mean"|"min"|"max"|"sum"|"std" (the population standard deviation, n)|"count"|"countAbove"|"mode" or "majority" (a categorical cube's most frequent class; the least of those tied)|"modeCount" (how many dates had the mode)|"variety" (how many classes)|"changes" (how many times the class changed from one valued date to the next)|"longestRun" (the class of the longest run of dates in one class; the least of those tied)|"longestRunLength" (how many dates that run lasted)|"frequency" (a layer for each class met: its count of dates)|"countIn" (how many dates in the classes given), "above"?: number (countAbove's threshold), "classes"?: [whole numbers 0–255, at most 256] (countIn's classes), "by"?: "all" (default: one layer) | "year" (a layer for each calendar year) | "month" (a layer for each month of the year, across years), "start"?, "end"?: dates (as for cube.trend), "months"?: [1–12] (a season across years), "minValid"?: 0–1 (a cell with a smaller share of its group's dates is left empty; default 0), "name"?: text} → raster layers of the statistic, one for each group (frequency: one for each class of each group)"#, )] } diff --git a/crates/engine/src/cmd/cube_catalog.rs b/crates/engine/src/cmd/cube_catalog.rs index 98fe7a617..a32a20dfb 100644 --- a/crates/engine/src/cmd/cube_catalog.rs +++ b/crates/engine/src/cmd/cube_catalog.rs @@ -14,10 +14,10 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("cube.catalog", "Data Cube Catalog", "Home › Add › Web Data", catalog).view().params( + C::new("cube.catalog", "Data Cube Catalog", "Home › Add › Web Data", catalog).roles(&[]).view().params( r#"{"theme"?: "vegetation"|"land cover"|"water"|"climate"|"ocean"|"atmosphere"|"elevation", "bbox"?: [west, south, east, north] | "view" (entries with data there; default: anywhere), "query"?: "words, all of them in an entry's name, provider or id"} → the data cubes ready to open (cube.open), each with its provider, themes, area, period, step, resolution, source (a STAC collection and band, or a Zarr variable) and licence"#, ), - C::new("cube.open", "Open a Data Cube", "Home › Add › Web Data", open).params( + C::new("cube.open", "Open a Data Cube", "Home › Add › Web Data", open).roles(&[]).params( r#"{"id": an entry of cube.catalog, "bbox"?: [west, south, east, north] | "view" (a STAC entry's area, default the map's extent; a Zarr entry's variable is read whole, but an area given must meet the entry's), "start"?, "end"?: dates, both or neither (default: the entry's last full calendar year), "maxCloud"?: percent (a STAC entry's cloud cover cap; default the entry's), "range"?: [low, high] (a STAC entry's values shown from black to white, in the band's units; default the entry's), "name"?: text} → the cube on the map, following the time slider, as stac.addCube or layer.addZarr adds it (desktop)"#, ), ] diff --git a/crates/engine/src/cmd/deeplearning.rs b/crates/engine/src/cmd/deeplearning.rs index ee1e94cd7..0c5470b25 100644 --- a/crates/engine/src/cmd/deeplearning.rs +++ b/crates/engine/src/cmd/deeplearning.rs @@ -52,10 +52,10 @@ pub fn commands() -> Vec { let mut r = json!({"model": m.name, "accuracy": m.accuracy, "classes": m.model.classes, "epochs": m.epochs}); r["replaced"] = json!(store(s, m)); Ok(r) - }) + }).roles(&[]) .when(need_raster) .params(r#"{"layer": raster, "training": polygon layer, "classField": field naming each polygon's class, "bands"?: [1-based], "hidden"?: [units per hidden layer] (default [32, 16]), "epochs"?: 1..500 (default 60), "model"?: name}"#), - C::new("dl.classify", "Deep Learning Tools", "Analyze › Image Classification", classify).when(need_raster).params( + C::new("dl.classify", "Deep Learning Tools", "Analyze › Image Classification", classify).roles(&[]).when(need_raster).params( r#"{"layer": raster, "model"?: a trained model's name, "training"?: polygon layer, "classField"?: field, "bands"?: [1-based], "hidden"?: [units], "epochs"?: 1..500 (without a model: train one first, as dl.train), "name"?: output, "ran"?: (internal: classes the app has worked out)}"#, ), C::new("dl.package", "Deep Learning Package", "Publish › Package", |s, p| { @@ -63,7 +63,7 @@ pub fn commands() -> Vec { let bytes = package(s, p)?; rustgis_data::io::write_atomic(Path::new(path), &bytes).map_err(|e| format!("{path}: {e}"))?; Ok(json!({"path": path, "bytes": bytes.len()})) - }) + }).roles(&[]) .view() .when(need_model) .params(r#"{"model"?: name (default: the last trained), "path": "*.rgisdlpk"}"#), @@ -91,16 +91,16 @@ pub fn commands() -> Vec { let name = m.name.clone(); store(s, m); Ok(json!({"model": name})) - }) + }).roles(&[]) .params(r#"{"path"?: "*.rgisdlpk", "data"?: base64}"#), C::new("dl.remove", "Delete Model", "Analyze › Image Classification", |s, p| { let i = model_index(s, p)?; s.project.dl_models.remove(i); Ok(Value::Null) - }) + }).roles(&[]) .when(need_model) .params(r#"{"model": name}"#), - C::new("imagery.hostedImagery", "Create Hosted Imagery", "Analyze › Imagery Sharing", hosted).view().when(need_raster).params( + C::new("imagery.hostedImagery", "Create Hosted Imagery", "Analyze › Imagery Sharing", hosted).roles(&[]).view().when(need_raster).params( r#"{"layer": raster, "path": "*.zip" (z/x/y.png tiles, tiles.json and a viewer page), "minZoom"?: 0..22, "maxZoom"?: 0..22 (default: the raster's resolution)}"#, ), ] diff --git a/crates/engine/src/cmd/design.rs b/crates/engine/src/cmd/design.rs index fd13fa6fe..1fdd96cd7 100644 --- a/crates/engine/src/cmd/design.rs +++ b/crates/engine/src/cmd/design.rs @@ -238,16 +238,16 @@ pub(crate) fn apply_with(s: &mut Session, id: u32, changes: Vec, new_def pub fn commands() -> Vec { vec![ // Data design. - C::new("data.domain", "Domains", "Data › Data Design", domain).when(need_feature_layer).params( + C::new("data.domain", "Domains", "Data › Data Design", domain).roles(&[]).when(need_feature_layer).params( r#"{"layer"?, "field": string, "coded"?: [[code, description], …], "min"?: number, "max"?: number, "clear"?: bool, "force"?: bool (set it even though existing values fall outside)}"#, ), - C::new("data.subtypes", "Subtypes", "Data › Data Design", subtypes).when(need_feature_layer).params( + C::new("data.subtypes", "Subtypes", "Data › Data Design", subtypes).roles(&[]).when(need_feature_layer).params( r#"{"layer"?, "field": integer field, "subtypes": [{"code": n, "name": string, "defaults"?: {field: value}}], "symbolize"?: bool, "clear"?: bool}"#, ), - C::new("data.attributeRule", "Attribute Rules", "Data › Data Design", attribute_rule).when(need_feature_layer).params( + C::new("data.attributeRule", "Attribute Rules", "Data › Data Design", attribute_rule).roles(&[]).when(need_feature_layer).params( r#"{"layer"?, "name": string, "kind": "calculation"|"constraint", "field"?: field set by a calculation, "expression": calculator expression or SQL where clause, "remove"?: bool}"#, ), - C::new("data.contingentValues", "Contingent Values", "Data › Data Design", contingent).when(need_feature_layer).params( + C::new("data.contingentValues", "Contingent Values", "Data › Data Design", contingent).roles(&[]).when(need_feature_layer).params( r#"{"layer"?, "name": string, "fields": [field, …], "combos": [[value, …], …], "remove"?: bool}"#, ), C::new("data.archive", "Add Archive", "Data › Archiving", |s, p| { @@ -256,7 +256,7 @@ pub fn commands() -> Vec { let l = s.map_mut().layer_mut(id).ok_or("no such layer")?; l.design.archive = if on { Some(l.design.archive.take().unwrap_or_default()) } else { None }; Ok(json!({"archiving": on})) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "on"?: bool}"#), C::new("data.archiveHistory", "Archive History", "Data › Archiving", |s, p| { @@ -270,23 +270,23 @@ pub fn commands() -> Vec { .map(|e| json!({"time": e.time, "feature": e.id, "action": match (&e.before, &e.after) { (None, _) => "insert", (_, None) => "delete", _ => "update" }})) .collect(); Ok(json!({"edits": a.len(), "recent": rows})) - }).params(r#"{"layer"?: name|id (default: the active layer)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id (default: the active layer)}"#) .view() .when(need_feature_layer), - C::new("data.archiveRestore", "Restore From Archive", "Data › Archiving", archive_restore) + C::new("data.archiveRestore", "Restore From Archive", "Data › Archiving", archive_restore).roles(&[]) .when(need_editable) .params(r#"{"layer"?, "time": seconds since 1970 (the layer as it was then)}"#), - C::new("data.relate", "Related Data", "Data › Relationship", relate).params( + C::new("data.relate", "Related Data", "Data › Relationship", relate).roles(&[]).params( r#"{"name": string, "origin": layer, "originKey": field, "destination": layer, "destinationKey": field, "remove"?: bool}"#, ), - C::new("data.related", "Select Related", "Data › Relationship", related) + C::new("data.related", "Select Related", "Data › Relationship", related).roles(&[]) .params(r#"{"name"?: relationship (all when omitted), "select"?: bool (select the related rows; default true)}"#), // Manage edits. C::new("edit.enable", "Edit", "Edit › Manage Edits", |s, p| { let set = &mut s.map_mut().editing.settings; set.enabled = b(p, "on").unwrap_or(!set.enabled); Ok(json!({"enabled": set.enabled})) - }) + }).roles(&[]) .params(r#"{"on"?: bool}"#), C::new("edit.settings", "Editing Settings", "Edit › Manage Edits", |s, p| { let set = &mut s.map_mut().editing.settings; @@ -294,7 +294,7 @@ pub fn commands() -> Vec { set.show_vertices = v; } Ok(json!({"enabled": set.enabled, "showVertices": set.show_vertices})) - }) + }).roles(&[]) .params(r#"{"showVertices"?: bool}"#), C::new("edit.status", "Edit Status", "Edit › Manage Edits", |s, _| { let rows: Vec = s @@ -303,9 +303,9 @@ pub fn commands() -> Vec { .filter_map(|(id, c)| s.map().layer(*id).map(|l| json!({"layer": l.name, "added": c.added, "modified": c.modified, "deleted": c.deleted}))) .collect(); Ok(json!({"unsaved": s.dirty(), "layers": rows})) - }) + }).roles(&[]) .view(), - C::new("edit.template", "Manage Templates", "Edit › Manage Edits", template).when(need_feature_layer).params( + C::new("edit.template", "Manage Templates", "Edit › Manage Edits", template).roles(&[]).when(need_feature_layer).params( r#"{"layer"?, "name": string, "defaults"?: {field: value}, "description"?: string, "remove"?: bool}"#, ), // Snapping. @@ -321,9 +321,9 @@ pub fn commands() -> Vec { } } Ok(json!({"enabled": sn.enabled, "tolerancePx": sn.tolerance_px, "vertex": sn.vertex, "edge": sn.edge, "end": sn.end})) - }) + }).roles(&[]) .params(r#"{"on"?: bool, "tolerancePx"?: number, "vertex"?: bool, "edge"?: bool, "end"?: bool}"#), - C::new("edit.snap", "Snap Point", "Edit › Snapping", snap).view().params(r#"{"lon", "lat"}"#), + C::new("edit.snap", "Snap Point", "Edit › Snapping", snap).roles(&[]).view().params(r#"{"lon", "lat"}"#), // Modify features. C::new("edit.move", "Move", "Edit › Features › Modify", |s, p| { let unit = distance_unit_metres(p)?; @@ -335,7 +335,7 @@ pub fn commands() -> Vec { return Err("dx and dy must be finite distances in the command's units (limit: 40,000 km per axis after ground-to-grid)".into()); } modify(s, |g| Some(g.map_coords(&|c: Coord| shift(c, dx, dy)))) - }) + }).roles(&[]) .when(need_editable_selection) .params(r#"{"dx": distance east, "dy": distance north, "units"?: "Meters" (default)|"Feet"|"Kilometers"|"Miles"}"#), C::new("edit.rotate", "Rotate", "Edit › Features › Modify", |s, p| { @@ -355,7 +355,7 @@ pub fn commands() -> Vec { merc_to_lonlat(Coord::new(c.x + x * cos - y * sin, c.y + x * sin + y * cos)) })) }) - }) + }).roles(&[]) .when(need_editable_selection) .params(r#"{"angle": degrees clockwise}"#), C::new("edit.scale", "Scale", "Edit › Features › Modify", |s, p| { @@ -371,7 +371,7 @@ pub fn commands() -> Vec { merc_to_lonlat(Coord::new(c.x + (m.x - c.x) * k, c.y + (m.y - c.y) * k)) })) }) - }) + }).roles(&[]) .when(need_editable_selection) .params(r#"{"factor": number}"#), C::new("edit.flip", "Flip Line", "Edit › Features › Modify", |s, _| { @@ -379,36 +379,36 @@ pub fn commands() -> Vec { Geometry::Line(parts) => Some(Geometry::Line(parts.iter().rev().map(|p| p.iter().rev().copied().collect()).collect())), _ => None, }) - }) + }).roles(&[]) .when(need_editable_selection), - C::new("edit.setGeometry", "Set Geometry", "Edit › Features › Modify", set_geometry) + C::new("edit.setGeometry", "Set Geometry", "Edit › Features › Modify", set_geometry).roles(&[]) .when(need_editable) .params(r#"{"layer"?, "row": number, "geometry": GeoJSON geometry}"#), - C::new("edit.vertex", "Edit Vertex", "Edit › Features › Modify", edit_vertex).when(need_editable).params( + C::new("edit.vertex", "Edit Vertex", "Edit › Features › Modify", edit_vertex).roles(&[]).when(need_editable).params( r#"{"layer"?, "row": number, "action": "move"|"insert"|"delete", "part"?: number, "ring"?: number, "index": number (insert: the vertex the new one follows), "lon"?, "lat"?}"#, ), - C::new("edit.merge", "Merge", "Edit › Features › Modify", merge).when(need_editable_selection), - C::new("edit.moveSharedVertex", "Move Shared Vertex", "Edit › Features › Modify", shared_vertex::run).when(need_editable).params( + C::new("edit.merge", "Merge", "Edit › Features › Modify", merge).roles(&[]).when(need_editable_selection), + C::new("edit.moveSharedVertex", "Move Shared Vertex", "Edit › Features › Modify", shared_vertex::run).roles(&[]).when(need_editable).params( r#"{"layer"?, "row": number, "part"?: number, "ring"?: number, "index": number, "lon": degrees, "lat": degrees}"#, ), - C::new("edit.explode", "Explode", "Edit › Features › Modify", explode).when(need_editable_selection), - C::new("edit.split", "Split", "Edit › Features › Modify", split).when(need_editable_selection).params(r#"{"lon", "lat"}"#), - C::new("edit.cutPolygon", "Cut Polygon", "Edit › Features › Modify", sketch::cut_polygon) + C::new("edit.explode", "Explode", "Edit › Features › Modify", explode).roles(&[]).when(need_editable_selection), + C::new("edit.split", "Split", "Edit › Features › Modify", split).roles(&[]).when(need_editable_selection).params(r#"{"lon", "lat"}"#), + C::new("edit.cutPolygon", "Cut Polygon", "Edit › Features › Modify", sketch::cut_polygon).roles(&[]) .when(need_editable_selection) .params(r#"{"line": GeoJSON LineString (lon/lat) drawn across the selected polygons}"#), - C::new("edit.reshape", "Reshape", "Edit › Features › Modify", sketch::reshape) + C::new("edit.reshape", "Reshape", "Edit › Features › Modify", sketch::reshape).roles(&[]) .when(need_editable_selection) .params(r#"{"line": GeoJSON LineString (lon/lat) drawn across the selected line or polygon's boundary}"#), - C::new("edit.trace", "Trace Edge Path", "Edit › Features", sketch::trace).view().params( + C::new("edit.trace", "Trace Edge Path", "Edit › Features", sketch::trace).roles(&[]).view().params( r#"{"from": [lon, lat], "to": [lon, lat], "layer"?, "row"?: number (else the nearest edge within the tolerance), "tolerancePx"?: number (default: the snapping tolerance)}"#, ), - C::new("edit.traverse", "Traverse", "Edit › Features › Modify", traverse).when(need_editable).params( + C::new("edit.traverse", "Traverse", "Edit › Features › Modify", traverse).roles(&[]).when(need_editable).params( r#"{"layer"?, "start": [lon, lat], "units"?: "Meters" (default)|"Feet"|"Kilometers"|"Miles", "legs": [{"distance": number, "bearing": degrees from north}]} (closed in a polygon layer, open in a line layer)"#, ), C::new("edit.tools", "Editor Tools", "Edit › Tools", |s, _| { let tools: Vec = s.registry.commands.iter().filter(|c| c.location.starts_with("Edit › Features")).map(|c| json!({"id": c.id, "label": c.label, "params": c.params})).collect(); Ok(Json::Array(tools)) - }) + }).roles(&[]) .view(), // Elevation and corrections. C::new("edit.elevation", "Elevation", "Edit › Elevation", |s, p| { @@ -431,7 +431,7 @@ pub fn commands() -> Vec { }; s.map_mut().editing.settings.elevation = e; Ok(Json::Null) - }) + }).roles(&[]) .params(r#"{"mode": "none"|"constant"|"surface", "z"?: number, "units"?: "Meters" (default)|"Feet"|"Kilometers"|"Miles", "surface"?: raster layer}"#), C::new("edit.groundToGrid", "Ground To Grid", "Edit › Corrections", |s, p| { let on = b(p, "on").unwrap_or(true); @@ -442,13 +442,13 @@ pub fn commands() -> Vec { } s.map_mut().editing.settings.ground_to_grid = on.then_some((k, r.clamp(-360.0, 360.0))); Ok(json!({"on": on})) - }) + }).roles(&[]) .params(r#"{"on"?: bool, "scale"?: factor, "rotation"?: degrees}"#), // Topology. - C::new("topology.rules", "Topology Options", "Edit › Manage Edits", topology_rules).params( + C::new("topology.rules", "Topology Options", "Edit › Manage Edits", topology_rules).roles(&[]).params( r#"{"add"?: {"rule": "mustNotOverlap"|"mustNotHaveGaps"|"mustNotSelfIntersect"|"mustNotHaveDangles"|"mustBeCoveredBy", "layer": layer, "by"?: polygon layer}, "remove"?: index, "clear"?: bool}"#, ), - C::new("topology.validate", "Error Inspector", "Edit › Manage Edits", topology_validate).view().params(r#"{"toleranceM"?: metres (dangles)}"#), + C::new("topology.validate", "Error Inspector", "Edit › Manage Edits", topology_validate).roles(&[]).view().params(r#"{"toleranceM"?: metres (dangles)}"#), ] } diff --git a/crates/engine/src/cmd/diagram.rs b/crates/engine/src/cmd/diagram.rs index ba2c6d1a8..5311b377f 100644 --- a/crates/engine/src/cmd/diagram.rs +++ b/crates/engine/src/cmd/diagram.rs @@ -19,15 +19,15 @@ enum Asked { pub fn commands() -> Vec { vec![ - C::new("layer.diagram", "Set Diagram", "Style › Drawing", set) + C::new("layer.diagram", "Set Diagram", "Style › Drawing", set).roles(&[]) .when(need_feature_layer) .params( r##"{"layer"?, "kind": "pie"|"bar", "fields": [numeric field names, or {"field": name, "color"?: "#RRGGBB"|"#RRGGBBAA"}] (a slice or bar each, in order; colours default to the palette new layers take), "size"?: points (fixed: a pie's radius or a bar chart's height, the same for every feature) | {"from"?: points (default 6), "to"?: points (default 40)} (grown with the categories' sum, square-root scaled as Proportional symbols are; the default), "startAngle"?: degrees clockwise from north (a pie's first slice; default 0), "labels"?: "none"|"value"|"percent"|"both" (what each slice or bar says beside it: its value in the legend's number format, its share of the total in percent, or both; default "none"), "background"?: "#RRGGBB"|"#RRGGBBAA"|null (the disc or box behind the chart; default none), "outline"?: {"color"?: "#RRGGBB"|"#RRGGBBAA" (default #4D4D4D), "width"?: points (default 0.75)}} (drawn on each feature over its symbol)"##, ), - C::new("layer.clearDiagram", "Clear Diagram", "Style › Drawing", |s, p| with(s, p, |l| l.diagram = None)) + C::new("layer.clearDiagram", "Clear Diagram", "Style › Drawing", |s, p| with(s, p, |l| l.diagram = None)).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?: name|id (removes the layer's diagram; the renderer is untouched)}"#), - C::new("layer.diagramInfo", "Diagram Settings", "Style › Drawing", info) + C::new("layer.diagramInfo", "Diagram Settings", "Style › Drawing", info).roles(&[]) .view() .when(need_feature_layer) .params(r#"{"layer"?: name|id} → {"kind", "fields": [{"field", "color"}], "size": points (fixed) | {"from", "to", "min", "max"} (scaled; min and max are the range of sums it was measured over), "startAngle"?, "labels", "background"?, "outline"} or null without one (layer.diagram takes it back as it is)"#), diff --git a/crates/engine/src/cmd/directions.rs b/crates/engine/src/cmd/directions.rs index ff27c8fed..b7b2c3556 100644 --- a/crates/engine/src/cmd/directions.rs +++ b/crates/engine/src/cmd/directions.rs @@ -11,7 +11,7 @@ use crate::render::rustgis_tiles::routing::{self, Avoid, Mode, Route, Router}; use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { - vec![C::new("map.route", "Directions", "Home › Navigate", route).params( + vec![C::new("map.route", "Directions", "Home › Navigate", route).roles(&[]).params( r#"{"from": [lon, lat] | "a place, an address or 'lat, lon'", "to": the same, "via"?: [places] (stops on the way), "mode"?: "drive" (default) | "bike" | "walk", "avoid"?: ["tolls", "highways", "ferries"] (driving), "imperial"?: bool (miles in the instructions), "language"?: "en-US" | "de" | …, "url"?: "a Valhalla server of your own", "addLayer"?: bool (adds the route and its steps as layers, default true), "zoom"?: bool (default true), "answer"?: "a routing answer already fetched (the app's; nothing is fetched)", "route"?: index of the route in answer}"#, )] } diff --git a/crates/engine/src/cmd/earthdata.rs b/crates/engine/src/cmd/earthdata.rs index b9f6e67ce..bab3e474e 100644 --- a/crates/engine/src/cmd/earthdata.rs +++ b/crates/engine/src/cmd/earthdata.rs @@ -12,16 +12,16 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("earthdata.collections", "Search NASA Earthdata Collections", "Home › Add › Web Data › NASA Earthaccess", collections) + C::new("earthdata.collections", "Search NASA Earthdata Collections", "Home › Add › Web Data › NASA Earthaccess", collections).roles(&[]) .view() .params(r#"{"keyword"?: text (free text: "sea surface temperature"), "shortName"?: "a collection's short name (HLSL30, ATL08, GEDI02_A…)", "cloudHosted"?: bool (only collections in NASA's cloud), "limit"?: 1–100 (20), "page"?: number (1), "answer"?: "CMR's JSON answer, when already fetched (the app's background request)"} — keyword or shortName is needed; searching needs no Earthdata Login"#), - C::new("earthdata.granules", "Search NASA Earthdata Granules", "Home › Add › Web Data › NASA Earthaccess", granules).params( + C::new("earthdata.granules", "Search NASA Earthdata Granules", "Home › Add › Web Data › NASA Earthaccess", granules).roles(&[]).params( r#"{"collection"?: "a collection's concept id (C…-PROVIDER, as earthdata.collections lists it)", "shortName"?: "or a short name: its cloud-hosted, newest version", "bbox"?: [west, south, east, north] | "view" (the map's extent, the default), "view"?: bool (false: everywhere), "start"?: "YYYY-MM-DD", "end"?: "YYYY-MM-DD", "limit"?: 1–100 (25), "page"?: number (1), "footprints"?: bool (adds the granules' footprints as a layer, default true), "name"?: text (the footprints layer's), "answer"?: "CMR's JSON answer, when already fetched"} — each granule lists its data files; add a GeoTIFF with layer.addUrl (files need an Earthdata Login token: earthdata.login)"#, ), - C::new("earthdata.login", "Earthdata Login Token", "Home › Add › Web Data › NASA Earthaccess", login).view().params( + C::new("earthdata.login", "Earthdata Login Token", "Home › Add › Web Data › NASA Earthaccess", login).roles(&[]).view().params( r#"{"token": "an Earthdata Login token (Earthdata Login › Generate Token)", "remember"?: bool (false: kept for the session only; true: also remembered on this computer, in the system's credential store, and read back when next needed; without it a token remembered before is deleted)} — never in the project; sent only to NASA's Earthdata data hosts (https://*.earthdatacloud.nasa.gov, *.earthdata.nasa.gov) when their files are read; an expired token is forgotten"#, ), - C::new("earthdata.logout", "Forget Earthdata Login Token", "Home › Add › Web Data › NASA Earthaccess", logout) + C::new("earthdata.logout", "Forget Earthdata Login Token", "Home › Add › Web Data › NASA Earthaccess", logout).roles(&[]) .view() .params("{} — forgets the token for the session and deletes the one remembered on this computer"), C::new("earthdata.status", "Earthdata Login Status", "Home › Add › Web Data › NASA Earthaccess", |_s, _p| { @@ -29,7 +29,7 @@ pub fn commands() -> Vec { earthdata::restore(); earthdata::forget_if_expired(); Ok(status()) - }) + }).roles(&[]) .view() .params("{} — whether a token is kept, its expiry, whether it is remembered on this computer, and whether an expired one was forgotten (give a new one with earthdata.login)"), ] diff --git a/crates/engine/src/cmd/edit.rs b/crates/engine/src/cmd/edit.rs index 24925571f..6554e1f44 100644 --- a/crates/engine/src/cmd/edit.rs +++ b/crates/engine/src/cmd/edit.rs @@ -8,23 +8,23 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("edit.calculateGeometry", "Calculate Geometry", "Attribute Table", crate::geometry_calc::run) + C::new("edit.calculateGeometry", "Calculate Geometry", "Attribute Table", crate::geometry_calc::run).roles(&[]) .when(super::design::need_editable) .params(r#"{"layer"?, "field": string, "create": bool, "overwrite": bool, "property": string, "unit": string, "epsg"?: number, "scope": "all"|"selected"}"#), - C::new("edit.addFeature", "Create Feature", "Edit › Features", add_feature) + C::new("edit.addFeature", "Create Feature", "Edit › Features", add_feature).roles(&[]) .when(super::design::need_editable) .params(r#"{"layer"?, "geometry": GeoJSON geometry (lon/lat), "attributes"?: {field: value}, "template"?: feature template}"#), - C::new("edit.delete", "Delete", "Edit › Features", delete).key("Delete").when(super::design::need_editable_selection), - C::new("edit.setValue", "Edit Attribute", "Attribute Table", set_value) + C::new("edit.delete", "Delete", "Edit › Features", delete).roles(&[]).key("Delete").when(super::design::need_editable_selection), + C::new("edit.setValue", "Edit Attribute", "Attribute Table", set_value).roles(&[]) .when(super::design::need_editable) .params(r#"{"layer"?, "row": number, "field": string, "value": any}"#), - C::new("edit.addField", "Add Field", "Data › Data Design › Fields", add_field) + C::new("edit.addField", "Add Field", "Data › Data Design › Fields", add_field).roles(&[]) .when(need_layer) .params(r#"{"layer"?, "name": string, "type": "long"|"double"|"text"|"boolean"}"#), - C::new("edit.deleteField", "Delete Field", "Data › Data Design › Fields", delete_field) + C::new("edit.deleteField", "Delete Field", "Data › Data Design › Fields", delete_field).roles(&[]) .when(need_layer) .params(r#"{"layer"?, "field": string}"#), - C::new("edit.calculateField", "Calculate Field", "Attribute Table", calculate_field) + C::new("edit.calculateField", "Calculate Field", "Attribute Table", calculate_field).roles(&[]) .when(super::design::need_editable) .params(r#"{"layer"?, "field": string, "expression": "an expression: fields, 'text', numbers, + - * / % ^ ||, functions (see the Expressions page)"}"#), ] diff --git a/crates/engine/src/cmd/effects.rs b/crates/engine/src/cmd/effects.rs index 64c4eea5c..7820f1a12 100644 --- a/crates/engine/src/cmd/effects.rs +++ b/crates/engine/src/cmd/effects.rs @@ -33,6 +33,7 @@ pub fn commands() -> Vec { let mode = Blend::from_code(m).ok_or_else(|| format!("unknown blend mode '{m}'"))?; with(s, p, |l| l.blend = mode) }) + .roles(&[]) .when(need_layer) .params(r#"{"layer"?, "mode": "normal"|"multiply"|"screen"|"overlay"|"darken"|"lighten"|"difference"}"#), C::new("layer.mask", "Masking", "Style › Drawing", |s, p| { @@ -55,6 +56,7 @@ pub fn commands() -> Vec { }; with(s, p, |l| l.mask = mask.clone()) }) + .roles(&[]) .when(need_layer) .params(r#"{"layer"?, "mask": polygon layer|null}"#), C::new("layer.swipe", "Swipe", "Style › Compare", |s, p| { @@ -70,6 +72,7 @@ pub fn commands() -> Vec { }; Ok(json!({"swipe": s.map().swipe.map(|(l, x)| json!({"layer": l, "position": x}))})) }) + .roles(&[]) .when(need_layer) .params(r#"{"layer"?, "position"?: 0..1, "off"?: bool}"#), C::new("raster.rotation", "Rotation Type", "Style › Rotation", |s, p| { @@ -80,9 +83,11 @@ pub fn commands() -> Vec { rl.style.rotation = a.rem_euclid(360.0); Ok(json!({"angle": rl.style.rotation})) }) + .roles(&[]) .when(need_raster) .params(r#"{"layer"?, "angle": degrees clockwise}"#), C::new("layer.displayFilters", "Display Filters", "Style › Drawing", display_filters) + .roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "filters": [{"name", "minScale"?, "maxScale"?, "where"}] (empty to clear)}"#), C::new("layer.aggregation", "Aggregation", "Style › Drawing", |s, p| { @@ -99,9 +104,10 @@ pub fn commands() -> Vec { }; with(s, p, |l| l.aggregation = agg) }) + .roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "shape": "hexagon"|"square"|"none", "size"?: points}"#), - C::new("layer.extrusion", "Layer Extrusion", "Style › Extrusion", extrusion).when(need_polygons).params( + C::new("layer.extrusion", "Layer Extrusion", "Style › Extrusion", extrusion).roles(&[]).when(need_polygons).params( r#"{"layer"?, "type": "none"|"height"|"field", "field"?: numeric field (metres), "height"?: metres, "factor"?: number}"#, ), C::new("layer.faces", "Faces", "Style › Faces", |s, p| { @@ -115,6 +121,7 @@ pub fn commands() -> Vec { } }) }) + .roles(&[]) .when(need_polygons) .params(r#"{"layer"?, "culling"?: bool, "lighting"?: bool}"#), C::new("layer.lightShadow", "Light and Shadow", "Style › Light and Shadow", |s, p| { @@ -128,6 +135,7 @@ pub fn commands() -> Vec { } }) }) + .roles(&[]) .when(need_polygons) .params(r#"{"layer"?, "lighting"?: bool, "shadows"?: bool}"#), ] diff --git a/crates/engine/src/cmd/embeddings.rs b/crates/engine/src/cmd/embeddings.rs index 66bc05b20..edc078423 100644 --- a/crates/engine/src/cmd/embeddings.rs +++ b/crates/engine/src/cmd/embeddings.rs @@ -18,13 +18,13 @@ pub fn commands() -> Vec { vec![ C::new("embeddings.datasets", "Satellite Embedding Datasets", "Home › Add › Web Data", |_, _| { Ok(json!({"datasets": emb::DATASETS.iter().map(|d| d.to_json()).collect::>()})) - }) + }).roles(&[]) .view(), // Not a view: it adds the footprints layer (by default). - C::new("embeddings.search", "Search Satellite Embeddings", "Home › Add › Web Data", search).params( + C::new("embeddings.search", "Search Satellite Embeddings", "Home › Add › Web Data", search).roles(&[]).params( r#"{"dataset": "alphaearth" | "earth-index" | "tessera", "year"?: number (AlphaEarth and Tessera: 2017–2025, default 2024), "bbox"?: [west, south, east, north] | "view" (the map's extent, the default), "footprints"?: bool (adds the tiles' footprints as a layer, default true)}"#, ), - C::new("embeddings.add", "Add Satellite Embeddings", "Home › Add › Web Data", add).params( + C::new("embeddings.add", "Add Satellite Embeddings", "Home › Add › Web Data", add).roles(&[]).params( r#"{"dataset": "alphaearth" | "earth-index" | "tessera", "url"?: "AlphaEarth's tile or Earth Index's file, as embeddings.search lists it (Earth Index: default the MGRS tile at the view's centre)", "year"?: number (Tessera, default 2024), "bands"?: [red, green, blue] (bands from 1; AlphaEarth default [2, 17, 10], Tessera [1, 2, 3]), "stretch"?: number (AlphaEarth: dequantised values shown over ±stretch, default 0.3), "dimension"?: number (Earth Index: a vector value kept as each point's `value`, from 1), "bbox"?: [west, south, east, north] | "view" (Earth Index and Tessera: the area read, default the map's extent), "name"?: text, "ran"?: (internal: what the app read)}"#, ), ] diff --git a/crates/engine/src/cmd/geocode.rs b/crates/engine/src/cmd/geocode.rs index 0501fde11..1efb6a15b 100644 --- a/crates/engine/src/cmd/geocode.rs +++ b/crates/engine/src/cmd/geocode.rs @@ -10,10 +10,10 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("map.findPlace", "By Place or Address…", "Home › Navigate › Locate", find_place).params( + C::new("map.findPlace", "By Place or Address…", "Home › Navigate › Locate", find_place).roles(&[]).params( r#"{"query": "a place or an address", "provider"?: "nominatim" (OpenStreetMap, default) | "photon" | "locatieserver" (PDOK, the Netherlands' addresses and places), "url"?: "a geocoding server of your own", "limit"?: 1–20, "go"?: bool (zooms to the first, default true)}"#, ), - C::new("layer.geocode", "Geocode Addresses", "Home › Add › Add Data", geocode).params( + C::new("layer.geocode", "Geocode Addresses", "Home › Add › Add Data", geocode).roles(&[]).params( r#"{"table"?: name|id (a standalone table, such as a CSV of addresses), "layer"?: name|id (or the active layer), "address": field | [fields] (joined with commas, such as street, city, country), "provider"?: "nominatim" | "photon" | "locatieserver" (PDOK, Dutch addresses; up to 10,000 a run), "url"?: "a geocoding server of your own (more than 100 addresses with nominatim or photon)", "name"?: text, "places"?: "answers already looked up, {address: {lon, lat, name} | null} (the app's background run; nothing is fetched)"}"#, ), ] diff --git a/crates/engine/src/cmd/geolibre.rs b/crates/engine/src/cmd/geolibre.rs index a8d78fd02..e307b6af2 100644 --- a/crates/engine/src/cmd/geolibre.rs +++ b/crates/engine/src/cmd/geolibre.rs @@ -6,7 +6,7 @@ use serde_json::{Value, json}; /// The importer reads files and URLs named inside the project, so callers require approval. pub fn commands() -> Vec { - vec![C::new("project.importGeolibre", "Import GeoLibre Project", "Project › Import", import_project).params( + vec![C::new("project.importGeolibre", "Import GeoLibre Project", "Project › Import", import_project).roles(&[]).params( r#"{"path"?: "*.geolibre|*.geolibre.json", "url"?: "https://…", "text"?: "project JSON", "data"?: "base64 JSON", "dir"?: "folder for local data"} → {map, layers, notes}"#, )] } diff --git a/crates/engine/src/cmd/gp.rs b/crates/engine/src/cmd/gp.rs index 4f54bbb5a..818ed2d26 100644 --- a/crates/engine/src/cmd/gp.rs +++ b/crates/engine/src/cmd/gp.rs @@ -10,18 +10,18 @@ use crate::{CmdResult, CommandSpec as C, GpRecord, Session}; pub fn commands() -> Vec { vec![ - C::new("gp.run", "Run Tool", "Analyze › Geoprocessing", run).when(need_layers).params(r#"{"tool": tool id, "params": {...} (a layer by name; a standalone table by name or id)}"#), - C::new("gp.spatialJoin", "Spatial Join", "Analyze › Overlay", spatial_join) + C::new("gp.run", "Run Tool", "Analyze › Geoprocessing", run).roles(&[crate::ParamRole { key: "params", role: crate::Role::ToolParams }]).when(need_layers).params(r#"{"tool": tool id, "params": {...} (a layer by name; a standalone table by name or id)}"#), + C::new("gp.spatialJoin", "Spatial Join", "Analyze › Overlay", spatial_join).roles(&[]) .when(need_layers) .params( r#"{"target": layer, "join": layer, "match"?: "Intersects" (default) | "Contains" | "Within", "operation"?: "Join one to one" (default) | "Join one to many", "keepAll"?: "Keep all" (default) | "Keep only matching", "prefix"?: "join_", "output"?: text}"#, ), - C::new("gp.zonalStats", "Zonal Statistics", "Analyze › Raster", zonal_stats) + C::new("gp.zonalStats", "Zonal Statistics", "Analyze › Raster", zonal_stats).roles(&[]) .when(need_layers) .params( r#"{"raster": raster, "polygons": polygon layer, "bands"?: [1-based band numbers] or single band (default 1), "stats"?: ["mean", "min", "max", "sum", "count", "stddev"] or "all" (default "all"), "prefix"?: text, "output"?: text}"#, ), - C::new("gp.bufferPoint", "Buffer A Point", "Analyze › Geoprocessing", buffer_point).params( + C::new("gp.bufferPoint", "Buffer A Point", "Analyze › Geoprocessing", buffer_point).roles(&[]).params( r#"{"lon": number, "lat": number, "distance": number, "units"?: "Kilometers" (default) | "Meters" | "Miles" | "Feet", "name"?: output}"#, ), C::new("gp.tools", "Tools", "Analyze › Geoprocessing", |_, _| { @@ -33,28 +33,28 @@ pub fn commands() -> Vec { }) .collect(), )) - }) + }).roles(&[]) .view(), - C::new("gp.geolibre", "GeoLibre Tools", "Analyze › Geoprocessing", |_, _| Ok(geolibre_status())).view(), + C::new("gp.geolibre", "GeoLibre Tools", "Analyze › Geoprocessing", |_, _| Ok(geolibre_status())).roles(&[]).view(), C::new("gp.installGeolibre", "Install GeoLibre Tools", "Analyze › Geoprocessing", |_, _| { let path = install_geolibre(&|_, _| {})?; #[cfg(all(feature = "geolibre", not(target_arch = "wasm32")))] rustgis_geolibre::set_runner(Some(std::path::PathBuf::from(&path)))?; Ok(json!({"path": path, "status": geolibre_status()})) - }) + }).roles(&[]) .view(), - C::new("gp.uninstallGeolibre", "Uninstall GeoLibre Tools", "Analyze › Geoprocessing", |_, _| uninstall_geolibre()).view(), - C::new("gp.finish", "Finish Tool Run", "Analyze › Geoprocessing", finish) + C::new("gp.uninstallGeolibre", "Uninstall GeoLibre Tools", "Analyze › Geoprocessing", |_, _| uninstall_geolibre()).roles(&[]).view(), + C::new("gp.finish", "Finish Tool Run", "Analyze › Geoprocessing", finish).roles(&[]) .params(r#"{"id": a finished run (the browser's GeoLibre tools run asynchronously)}"#), - C::new("gp.fileDelivered", "Finish File Export", "Analyze › Geoprocessing", file_delivered) + C::new("gp.fileDelivered", "Finish File Export", "Analyze › Geoprocessing", file_delivered).roles(&[]) .params(r#"{"id": pending file id, "error"?: download failure message}"#), - C::new("gp.setGeolibre", "Locate GeoLibre Runner", "Analyze › Geoprocessing", set_geolibre) + C::new("gp.setGeolibre", "Locate GeoLibre Runner", "Analyze › Geoprocessing", set_geolibre).roles(&[]) .view() .params(r#"{"path"?: "the geolibre executable (empty: look in the usual places)"}"#), C::new("gp.clearHistory", "Clear History", "Analyze › Geoprocessing › History", |s, _| { s.project.gp_history.clear(); Ok(Value::Null) - }), + }).roles(&[]), ] } @@ -1225,6 +1225,7 @@ fn queue_file(s: &mut Session, file_name: String, bytes: Vec, record: GpReco Ok(json!({"pending": true, "download": id, "file": file_name})) } +/// Records a completed file delivery through the generated-message history seam. fn file_delivered(s: &mut Session, p: &Value) -> CmdResult { let id = p.get("id").and_then(Value::as_u64).ok_or("id is required")?; let i = s.file_outputs.iter().position(|f| f.id == id).ok_or("no pending file with that id")?; @@ -1244,7 +1245,7 @@ fn file_delivered(s: &mut Session, p: &Value) -> CmdResult { s.notify(message.clone(), !file.record.ok); } let out = json!({"file": file.file_name, "delivered": file.record.ok, "messages": file.record.messages}); - s.project.gp_history.push(file.record.scrubbed()); + s.push_gp_record(file.record); if s.project.gp_history.len() > 500 { s.project.gp_history.remove(0); } @@ -1336,10 +1337,17 @@ fn apply(s: &mut Session, ran: Ran, params: Value, res: Result { - s.project.gp_history.push( - GpRecord { tool: name, id, params, command, at, ok: false, messages: vec![e.clone()], millis, outputs: Vec::new() } - .scrubbed(), - ); + s.push_gp_record(GpRecord { + tool: name, + id, + params, + command, + at, + ok: false, + messages: vec![e.clone()], + millis, + outputs: Vec::new(), + }); return Err(e); } Ok(r) => r, @@ -1361,10 +1369,17 @@ fn apply(s: &mut Session, ran: Ran, params: Value, res: Result Vec::new(), }; let outputs = ids.iter().filter_map(|id| s.map().layer(*id).map(|l| l.name.clone())).collect(); - s.project - .gp_history - .push(GpRecord { tool: name, id, params, command, at, ok: true, messages: result.messages, millis, outputs }.scrubbed()); + s.push_gp_record(GpRecord { tool: name, id, params, command, at, ok: true, messages: result.messages, millis, outputs }); if s.project.gp_history.len() > 500 { s.project.gp_history.remove(0); } diff --git a/crates/engine/src/cmd/gpkg.rs b/crates/engine/src/cmd/gpkg.rs index 2d61af3cd..9bfd33696 100644 --- a/crates/engine/src/cmd/gpkg.rs +++ b/crates/engine/src/cmd/gpkg.rs @@ -11,14 +11,14 @@ use serde_json::{Value, json}; pub fn commands() -> Vec { vec![ - C::new("gpkg.addLayer", "Edit GeoPackage Table", "Home › Add › Add Data", add_layer).params( + C::new("gpkg.addLayer", "Edit GeoPackage Table", "Home › Add › Add Data", add_layer).roles(&[]).params( r#"{"path": "a GeoPackage file on this computer", "table"?: "its feature table (needed when it has several)", "limit"?: "the most rows (default and most: 100000)", "name"?: string}"#, ), - C::new("gpkg.pendingEdits", "GeoPackage Edits", "Edit › Manage Edits", pending_edits).view() + C::new("gpkg.pendingEdits", "GeoPackage Edits", "Edit › Manage Edits", pending_edits).roles(&[]).view() .params(r#"{"layer"?: "a layer added with gpkg.addLayer (default: the active one)"}"#), - C::new("gpkg.saveEdits", "Save to GeoPackage", "Edit › Manage Edits", save_edits) + C::new("gpkg.saveEdits", "Save to GeoPackage", "Edit › Manage Edits", save_edits).roles(&[]) .params(r#"{"layer"?: "a layer added with gpkg.addLayer (default: the active one)"}"#), - C::new("gpkg.discardEdits", "Discard GeoPackage Edits", "Edit › Manage Edits", discard_edits) + C::new("gpkg.discardEdits", "Discard GeoPackage Edits", "Edit › Manage Edits", discard_edits).roles(&[]) .params(r#"{"layer"?: "a layer added with gpkg.addLayer (default: the active one)"}"#), ] } diff --git a/crates/engine/src/cmd/graphics.rs b/crates/engine/src/cmd/graphics.rs index 88a203ca7..4d89de39a 100644 --- a/crates/engine/src/cmd/graphics.rs +++ b/crates/engine/src/cmd/graphics.rs @@ -22,21 +22,21 @@ pub fn commands() -> Vec { let name = ps(p, "name").map(str::trim).filter(|n| !n.is_empty()).unwrap_or("Graphics"); let id = new_layer(s, name); Ok(json!({"layer": id})) - }) + }).roles(&[]) .params(r#"{"name"?: string}"#), - C::new("graphics.add", "Add Graphic", "Edit › Graphics", add).params( + C::new("graphics.add", "Add Graphic", "Edit › Graphics", add).roles(&[]).params( r##"{"shape": "point"|"text"|"line"|"curve"|"freehand"|"polygon"|"lasso"|"rectangle"|"circle"|"ellipse"|"picture", "points": [[lon, lat], …] (rectangle, ellipse, picture: two corners; circle: centre and a point on the edge), "layer"?: graphics layer, "text"?: string (dynamic tokens: {map} {scale} {date} {time} {crs} {layers} {center}), "fill"?: "#RRGGBBAA", "outline"?: "#RRGGBBAA", "width"?: number, "size"?: number, "fontSize"?: number, "bold"?: bool, "italic"?: bool, "align"?: "left"|"center"|"right" (a text's lines in their block, which stays centred on its point), "letterSpacing"?: points, -5 to 50, "image"?: base64 PNG/JPEG, "path"?: picture file}"##, ), - C::new("graphics.style", "Graphic Style", "Edit › Graphics", style) + C::new("graphics.style", "Graphic Style", "Edit › Graphics", style).roles(&[]) .when(super::need_selection) .params(r##"{"fill"?: "#RRGGBBAA", "outline"?: "#RRGGBBAA", "width"?: number, "size"?: number, "fontSize"?: number, "bold"?: bool, "italic"?: bool, "align"?: "left"|"center"|"right", "letterSpacing"?: points, -5 to 50, "text"?: string}"##), - C::new("graphics.edit", "Edit Graphic", "Edit › Graphics", edit).params( + C::new("graphics.edit", "Edit Graphic", "Edit › Graphics", edit).roles(&[]).params( r##"{"layer": graphics layer, "row": number, "points"?: [[lon, lat]] (a point or a text: where it sits), "text"?: string, "fontSize"?: number, "bold"?: bool, "italic"?: bool, "align"?: "left"|"center"|"right", "letterSpacing"?: points, -5 to 50, "fill"?: "#RRGGBBAA", "outline"?: "#RRGGBBAA", "width"?: number, "size"?: number}"##, ), C::new("graphics.keepTool", "Keep the last tool active", "Edit › Graphics", |s, p| { s.keep_tool = b(p, "on").unwrap_or(!s.keep_tool); Ok(json!({"keepTool": s.keep_tool})) - }) + }).roles(&[]) .view() .params(r#"{"on"?: bool}"#), ] diff --git a/crates/engine/src/cmd/group.rs b/crates/engine/src/cmd/group.rs index 1c6063c31..d868f4f3a 100644 --- a/crates/engine/src/cmd/group.rs +++ b/crates/engine/src/cmd/group.rs @@ -10,38 +10,38 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("group.add", "New Group", "Layers", add).params( + C::new("group.add", "New Group", "Layers", add).roles(&[]).params( r#"{"name"?: text (default New Group), "parent"?: name|id (the group it goes in), "layers"?: [name|id] (moved into it, out of any other group; together in the drawing order at the topmost)}"#, ), - C::new("group.set", "Group Properties", "Layers", set).params( + C::new("group.set", "Group Properties", "Layers", set).roles(&[]).params( r#"{"group": name|id, "name"?: text, "visible"?: bool (toggles when no other property is given), "transparency"?: 0-100 (on top of each layer's own)}"#, ), - C::new("group.expand", "Expand Group", "Layers", expand) + C::new("group.expand", "Expand Group", "Layers", expand).roles(&[]) .params(r#"{"group": name|id, "on"?: bool (toggles when omitted)}"#) .view(), - C::new("group.scaleRange", "Group Visibility Range", "Layers", scale_range).params( + C::new("group.scaleRange", "Group Visibility Range", "Layers", scale_range).roles(&[]).params( r#"{"group": name|id, "min"?: scale (out beyond), "max"?: scale (in beyond); both set on each call, one left out is cleared; its layers draw only within it and their own}"#, ), - C::new("group.move", "Move Group", "Layers", move_group).params( + C::new("group.move", "Move Group", "Layers", move_group).roles(&[]).params( r#"{"group": name|id, "to": position among the other layers, counted in layers (0 = top; past the end, the bottom; never inside another group: above it; a group in another stays in it, at most at its top or bottom)}"#, ), - C::new("group.setParent", "Move Group To Group", "Layers", set_parent).params( + C::new("group.setParent", "Move Group To Group", "Layers", set_parent).roles(&[]).params( r#"{"group": name|id, "parent": name|id|null (the group it goes in; null: the top level), "before"?: name|id|null (a layer or group: places the group just above it; null puts it at the end of the parent, or the bottom at the top level; omitted: next to the parent's layers, or where it is at the top level)}"#, ), - C::new("group.indent", "Indent Group", "Layers", indent_group) + C::new("group.indent", "Indent Group", "Layers", indent_group).roles(&[]) .params(r#"{"group": name|id} (into the group just above it in the Layers pane, as its last item)"#), - C::new("group.outdent", "Outdent Group", "Layers", outdent_group) + C::new("group.outdent", "Outdent Group", "Layers", outdent_group).roles(&[]) .params(r#"{"group": name|id} (out of the group it is in, up one level, just below that group)"#), - C::new("group.ungroup", "Ungroup", "Layers", ungroup) + C::new("group.ungroup", "Ungroup", "Layers", ungroup).roles(&[]) .params(r#"{"group": name|id} (its layers and subgroups stay, one level up)"#), - C::new("group.remove", "Remove Group", "Layers", remove).params(r#"{"group": name|id} (and its layers and subgroups)"#), - C::new("layer.group", "Move To Group", "Layers", move_to_group) + C::new("group.remove", "Remove Group", "Layers", remove).roles(&[]).params(r#"{"group": name|id} (and its layers and subgroups)"#), + C::new("layer.group", "Move To Group", "Layers", move_to_group).roles(&[]) .when(super::need_layer) .params(r#"{"layer"?: name|id | [name|id] (default: the layer selection), "group": name|id|null (null takes it out of its group), "before"?: name|id|null (places a single layer just above this layer, one of the group's or its subgroups'; null puts it at the group's end, or the bottom out of a group; omitted: next to the group's layers, or where it is when taken out)}"#), - C::new("layer.indent", "Indent Layer", "Layers", indent_layer) + C::new("layer.indent", "Indent Layer", "Layers", indent_layer).roles(&[]) .when(super::need_layer) .params(r#"{"layer"?: name|id (default: the active layer)} (into the group just above it in the Layers pane, as its last item)"#), - C::new("layer.outdent", "Outdent Layer", "Layers", outdent_layer) + C::new("layer.outdent", "Outdent Layer", "Layers", outdent_layer).roles(&[]) .when(super::need_layer) .params(r#"{"layer"?: name|id (default: the active layer)} (out of its group, up one level, just below that group)"#), ] diff --git a/crates/engine/src/cmd/history.rs b/crates/engine/src/cmd/history.rs index 3a22bd151..f1eed6a75 100644 --- a/crates/engine/src/cmd/history.rs +++ b/crates/engine/src/cmd/history.rs @@ -14,70 +14,82 @@ use crate::{CmdResult, CommandSpec as C, GpRecord, Session}; pub fn commands() -> Vec { vec![ C::new("gp.historyCode", "Copy Run As Code", "Analyze › Geoprocessing", code) + .roles(&[]) .view() .params(r#"{"run": index in the history (0 = the first run), "as"?: "python" (default)|"command" (a Script pane line)|"cli"}"#), - C::new("gp.historyScript", "Save History As Python Script", "Analyze › Geoprocessing", script).view(), + C::new("gp.historyScript", "Save History As Python Script", "Analyze › Geoprocessing", script).roles(&[]).view(), ] } +/// Formats a replayable code fragment inside the guarded history projection. fn code(s: &mut Session, p: &Value) -> CmdResult { let i = uint(p, "run").and_then(|i| usize::try_from(i).ok()).ok_or("run is required: a run's place in the history, from 0")?; - let r = s.project.gp_history.get(i).ok_or("no run there in the history")?; - let (cmd, params) = replay(r)?; - let text = match ps(p, "as").unwrap_or("python") { - "python" => python(&cmd, ¶ms), - "command" => format!("{cmd} {params}"), - "cli" => format!("rustgis-cli run --project {} --cmd {}", shell(&project_file(s)), shell(&format!("{cmd}={params}"))), - other => return Err(format!("unknown kind '{other}': python, command or cli")), - }; + let project = project_file(s); + let r = s.project.gp_history.get_mut(i).ok_or("no run there in the history")?; + let (text, report) = crate::durable::output(r, None, |r| { + let (cmd, params) = replay_written(r)?; + match ps(p, "as").unwrap_or("python") { + "python" => Ok(python(&cmd, ¶ms)), + "command" => Ok(format!("{cmd} {params}")), + "cli" => Ok(format!("rustgis-cli run --project {} --cmd {}", shell(&project), shell(&format!("{cmd}={params}")))), + other => Err(format!("unknown kind '{other}': python, command or cli")), + } + })?; + crate::durable::notify(s, &report); Ok(json!({"code": text})) } /// The history as a Python script that opens the project and runs each tool again, in order. A /// run that failed, or can't be run again, is there as a comment. fn script(s: &mut Session, _: &Value) -> CmdResult { - let runs = &s.project.gp_history; - if runs.is_empty() { + if s.project.gp_history.is_empty() { return Err("the history is empty: run a tool first".into()); } - let name = &s.project.name; - let mut text = format!( - "# The geoprocessing history of the RustGIS project {}, run again in order.\n\ + let name = s.project.name.clone(); + let project = project_file(s); + let unsaved = s.path.is_none(); + let dirty = s.dirty(); + let (result, report) = crate::durable::output(&mut s.project.gp_history, None, |runs| { + let mut text = format!( + "# The geoprocessing history of the RustGIS project {}, run again in order.\n\ # Layers are named as they were then, so this opens the project they belong to.\n", - quoted(name) - ); - // The script opens the saved file: a layer added since, which a run may have used, isn't - // there until the project is saved again. - if s.path.is_none() { - text.push_str("# The project hasn't been saved yet: save it as this file first.\n"); - } else if s.dirty() { - text.push_str("# The project has changes that aren't saved: save it first, so its layers are there.\n"); - } - text.push_str(&format!("import rustgis\n\ns = rustgis.Session({})\n", quoted(&project_file(s)))); - for (n, r) in runs.iter().enumerate() { - let when = if r.at > 0 { - let (date, time) = rustgis_map::graphics::civil(r.at); - format!(" ({date} {time} UTC)") - } else { - String::new() - }; - text.push_str(&format!("\n# {}. {}{when}", n + 1, line(&r.tool))); - match replay(r) { - Ok((cmd, params)) if r.ok => text.push_str(&format!("\n{}\n", python(&cmd, ¶ms))), - Ok((cmd, params)) => { - // An error may quote a connection, password and all: redacted as a parameter's - // value is, not for its addresses only. - let why = r.messages.first().map(|m| line(&redacted_text(m))).unwrap_or_default(); - text.push_str(&format!(": failed when it ran ({why})\n")); - for l in python(&cmd, ¶ms).lines() { - text.push_str(&format!("# {l}\n")); + quoted(&name) + ); + // The script opens the saved file: a layer added since, which a run may have used, isn't + // there until the project is saved again. + if unsaved { + text.push_str("# The project hasn't been saved yet: save it as this file first.\n"); + } else if dirty { + text.push_str("# The project has changes that aren't saved: save it first, so its layers are there.\n"); + } + text.push_str(&format!("import rustgis\n\ns = rustgis.Session({})\n", quoted(&project))); + for (n, r) in runs.iter().enumerate() { + let when = if r.at > 0 { + let (date, time) = rustgis_map::graphics::civil(r.at); + format!(" ({date} {time} UTC)") + } else { + String::new() + }; + text.push_str(&format!("\n# {}. {}{when}", n + 1, line(&r.tool))); + match replay_written(r) { + Ok((cmd, params)) if r.ok => text.push_str(&format!("\n{}\n", python(&cmd, ¶ms))), + Ok((cmd, params)) => { + // An error may quote a connection, password and all: redacted as a parameter's + // value is, not for its addresses only. + let why = r.messages.first().map(|m| line(&redacted_text(m))).unwrap_or_default(); + text.push_str(&format!(": failed when it ran ({why})\n")); + for l in python(&cmd, ¶ms).lines() { + text.push_str(&format!("# {l}\n")); + } } + Err(e) => text.push_str(&format!(": {}\n", line(&e))), } - Err(e) => text.push_str(&format!(": {}\n", line(&e))), } - } - let file = format!("{}_history.py", file_stem(name)); - Ok(json!({"fileName": file, "text": text})) + let file = format!("{}_history.py", file_stem(&name)); + Ok(json!({"fileName": file, "text": text})) + })?; + crate::durable::notify(s, &report); + Ok(result) } /// The commands other than `gp.run` a run is copied as. The history comes from the project file, @@ -88,14 +100,21 @@ const REPLAYED: &[&str] = &["gp.bufferPoint"]; /// The command that runs `r` again, with its parameters: `gp.run` with the tool, or the command /// that ran it (a point buffered where the map was clicked, `gp.bufferPoint`). pub fn replay(r: &GpRecord) -> Result<(String, Value), String> { + let (id, params) = replay_written(r)?; + let (clean, _) = crate::roles::command(&id, ¶ms, 0); + Ok((id, clean.unwrap_or(params))) +} + +/// Formats a record already inside the durable output transaction; data stays replayable. +fn replay_written(r: &GpRecord) -> Result<(String, Value), String> { if let Some((cmd, params)) = &r.command { let Some(cmd) = REPLAYED.iter().find(|c| **c == cmd.as_str()) else { return Err(format!("{} can't be run again: the history names a command that isn't a geoprocessing one", r.tool)); }; - return Ok((cmd.to_string(), redacted(params))); + return Ok((cmd.to_string(), params.clone())); } let id = tool_of(r).ok_or_else(|| format!("{} can't be run again: the history doesn't say which tool it was", r.tool))?; - Ok(("gp.run".into(), json!({"tool": id, "params": redacted(&r.params)}))) + Ok(("gp.run".into(), json!({"tool": id, "params": r.params}))) } /// The id of the tool `r` ran with `gp.run`: kept with the run, or, in projects from before it @@ -119,8 +138,7 @@ pub(crate) fn redacted(v: &Value) -> Value { Value::String(t) => Value::String(redacted_text(t)), Value::Array(a) => Value::Array(a.iter().map(redacted).collect()), Value::Object(o) => { - let web = |v: &Value| v.as_str().is_some_and(|t| t.trim_start().get(..4).is_some_and(|p| p.eq_ignore_ascii_case("http"))); - let beside_address = |k: &str| o.iter().any(|(sibling, v)| sibling != k && web(v)); + let beside_address = |k: &str| o.iter().any(|(sibling, v)| sibling != k && crate::project_log::address_context(sibling, v, 0)); let secret = |k: &str, v: &Value| { secret_key(k) && !matches!(v, Value::Bool(_) | Value::Null) diff --git a/crates/engine/src/cmd/imagery.rs b/crates/engine/src/cmd/imagery.rs index a06df1919..002b2920a 100644 --- a/crates/engine/src/cmd/imagery.rs +++ b/crates/engine/src/cmd/imagery.rs @@ -38,42 +38,42 @@ fn first() -> String { pub fn commands() -> Vec { vec![ - C::new("raster.georeference", "Georeference", "Analyze › Alignment", georeference).params( + C::new("raster.georeference", "Georeference", "Analyze › Alignment", georeference).roles(&[]).params( r#"{"layer": raster, "points"?: [[column, row, lon, lat], …] (two or more; with "transform": [column, row, x, y, on?]), "extent"?: [minLon, minLat, maxLon, maxLat], "transform"?: "similarity"|"affine"|"projective"|"polynomial2"|"polynomial3"|"spline" (fits the points and warps the raster onto a north-up grid covering the warped image; without it the points and extent only shift and scale the grid), "pointsCrs"?: EPSG code or "EPSG:n" of the points' x and y (default 4326), "crs"?: the output's (default pointsCrs), "resampling"?: "nearest"|"bilinear"|"cubic" (default bilinear), "cellSize"?: output units (default about the source pixel's size on the ground), "name"?: new layer for the result (default replaces the layer's raster), "path"?: "*.tif" (also writes the warp as a cloud-optimised GeoTIFF)}"#, ), - C::new("raster.project", "Project Raster", "Analyze › Alignment", project_raster).params( + C::new("raster.project", "Project Raster", "Analyze › Alignment", project_raster).roles(&[]).params( r#"{"layer": raster, "crs"?: EPSG code or "EPSG:n" of the output (default the layer's), "cellSize"?: number or [x, y] in the output's units (default about the layer's pixel there), "origin"?: [x, y] a cell corner falls on (default [0, 0], so grids of the same cellSize line up), "extent"?: polygon layer (its selected or all features' bounds) | [west, south, east, north] in degrees | "view" (default the layer's footprint), "grid"?: raster layer whose coordinate system, cells and extent are taken exactly (instead of crs, cellSize, origin and extent), "resampling"?: "nearest" (default) | "bilinear" | "cubic" | "average" (the mean of the pixels whose centres fall in a cell) | "mode" (their most frequent value, the least of those tied), "name"?: new layer (default " projected"), "path"?: GeoTIFF to write it to}"#, ), - C::new("raster.rasterize", "Polygons to Raster", "Analyze › Alignment", rasterize).params( + C::new("raster.rasterize", "Polygons to Raster", "Analyze › Alignment", rasterize).roles(&[]).params( r#"{"layer": polygon layer (its selected or all features; a later one over an earlier), "field"?: numeric field (its values) or text field (codes 1 to n in sorted order of its values, listed in a table named after the result with classes), "value"?: number every polygon gives when there is no field (default 1), "nodata"?: number of the cells under no polygon (default 0), "crs"?: EPSG code or "EPSG:n" of the output (default the layer's), "cellSize"?: number or [x, y] in the output's units (required without grid), "origin"?: [x, y] a cell corner falls on (default [0, 0], so grids of the same cellSize line up), "extent"?: polygon layer (its selected or all features' bounds) | [west, south, east, north] in degrees | "view" (default the polygons' bounds), "grid"?: raster layer whose coordinate system, cells and extent are taken exactly (instead of crs, cellSize, origin and extent), "name"?: new layer (default " raster"), "path"?: GeoTIFF to write it to}"#, ), // A fit is a measurement of the points, not a project change: no undo step. - C::new("raster.gcpFit", "Control Point Fit", "Analyze › Alignment", gcp_fit).view().params( + C::new("raster.gcpFit", "Control Point Fit", "Analyze › Alignment", gcp_fit).roles(&[]).view().params( r#"{"points": [[column, row, x, y, on?], …] (on: 1/0 or true/false, default on), "transform"?: "similarity"|"affine"|"projective"|"polynomial2"|"polynomial3"|"spline" (default: similarity under 3 points, else affine), "compare"?: bool (fit every transform and recommend one; transform and leaveOneOut are then ignored), "leaveOneOut"?: bool (default true)}"#, ), - C::new("raster.change", "Change Detection", "Analyze › Image Analysis", change) + C::new("raster.change", "Change Detection", "Analyze › Image Analysis", change).roles(&[]) .params(r#"{"before": raster, "after": raster, "band"?: 1-based, "method"?: "difference"|"ratio"|"percent", "name"?, "ran"?: (internal: a raster the app has worked out)}"#), - C::new("raster.spectral", "Spectral Analysis", "Analyze › Image Analysis", spectral) + C::new("raster.spectral", "Spectral Analysis", "Analyze › Image Analysis", spectral).roles(&[]) .view() .params(r#"{"layer": raster, "lon"?, "lat"?, "zone"?: polygon layer (the selected or first polygon)}"#), C::new("raster.classify", "Classification Wizard", "Analyze › Image Classification", |s, p| { super::workflows::made_or_ran(s, p, "raster.classify") - }) + }).roles(&[]) .params(r#"{"layer": raster, "classes"?: 2..50, "bands"?: [1-based], "name"?: output, "ran"?: (internal: classes the app has worked out)}"#), - C::new("raster.classifySupervised", "Classification Tools", "Analyze › Image Classification", supervised) + C::new("raster.classifySupervised", "Classification Tools", "Analyze › Image Classification", supervised).roles(&[]) .params(r#"{"layer": raster, "training": polygon layer, "classField": field, "bands"?: [1-based], "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#), - C::new("raster.index", "Indices", "Analyze › Imagery Tools", index).params( + C::new("raster.index", "Indices", "Analyze › Imagery Tools", index).roles(&[]).params( r#"{"layer": raster, "index": "ndvi"|"ndwi"|"savi"|"nbr"|"ndbi"|"evi", "red"?, "green"?, "blue"?, "nir"?, "swir"?: band numbers, "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#, ), - C::new("raster.process", "Process", "Analyze › Imagery Tools", process) + C::new("raster.process", "Process", "Analyze › Imagery Tools", process).roles(&[]) .params(r#"{"layer": raster, "operation": "clip"|"resample", "extent"?: [minLon, minLat, maxLon, maxLat] (clip; the view when omitted), "polygon"?: polygon layer (clip to its selected or all polygons; cells outside become no-data), "selectedOnly"?: bool (clip to selected polygons when true, all when false), "factor"?: n (resample: n×n pixels become one), "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#), - C::new("raster.mosaic", "Mosaic Rasters", "Analyze › Imagery Tools", raster_mosaic).params( + C::new("raster.mosaic", "Mosaic Rasters", "Analyze › Imagery Tools", raster_mosaic).roles(&[]).params( r#"{"inputs": [raster layer or file path, …] (two or more, in one coordinate system and cell size; where they overlap the first wins), "name"?: output, "path"?: GeoTIFF to write the mosaic to}"#, ), - C::new("raster.editPixels", "Pixel Editor", "Analyze › Imagery Tools", edit_pixels) + C::new("raster.editPixels", "Pixel Editor", "Analyze › Imagery Tools", edit_pixels).roles(&[]) .params(r#"{"layer": raster, "polygon": [[lon, lat], …], "value"?: number (no-data when omitted), "band"?: 1-based (all when omitted)}"#), // Measurements are session results, not project changes: no undo step. - C::new("raster.measure", "Mensuration Tools", "Analyze › Mensuration", measure).view().params( + C::new("raster.measure", "Mensuration Tools", "Analyze › Mensuration", measure).roles(&[]).view().params( r#"{"layer"?: elevation raster, "kind": "distance"|"area"|"height"|"volume", "points": [[lon, lat], …], "base"?: metres (volume)}"#, ), C::new("raster.measurements", "Results", "Analyze › Mensuration", |s, p| { @@ -81,10 +81,10 @@ pub fn commands() -> Vec { s.measurements.clear(); } Ok(Value::Array(s.measurements.clone())) - }) + }).roles(&[]) .view() .params(r#"{"clear"?: bool}"#), - C::new("imagery.workspace", "New Workspace", "Analyze › Ortho Mapping", new_workspace).params( + C::new("imagery.workspace", "New Workspace", "Analyze › Ortho Mapping", new_workspace).roles(&[]).params( r#"{"kind": "ortho"|"reality", "name"?: string, "images": [raster layers], "cellSize"?: degrees, "method"?: "first"|"mean"}"#, ), C::new("imagery.workspaceOptions", "Workspace Options", "Analyze › Ortho Mapping", |s, p| { @@ -97,10 +97,10 @@ pub fn commands() -> Vec { w.method = method(p, "first")?; } serde_json::to_value(&*w).map_err(|e| e.to_string()) - }) + }).roles(&[]) .params(r#"{"name": workspace, "cellSize"?: degrees|null, "method"?: "first"|"mean"}"#), - C::new("imagery.mosaic", "Build Mosaic", "Analyze › Ortho Mapping", mosaic).params(r#"{"name": workspace, "ran"?: (internal: a raster the app has worked out)}"#), - C::new("imagery.mesh", "Export Mesh", "Analyze › Reality Mapping", mesh) + C::new("imagery.mosaic", "Build Mosaic", "Analyze › Ortho Mapping", mosaic).roles(&[]).params(r#"{"name": workspace, "ran"?: (internal: a raster the app has worked out)}"#), + C::new("imagery.mesh", "Export Mesh", "Analyze › Reality Mapping", mesh).roles(&[]) .view() .params(r#"{"layer": elevation raster, "path"?: "*.obj" (the text is returned without one), "step"?: pixels between vertices}"#), ] diff --git a/crates/engine/src/cmd/items.rs b/crates/engine/src/cmd/items.rs index 07df243d0..6a9812c1d 100644 --- a/crates/engine/src/cmd/items.rs +++ b/crates/engine/src/cmd/items.rs @@ -30,14 +30,14 @@ pub fn commands() -> Vec { s.project.folders.push(path.to_string()); } Ok(json!({"folders": s.project.folders})) - }) + }).roles(&[]) .params(r#"{"path": folder}"#), C::new("project.removeFolder", "Remove Folder", "Browser", |s, p| { let path = ps(p, "path").ok_or("path is required")?; let before = s.project.folders.len(); s.project.folders.retain(|f| f != path); if s.project.folders.len() == before { Err(format!("{path} isn't connected")) } else { Ok(Value::Null) } - }) + }).roles(&[]) .params(r#"{"path": folder}"#), C::new("project.connections", "Connections", "Browser", |s, _| { let folders: Vec = s.project.folders.iter().map(|f| json!({"path": f, "available": Path::new(f).is_dir()})).collect(); @@ -50,21 +50,21 @@ pub fn commands() -> Vec { "ramps": s.project.ramps.iter().map(|r| r.name().to_string()).collect::>(), "tasks": s.project.tasks.iter().map(|t| t.name.clone()).collect::>(), })) - }) + }).roles(&[]) .view(), // Map and layer files. - C::new("map.save", "Save Map File", "Publish › Save As", |s, p| save_item(s, "map", p)) + C::new("map.save", "Save Map File", "Publish › Save As", |s, p| save_item(s, "map", p)).roles(&[]) .view() .params(r#"{"path": "*.rgismap", "map"?: name}"#), - C::new("map.import", "Import", "Command Search", import_maps).params(r#"{"path": "*.rgismap|*.rgis|*.rgispkg" | "text": map file JSON | "data": base64 file}"#), - C::new("layer.save", "Save Layer File", "Publish › Save As", |s, p| save_item(s, "layer", p)) + C::new("map.import", "Import", "Command Search", import_maps).roles(&[crate::ParamRole { key: "text", role: crate::Role::Document(crate::Document::Map) }]).params(r#"{"path": "*.rgismap|*.rgis|*.rgispkg" | "text": map file JSON | "data": base64 file}"#), + C::new("layer.save", "Save Layer File", "Publish › Save As", |s, p| save_item(s, "layer", p)).roles(&[]) .view() .when(need_layer) .params(r#"{"layer"?, "path": "*.rgislyr"}"#), C::new("layer.addLayerFile", "Add Layer File", "Home › Add", |s, p| { let text = read_text(p)?; super::layer::add_layer_text(s, &text) - }) + }).roles(&[]) .params(r#"{"path"?: "*.rgislyr", "text"?: layer file JSON, "data"?: base64 file}"#), // Styles. C::new("style.new", "New Style", "Style › Styles", |s, p| { @@ -74,7 +74,7 @@ pub fn commands() -> Vec { let item = StyleItem { name: name.clone(), geometry: l.geometry_type(), renderer: l.renderer.clone(), labels: l.labels.clone(), transparency: l.transparency }; add_style(s, item); Ok(json!({"style": name})) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "name"?: string}"#), C::new("style.add", "Add Style", "Style › Styles", |s, p| { @@ -87,9 +87,9 @@ pub fn commands() -> Vec { n += 1; } Ok(json!({"added": n})) - }) + }).roles(&[]) .params(r#"{"path"?: "*.rgisstyle", "text"?: style file JSON, "data"?: base64 file}"#), - C::new("style.import", "Import Styles", "Style › Styles", import_styles).params(r#"{"path"?: "*.rgislyr|*.rgismap|*.rgis", "text"?: JSON, "data"?: base64 file}"#), + C::new("style.import", "Import Styles", "Style › Styles", import_styles).roles(&[]).params(r#"{"path"?: "*.rgislyr|*.rgismap|*.rgis", "text"?: JSON, "data"?: base64 file}"#), C::new("style.apply", "Apply Style", "Style › Styles", |s, p| { let name = ps(p, "name").ok_or("name is required")?; let st = s.project.styles.iter().find(|t| t.name.eq_ignore_ascii_case(name)).cloned().ok_or_else(|| format!("no style '{name}'"))?; @@ -124,7 +124,7 @@ pub fn commands() -> Vec { s.notify(format!("{lost} saved label move(s) dropped with label classes style {name} doesn't have"), false); } Ok(Value::Null) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "name": style}"#), C::new("style.remove", "Delete Style", "Browser", |s, p| { @@ -132,9 +132,9 @@ pub fn commands() -> Vec { let before = s.project.styles.len(); s.project.styles.retain(|t| !t.name.eq_ignore_ascii_case(name)); if s.project.styles.len() == before { Err(format!("no style '{name}'")) } else { Ok(Value::Null) } - }) + }).roles(&[]) .params(r#"{"name": style}"#), - C::new("style.save", "Save Style Items", "Publish › Save As", |s, p| save_item(s, "styles", p)) + C::new("style.save", "Save Style Items", "Publish › Save As", |s, p| save_item(s, "styles", p)).roles(&[]) .view() .params(r#"{"path": "*.rgisstyle"}"#), // Custom colour ramps. @@ -167,7 +167,7 @@ pub fn commands() -> Vec { s.project.ramps.push(r); } Ok(json!({"added": names})) - }) + }).roles(&[]) .params( r#"{"path"?: "*.rgisramp|*.xml|*.qml|*.txt|*.cpt", "text"?: file contents, "data"?: base64 file, "file"?: file name for "text" or "data", "format"?: "rgisramp"|"qgis"|"gdal"|"cpt", "name"?: name for a single ramp, "kind"?: "sequential"|"diverging"|"qualitative"}"#, ), @@ -176,16 +176,16 @@ pub fn commands() -> Vec { let before = s.project.ramps.len(); s.project.ramps.retain(|r| !r.name().eq_ignore_ascii_case(name.trim())); if s.project.ramps.len() == before { Err(format!("no colour ramp '{name}'")) } else { Ok(Value::Null) } - }) + }).roles(&[]) .params(r#"{"name": colour ramp}"#), - C::new("ramp.save", "Save Color Ramps", "Publish › Save As", |s, p| save_item(s, "ramps", p)) + C::new("ramp.save", "Save Color Ramps", "Publish › Save As", |s, p| save_item(s, "ramps", p)).roles(&[]) .view() .params(r#"{"path": "*.rgisramp"}"#), // Project templates. - C::new("project.saveTemplate", "Save Project Template", "Publish › Save As", |s, p| save_item(s, "template", p)) + C::new("project.saveTemplate", "Save Project Template", "Publish › Save As", |s, p| save_item(s, "template", p)).roles(&[]) .view() .params(r#"{"path": "*.rgistemplate", "name"?: template name}"#), - C::new("project.newTemplate", "New Custom Template", "Publish › Save As", |s, p| save_item(s, "template", p)) + C::new("project.newTemplate", "New Custom Template", "Publish › Save As", |s, p| save_item(s, "template", p)).roles(&[]) .view() .params(r#"{"path": "*.rgistemplate", "name"?: template name}"#), C::new("project.fromTemplate", "Open Custom Template", "Command Search", |s, p| { @@ -204,21 +204,21 @@ pub fn commands() -> Vec { super::connections::resolve_project(s); super::connections::note_waiting(s); Ok(Value::Null) - }) + }).roles(&[]) .view() .params(r#"{"path"?: "*.rgistemplate", "text"?: template JSON, "data"?: base64 file, "name"?: new project name}"#), // Packages. - C::new("project.package", "Package Project", "Publish › Package", |s, p| save_item(s, "projectPackage", p)) + C::new("project.package", "Package Project", "Publish › Package", |s, p| save_item(s, "projectPackage", p)).roles(&[]) .view() .params(r#"{"path": "*.rgispkg"}"#), - C::new("map.package", "Package Map", "Publish › Package", |s, p| save_item(s, if b(p, "mobile").unwrap_or(false) { "mobilePackage" } else { "mapPackage" }, p)) + C::new("map.package", "Package Map", "Publish › Package", |s, p| save_item(s, if b(p, "mobile").unwrap_or(false) { "mobilePackage" } else { "mapPackage" }, p)).roles(&[]) .view() .params(r#"{"path": "*.rgismpk", "mobile"?: bool (simplify geometry and drop hidden layers, for phones and tablets), "tolerance"?: degrees}"#), - C::new("gp.package", "Package Results", "Publish › Package", |s, p| save_item(s, "gpPackage", p)) + C::new("gp.package", "Package Results", "Publish › Package", |s, p| save_item(s, "gpPackage", p)).roles(&[]) .view() .params(r#"{"path": "*.rgisgpk"}"#), // Layer templates. - C::new("layer.template", "Layer Template", "Data › Layer Templates", layer_template) + C::new("layer.template", "Layer Template", "Data › Layer Templates", layer_template).roles(&[]) .params(r#"{"template": "points"|"lines"|"polygons"|"places"|"routes"|"areas"|"observations", "name"?: string}"#), // Tasks. C::new("task.new", "New Task", "Analyze › Tasks", |s, p| { @@ -257,7 +257,7 @@ pub fn commands() -> Vec { } s.project.tasks.push(Task { name: name.clone(), description: ps(p, "description").unwrap_or("").chars().take(2000).collect(), steps: steps.into_iter().take(200).collect() }); Ok(json!({"task": name})) - }) + }).roles(&[crate::ParamRole { key: "steps", role: crate::Role::TaskSteps }]) .params(r#"{"name"?: string, "description"?: string, "steps"?: [{"title", "instructions"?, "command": id, "params"?: {}}], "fromHistory"?: bool (one step per geoprocessing run), "imported"?: bool (from a file: its steps may not touch files)}"#), C::new("task.addStep", "Add Task Step", "Tasks", |s, p| { let step: TaskStep = serde_json::from_value(p.clone()).map_err(|e| format!("a step needs a title and a command: {e}"))?; @@ -270,7 +270,7 @@ pub fn commands() -> Vec { } t.steps.push(step); Ok(json!({"steps": t.steps.len()})) - }) + }).roles(&[]) .params(r#"{"task"?: name, "title": string, "instructions"?: string, "command": id, "params"?: {}}"#), // Editing a step's words, its place or the step itself. Its command and parameters stay // as they were, so a task from a file stays as untrusted as it came. @@ -295,7 +295,7 @@ pub fn commands() -> Vec { st.title = title; st.instructions = instructions; Ok(json!({"step": step})) - }) + }).roles(&[]) .params(r#"{"task"?: name, "step": number (from 0), "title"?: string, "instructions"?: string}"#), C::new("task.moveStep", "Move Task Step", "Tasks", |s, p| { let to = match p.get("to") { @@ -313,7 +313,7 @@ pub fn commands() -> Vec { let st = t.steps.remove(step); t.steps.insert(to, st); Ok(json!({"step": to})) - }) + }).roles(&[]) .params(r#"{"task"?: name, "step": number (from 0), "to": number (from 0)}"#), C::new("task.removeStep", "Delete Task Step", "Tasks", |s, p| { let step = step_index(p)?; @@ -323,9 +323,9 @@ pub fn commands() -> Vec { } t.steps.remove(step); Ok(json!({"steps": t.steps.len()})) - }) + }).roles(&[]) .params(r#"{"task"?: name, "step": number (from 0)}"#), - C::new("task.run", "Run Task", "Tasks", run_task).params(r#"{"task"?: name, "step"?: number (one step; all when omitted)}"#), + C::new("task.run", "Run Task", "Tasks", run_task).roles(&[]).params(r#"{"task"?: name, "step"?: number (one step; all when omitted)}"#), C::new("task.remove", "Delete Task", "Tasks", |s, p| { let name = ps(p, "task").ok_or("task is required")?; let before = s.project.tasks.len(); @@ -333,9 +333,9 @@ pub fn commands() -> Vec { // A later task by this name isn't the user's just because of its name. s.trusted_tasks.remove(&name.to_ascii_lowercase()); if s.project.tasks.len() == before { Err(format!("no task '{name}'")) } else { Ok(Value::Null) } - }) + }).roles(&[]) .params(r#"{"task": name}"#), - C::new("task.save", "Save Task Item", "Publish › Save As", |s, p| save_item(s, "task", p)) + C::new("task.save", "Save Task Item", "Publish › Save As", |s, p| save_item(s, "task", p)).roles(&[]) .view() .params(r#"{"task"?: name, "path": "*.rgistask"}"#), C::new("task.import", "Import Task", "Analyze › Tasks", |s, p| { @@ -347,10 +347,10 @@ pub fn commands() -> Vec { } let steps = serde_json::to_value(&t.steps).map_err(|e| e.to_string())?; s.run("task.new", &json!({"name": t.name, "description": t.description, "steps": steps, "imported": true})) - }) + }).roles(&[crate::ParamRole { key: "text", role: crate::Role::Document(crate::Document::Task) }]) .params(r#"{"path"?: "*.rgistask", "text"?: task JSON, "data"?: base64 file}"#), // 3D export. - C::new("share.export3d", "Export To 3D File", "Publish › Output", export_3d) + C::new("share.export3d", "Export To 3D File", "Publish › Output", export_3d).roles(&[]) .view() .when(need_feature_layer) .params(r#"{"layer"?, "path": "*.obj", "heightField"?: field (extrusion height in metres), "height"?: metres, "zScale"?: number}"#), @@ -576,7 +576,14 @@ fn carry_moves(old: &rustgis_map::symbology::Labeling, new: &mut rustgis_map::sy /// The project without its features (layer schemas, symbology, layouts and styles stay), and how /// many saved label moves went with them. -fn template(p: &Project) -> (Project, usize) { +struct TemplateLosses { + label_moves: usize, + offline_areas: usize, + archived_edits: usize, + source_references: usize, +} + +fn template(p: &Project) -> (Project, TemplateLosses) { let mut t = p.clone(); t.gp_history.clear(); // Saved label moves name features, and a template has none: the layers', the saved styles' @@ -594,11 +601,36 @@ fn template(p: &Project) -> (Project, usize) { c.moved.clear(); } } + let offline_areas = t.offline_areas.len(); + let archived_edits = t + .maps + .iter() + .chain(t.offline_areas.iter().map(|a| &a.data)) + .flat_map(|m| &m.layers) + .filter_map(|l| l.design.archive.as_ref()) + .fold(0usize, |count, archive| count.saturating_add(archive.len())); + t.offline_areas.clear(); + let source_references = t + .maps + .iter() + .flat_map(|m| &m.layers) + .fold(0usize, |count, l| { + count + .saturating_add(usize::from(l.dataset.source.is_some())) + .saturating_add(usize::from(l.source.is_some())) + .saturating_add(usize::from(l.postgis.is_some())) + .saturating_add(usize::from(l.gpkg.is_some())) + }) + .saturating_add(t.tables.iter().filter(|table| table.source.is_some()).count()); for m in &mut t.maps { for l in &mut m.layers { l.dataset.features.clear(); l.selection.clear(); l.dataset.source = None; + l.source = None; + if let Some(archive) = &mut l.design.archive { + archive.clear(); + } // A feature-less template has no loaded rows to save back to a source. l.postgis = None; l.gpkg = None; @@ -613,31 +645,30 @@ fn template(p: &Project) -> (Project, usize) { } for t in &mut t.tables { t.features.clear(); + t.source = None; } - (t, moves) + (t, TemplateLosses { label_moves: moves, offline_areas, archived_edits, source_references }) } /// Packages only completed imports; runtime download reservations cannot be reopened. /// Refuses JSON beyond the project or ZIP inflation limit before creating the archive. -fn package_project(s: &Session) -> Result, String> { +fn package_project(s: &mut Session) -> Result, String> { s.project.ensure_imports_complete()?; let mut p = s.project.clone(); let mut budget = MAX_EMBED; for m in &mut p.maps { *m = embedded_map(m, &mut budget); } - crate::project::strip_addresses(p.addresses_mut()); - // Recorded parameters (history, tasks, models) lose only their login bindings, as when saving. - crate::project::clear_login_bindings(p.recorded_mut()); - let text = serde_json::to_string(&p).map_err(|e| e.to_string())?; + let name = p.name.clone(); + let (text, report) = crate::durable::output_owned(p, |p| serde_json::to_string(p).map_err(|e| e.to_string()))?; + crate::durable::notify(s, &report); let limit = crate::project::MAX_BYTES.min(rustgis_data::zip::MAX_TOTAL as u64); if text.len() as u64 > limit { return Err(format!("the project is too large to package (maximum {} MiB)", limit >> 20)); } - let readme = - format!("{} — a RustGIS project package.\nOpen it with RustGIS (Project › Open), or unzip it for {}.rgis.\n", p.name, p.name); + let readme = format!("{} — a RustGIS project package.\nOpen it with RustGIS (Project › Open), or unzip it for {}.rgis.\n", name, name); rustgis_data::zip::write(&[ - (format!("{}.{}", safe(&p.name), crate::project::EXTENSION), text.into_bytes()), + (format!("{}.{}", safe(&name), crate::project::EXTENSION), text.into_bytes()), ("README.txt".into(), readme.into_bytes()), ]) } @@ -648,7 +679,7 @@ fn safe(name: &str) -> String { } /// A package of the active map; `mobile` drops hidden layers and simplifies geometry. -fn package_map(s: &Session, mobile: bool, tol: f64) -> Result, String> { +fn package_map(s: &mut Session, mobile: bool, tol: f64) -> Result, String> { let mut m = embedded_map(s.map(), &mut MAX_EMBED.clone()); if mobile { // Groups folded into their layers: a hidden group's layers are dropped with the others. @@ -675,28 +706,23 @@ fn package_map(s: &Session, mobile: bool, tol: f64) -> Result, String> { } } crate::project::ensure_map_imports_complete(&m)?; - crate::project::strip_addresses(m.addresses_mut()); - let text = serde_json::to_string(&m).map_err(|e| e.to_string())?; + let name = m.name.clone(); + let (text, report) = crate::durable::output_owned(m, |m| serde_json::to_string(m).map_err(|e| e.to_string()))?; + crate::durable::notify(s, &report); let kind = if mobile { "mobile map package" } else { "map package" }; - let readme = format!("{} — a RustGIS {kind}.\nAdd it to a project with Data › Import.\n", m.name); - rustgis_data::zip::write(&[(format!("{}.rgismap", safe(&m.name)), text.into_bytes()), ("README.txt".into(), readme.into_bytes())]) + let readme = format!("{} — a RustGIS {kind}.\nAdd it to a project with Data › Import.\n", name); + rustgis_data::zip::write(&[(format!("{}.rgismap", safe(&name)), text.into_bytes()), ("README.txt".into(), readme.into_bytes())]) } /// The geoprocessing history with the layers its runs made, as GeoJSON. -fn package_gp(s: &Session) -> Result, String> { +fn package_gp(s: &mut Session) -> Result, String> { if s.project.gp_history.is_empty() { return Err("no geoprocessing has been run yet".into()); } - let mut history = s.project.gp_history.clone(); - let mut addresses = Vec::new(); - for record in &mut history { - crate::project::recorded_addresses(&mut record.params, &mut addresses); - if let Some((_, params)) = &mut record.command { - crate::project::recorded_addresses(params, &mut addresses); - } - } - crate::project::clear_login_bindings(addresses); - let mut entries = vec![("history.json".to_string(), serde_json::to_vec_pretty(&history).map_err(|e| e.to_string())?)]; + let (history, report) = + crate::durable::output(&mut s.project.gp_history, None, |h| serde_json::to_vec_pretty(h).map_err(|e| e.to_string()))?; + crate::durable::notify(s, &report); + let mut entries = vec![("history.json".to_string(), history)]; let mut layers = std::collections::HashSet::new(); let mut files = std::collections::HashSet::new(); for rec in s.project.gp_history.iter().filter(|r| r.ok) { @@ -1098,14 +1124,11 @@ pub const ITEMS: &[(&str, &str, &str)] = &[ pub fn item_bytes(s: &mut Session, kind: &str, p: &Value) -> Result<(String, Vec), String> { let (name, bytes) = match kind { "map" => { - let map = match ps(p, "map") { - Some(n) => s.project.maps.iter().find(|m| m.name.eq_ignore_ascii_case(n)).ok_or_else(|| format!("no map '{n}'"))?, - None => s.map(), - }; + let map = saved_map(s, p)?; crate::project::ensure_map_imports_complete(map)?; - let mut map = embedded_map(map, &mut MAX_EMBED.clone()); - crate::project::strip_addresses(map.addresses_mut()); - (map.name.clone(), to_json(&map)?) + let map = embedded_map(map, &mut MAX_EMBED.clone()); + let name = map.name.clone(); + (name, durable_bytes(s, map)?) } "layer" => { let id = s.layer_id(p)?; @@ -1113,9 +1136,9 @@ pub fn item_bytes(s: &mut Session, kind: &str, p: &Value) -> Result<(String, Vec if l.dataset.source.as_deref().is_some_and(|source| source.starts_with("geolibre-pending:")) { return Err("this layer is still downloading; wait for it to finish before saving".into()); } - let mut layer = embed_layer(l, &mut MAX_EMBED.clone()); - crate::project::strip_addresses(layer.addresses_mut().collect()); - (layer.name.clone(), to_json(&layer)?) + let layer = embed_layer(l, &mut MAX_EMBED.clone()); + let name = layer.name.clone(); + (name, durable_bytes(s, layer)?) } "styles" => { if s.project.styles.is_empty() { @@ -1131,17 +1154,25 @@ pub fn item_bytes(s: &mut Session, kind: &str, p: &Value) -> Result<(String, Vec } "template" => { s.project.ensure_imports_complete()?; - let (mut t, moves) = template(&s.project); - if moves > 0 { - s.notify(format!("{moves} saved label move(s) left out of the template, which keeps no features"), false); - } - // As a package: what it reads keeps its plain address, what it records no binding. - crate::project::strip_addresses(t.addresses_mut()); - crate::project::clear_login_bindings(t.recorded_mut()); + let (mut t, TemplateLosses { label_moves, offline_areas, archived_edits, source_references }) = template(&s.project); if let Some(n) = ps(p, "name").map(str::trim).filter(|n| !n.is_empty()) { t.name = n.chars().take(200).collect(); } - (t.name.clone(), to_json(&t)?) + let name = t.name.clone(); + let bytes = durable_bytes(s, t)?; + if source_references > 0 { + s.notify(format!("{source_references} data source reference(s) left out of the template, which keeps no features"), false); + } + if label_moves > 0 { + s.notify(format!("{label_moves} saved label move(s) left out of the template, which keeps no features"), false); + } + if offline_areas > 0 { + s.notify(format!("{offline_areas} offline area(s) left out of the template, which keeps no features"), false); + } + if archived_edits > 0 { + s.notify(format!("{archived_edits} archived edit record(s) left out of the template, which keeps no features"), false); + } + (name, bytes) } "projectPackage" => (s.project.name.clone(), package_project(s)?), "mapPackage" | "mobilePackage" => { @@ -1151,13 +1182,11 @@ pub fn item_bytes(s: &mut Session, kind: &str, p: &Value) -> Result<(String, Vec "gpPackage" => ("results".to_string(), package_gp(s)?), "bookmarks" => (format!("{} bookmarks", s.map().name), super::navigate::bookmarks_file(s.map())?), "task" => { - let mut t = task_ref(s, p)?.clone(); - let mut addresses = Vec::new(); - for step in &mut t.steps { - crate::project::recorded_addresses(&mut step.params, &mut addresses); - } - crate::project::clear_login_bindings(addresses); - (t.name.clone(), to_json(&t)?) + let t = task_mut(s, p)?; + let name = t.name.clone(); + let (bytes, report) = crate::durable::output(t, None, to_json)?; + crate::durable::notify(s, &report); + (name, bytes) } "obj" => { let id = s.layer_id(p)?; @@ -1209,3 +1238,18 @@ pub(crate) fn save_item(s: &mut Session, kind: &str, p: &Value) -> CmdResult { rustgis_data::io::write_atomic(Path::new(&path), &bytes).map_err(|e| format!("{path}: {e}"))?; Ok(json!({"path": path, "bytes": bytes.len()})) } + +/// Consumes the existing item draft, emitting losses after serialization has completed. +fn durable_bytes(s: &mut Session, value: T) -> Result, String> { + let (bytes, report) = crate::durable::output_owned(value, to_json)?; + crate::durable::notify(s, &report); + Ok(bytes) +} + +/// Chooses the saved map with the executor's existing name/default rule. +fn saved_map<'a>(s: &'a Session, p: &Value) -> Result<&'a Map, String> { + match ps(p, "map") { + Some(n) => s.project.maps.iter().find(|m| m.name.eq_ignore_ascii_case(n)).ok_or_else(|| format!("no map '{n}'")), + None => Ok(s.map()), + } +} diff --git a/crates/engine/src/cmd/labels.rs b/crates/engine/src/cmd/labels.rs index e62d4801c..f46a1133d 100644 --- a/crates/engine/src/cmd/labels.rs +++ b/crates/engine/src/cmd/labels.rs @@ -93,21 +93,21 @@ pub fn commands() -> Vec { let d = &mut s.map_mut().label_display; d.locked = b(p, "on").unwrap_or(!d.locked); Ok(json!({"locked": d.locked})) - }) + }).roles(&[]) .view() .params(r#"{"on"?: bool}"#), C::new("labels.pause", "Pause", "Labeling › Map Labeling", |s, p| { let d = &mut s.map_mut().label_display; d.paused = b(p, "on").unwrap_or(!d.paused); Ok(json!({"paused": d.paused})) - }) + }).roles(&[]) .view() .params(r#"{"on"?: bool}"#), C::new("labels.viewUnplaced", "View Unplaced", "Labeling › Map Labeling", |s, p| { let d = &mut s.map_mut().label_display; d.show_unplaced = b(p, "on").unwrap_or(!d.show_unplaced); Ok(json!({"showUnplaced": d.show_unplaced})) - }) + }).roles(&[]) .view() .params(r#"{"on"?: bool}"#), C::new("labels.fonts", "Label Fonts", "View › Windows › Labeling", |_, p| { @@ -119,16 +119,16 @@ pub fn commands() -> Vec { let fonts: Vec<&str> = all.iter().filter(|f| f.listed && f.name.to_lowercase().contains(&search)).map(|f| f.name.as_str()).collect(); Ok(json!({"fonts": fonts})) - }) + }).roles(&[]) .view() .params(r#"{"search"?: text in the family name} → {"fonts": [the installed font families, by name], "indexing"?: true (still being indexed: none listed yet)}"#), - C::new("labels.convert", "Convert To Annotation", "Labeling › Map Labeling", convert) + C::new("labels.convert", "Convert To Annotation", "Labeling › Map Labeling", convert).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "name"?: string}"#), - C::new("labels.scaleRange", "Label Visibility Range", "Labeling › Label Range", scale_range) + C::new("labels.scaleRange", "Label Visibility Range", "Labeling › Label Range", scale_range).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class"?: name (none: the Default class, by whatever name it has), "min"?: scale|null (out beyond), "max"?: scale|null (in beyond), "clear"?: bool (clears the scale and distance limits, as Clear Limits does)}"#), - C::new("labels.distanceRange", "Label Distance Range", "Labeling › Label Range", distance_range) + C::new("labels.distanceRange", "Label Distance Range", "Labeling › Label Range", distance_range).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class"?: name (none: the Default class, by whatever name it has), "closest"?: metres|null, "farthest"?: metres|null, "clear"?: bool}"#), C::new("labels.placement", "Label Placement", "Labeling › Label Placement", |s, p| { @@ -226,7 +226,7 @@ pub fn commands() -> Vec { s.notify(note, false); } Ok(done) - }) + }).roles(&[]) .when(need_feature_layer) .params( r#"{"layer"?, "class"?: name (none: the Default class, by whatever name it has), "position"?: "auto"|"center"|"above"|"below"|"left"|"right"|"aboveRight"|"aboveLeft"|"belowRight"|"belowLeft"|"parallel" (along a line, straight)|"curved" (along a line, following its bends) (required without offset, repeat, otherPositions, showAll or callout), "offset"?: [right, down] points (-200 to 200 each) the label moves from where its position puts it ([0, 0]: none; a label along a line isn't moved), "repeat"?: points (10 to 10000) between a line's labels, spaced out from the one at its middle, at most 32 a line (0 or null: the one at its middle; parallel and curved only), "otherPositions"?: bool (a label whose place is taken tries others: a point's round the point, corners first; a line's along the line, nearest its middle first, placed parallel or curved without a repeat; true is refused for any other line label and on polygons), "showAll"?: bool (every label drawn, over the others where they crowd), "callout"?: true|false|points (5 to 500: a label crowded off its spot moves to a free one nearby, a line drawn back to its feature at most this long; true keeps its length, or 40; false, 0 or null: none; not labels along a line or with an icon)}"#, @@ -291,7 +291,7 @@ pub fn commands() -> Vec { // Locked, the labels are locked again where they are placed next (`with_labels`: the // moves changed), as the one moved may not have been placed when the lock was taken. Ok(r) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class"?: name (none: the Default class), "feature": the feature's id, "to": [x, y] (longitude, latitude: where the label's text is centred) | null (back where its class puts it)}"#), C::new("labels.resetMoved", "Reset Moved", "Labeling › Label Placement", |s, p| { @@ -311,12 +311,12 @@ pub fn commands() -> Vec { // As after a move, those put back are locked again where they are placed next; with // nothing moved, nothing changes and the lock holds (`with_labels`). Ok(r) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class"?: name (none: the Default class), "all"?: bool (every class of the layer)}"#), // Label classes: each labels every feature its clause takes, as in ArcGIS Pro; the classes // listed first, Default last, are placed first (`Labeling`, #732). - C::new("labels.class.add", "Add Label Class", "View › Windows › Labeling", add_class) + C::new("labels.class.add", "Add Label Class", "View › Windows › Labeling", add_class).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "name": string, "where"?: sql (a definition query's language; none takes every feature), "from"?: class (whose field and text symbol it starts with; none: the Default class)}"#), C::new("labels.class.remove", "Remove Label Class", "View › Windows › Labeling", |s, p| { @@ -326,7 +326,7 @@ pub fn commands() -> Vec { labels.classes.remove(i); Ok(()) }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class": name (not Default)}"#), // Every class but Default; none to remove is no error, so a script can start with it. @@ -338,7 +338,7 @@ pub fn commands() -> Vec { Ok(()) })?; Ok(json!({"removed": removed})) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?}"#), C::new("labels.class.rename", "Rename Label Class", "View › Windows › Labeling", |s, p| { @@ -349,7 +349,7 @@ pub fn commands() -> Vec { labels.class_mut(&class).ok_or_else(|| format!("no label class named '{class}'"))?.name = name; Ok(()) }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class": name, "name": string}"#), C::new("labels.class.update", "Update Label Class", "View › Windows › Labeling", |s, p| { @@ -364,7 +364,7 @@ pub fn commands() -> Vec { c.visible = v; } }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class"?: name (none: the Default class, by whatever name it has), "where"?: sql|null, "visible"?: bool}"#), C::new("labels.class.move", "Move Label Class", "View › Windows › Labeling", |s, p| { @@ -377,10 +377,10 @@ pub fn commands() -> Vec { labels.classes.insert(to, c); Ok(()) }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class": name (not Default), "to": number (its place among the classes listed before Default, 0 first: placed first when labels compete for room)}"#), - C::new("labels.by", "Labels by Expression", "View › Windows › Labeling", labels_by) + C::new("labels.by", "Labels by Expression", "View › Windows › Labeling", labels_by).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "class"?: name (none: the Default class, by whatever name it has), "text"?: expression|null (each label's text, in place of the class's field; a feature it gives null or blanks for isn't labelled), "size"?: expression|null (the text's size in points, 4 to 72), "color"?: expression|null (the text's colour: '#RRGGBB', a CSS colour name or 'r,g,b[,a]' as color_rgb gives it), "angle"?: expression|null (a point's or polygon's label turned by it, in degrees clockwise; labels along lines keep the line's)} (each expression in the language of queries, per feature, @map_scale the map's scale; a key left out keeps its expression, null or "" removes it; where a size, color or angle expression gives null the class's own setting is used)"#), ] diff --git a/crates/engine/src/cmd/layer.rs b/crates/engine/src/cmd/layer.rs index 881206836..8a90c18c1 100644 --- a/crates/engine/src/cmd/layer.rs +++ b/crates/engine/src/cmd/layer.rs @@ -15,45 +15,45 @@ use crate::{CmdResult, CommandSpec as C, Session}; /// Registers layer commands shared by the UI, CLI and command search. pub fn commands() -> Vec { vec![ - C::new("layer.addData", "Add Data", "Home › Add", add_data) + C::new("layer.addData", "Add Data", "Home › Add", add_data).roles(&[]) .params(r#"{"path": string, "name"?: string, "layer"?: "GeoPackage, GDAL, DuckDB or DXF layer", "paperSpace"?: "bool: a DXF's layout entities too", "stream"?: bool, "as"?: "scan" (a TIFF, PNG or JPEG, even a photo or a GeoTIFF, as a picture with no coordinates), "crs"?: "EPSG:… (or WKT) for Shapefile coordinates (overrides .prj), a CSV, a CityJSON model, or a GDAL or DuckDB file that doesn't say", "x"?: "CSV x/longitude column", "y"?: "CSV y/latitude column", "wkt"?: "CSV WKT geometry column", "delimiter"?: ","|";"|"\\t"|"|"}"#), - C::new("layer.addPhotos", "Add Photos", "Home › Add", add_photos).params( + C::new("layer.addPhotos", "Add Photos", "Home › Add", add_photos).roles(&[]).params( r#"{"paths"?: [JPEG photo files], "folder"?: "a folder of photos (every JPEG in it)", "name"?: "the layer (default Photos; photos join it when it exists)"}"#, ), - C::new("info.fileLayers", "List Layers in File", "Home › Add › Add Data", file_layers) + C::new("info.fileLayers", "List Layers in File", "Home › Add › Add Data", file_layers).roles(&[]) .view() .params(r#"{"path": string, "crs"?: "EPSG:… for a .city.json that doesn't say (as layer.addData)"}"#), - C::new("info.fileColumns", "List Columns in File", "Home › Add › Add Data › Delimited Text", file_columns).view().params( + C::new("info.fileColumns", "List Columns in File", "Home › Add › Add Data › Delimited Text", file_columns).roles(&[]).view().params( r#"{"path": "a delimited text file (CSV, TSV, TXT) or a spreadsheet", "delimiter"?: ","|";"|"\\t"|"|", "layer"?: "a spreadsheet's sheet (the first by default)"}"#, ), - C::new("layer.addSql", "Add Data From SQL", "Home › Add", add_sql).params( + C::new("layer.addSql", "Add Data From SQL", "Home › Add", add_sql).roles(&[]).params( r#"{"sql": "one DuckDB SELECT; DuckDB Spatial's ST_ functions, read_parquet('…'), web addresses", "name"?: string, "database"?: "a .duckdb file or a PostgreSQL connection (postgres:host=… dbname=…) whose tables the query names", "password"?: "the connection's, kept for this session only", "crs"?: "EPSG:… of a geometry that doesn't say", "stream"?: bool, "ran"?: "a result the app ran (the SQL pane)"}"#, ), - C::new("layer.addDatabase", "Add Data From Database", "Home › Add › Add Data", add_database).params( + C::new("layer.addDatabase", "Add Data From Database", "Home › Add › Add Data", add_database).roles(&[]).params( r#"{"database": "postgres:host=… dbname=… user=… (or a postgresql:// address; PostGIS tables are read live), a mysql://user@host:port/database address (MySQL or MariaDB, read live; mariadb:// too), an mssql://user@host:1433/database address (SQL Server or Azure SQL, read live; ?encrypt=strict|false, ?trust-server-certificate=true for a self-signed certificate, ?instance=NAME; on Windows DOMAIN\\user or no user signs in with Windows), or a .duckdb file", "password"?: "kept for this session only, never saved", "layer"?: "a table, as table or schema.table (table:column for one of several geometry columns; default: every table with a geometry column)", "crs"?: "EPSG:… of a geometry that doesn't say", "name"?: string, "stream"?: bool}"#, ), - C::new("info.databaseTables", "List Database Tables", "Home › Add › Add Data › Database", database_tables) + C::new("info.databaseTables", "List Database Tables", "Home › Add › Add Data › Database", database_tables).roles(&[]) .view() .params(r#"{"database": "as layer.addDatabase", "password"?: string}"#), - C::new("layer.defineProjection", "Define Projection", "Layers", define_projection) + C::new("layer.defineProjection", "Define Projection", "Layers", define_projection).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?: name|id, "crs": "EPSG:… or WKT; reinterpret native coordinates, without rewriting the source file"}"#), - C::new("layer.loadAll", "Load All Features", "Layers", load_all).when(need_streamed).params(r#"{"layer"?: name|id}"#), - C::new("layer.addUrl", "Add Data From URL", "Home › Add › Add Data", add_url).params( + C::new("layer.loadAll", "Load All Features", "Layers", load_all).roles(&[]).when(need_streamed).params(r#"{"layer"?: name|id}"#), + C::new("layer.addUrl", "Add Data From URL", "Home › Add › Add Data", add_url).roles(&[]).params( r#"{"url": "https://…/data.geojson|.csv|.parquet|.fgb|.gpkg|.zip|.shp (its .dbf fetched beside it, and its .shx, .prj and .cpg when there)|.kml|.kmz|.gpx|.tif|.jp2 (JPEG 2000: desktop, through GDAL) (the format from the address, else the contents) | a .pmtiles archive", "name"?: text, "layer"?: "one layer of a zip, KML or GPX", "zoom"?: "bool: true frames everything added; false keeps the view, even on an empty map; omitted, the first content on an empty map is framed (a GeoParquet, FlatGeobuf or cloud-optimised GeoTIFF read in place likewise; a PMTiles or MBTiles archive ignores zoom and is framed on its bounds unless it covers the whole world)", "x"?, "y"?, "wkt"?, "crs"?, "delimiter"?, "paperSpace"?, "stream"?, "as"? (as layer.addData), "whole"?: "bool: download a cloud-optimised GeoTIFF whole (as a raster layer) instead of reading it in place (an address without a file name is read in place when its first bytes are a cloud-optimised GeoTIFF)", "cog"?: "bool: the address is a cloud-optimised GeoTIFF whatever its name says (the app's probe found it so)", "bbox"?: "[west, south, east, north] in degrees | "view": a cloud-optimised GeoTIFF's window as a raster layer (desktop and command line), read by range at full resolution, in the file's coordinate system, with its colour table and class names; at most 64 million values (all bands)", "cellSize"?: "with bbox: the coarsest overview whose cells are at most this size is read (the file's units; metres for longitude/latitude, a degree taken as 111.32 km)", "cube"?: "bool: a data cube's file: a tile layer that follows the time slider (stac.addCube adds them), kept one when the project opens again", "start"?, "end"?, "unit"?, "interval"?, "dates"?: "a GeoTIFF address with date placeholders ({date:YYYY}, {YYYY}{MM}{DD}), one file per date: the dates it spans and steps through (as time.set), turning the time slider on", "connection"?: "the id of the saved web-service connection it comes from (as service.add)", "anonymous"?: bool (internal: a plugin-origin download must never adopt a saved login), "request"?: "number (internal: the download a plugin command left to the app, #1202)"}"#, ), - C::new("layer.addDownload", "Add Downloaded Data", "Home › Add", add_download) + C::new("layer.addDownload", "Add Downloaded Data", "Home › Add", add_download).roles(&[]) .params(r#"{"id": a download the app finished (Home › Add › Add Data › From URL), "url": its address, "name"?: text, "layer"?: text, "zoom"?: bool, "x"?, "y"?, "wkt"?, "crs"?, "delimiter"?, "paperSpace"?, "stream"?, "as"? (as layer.addData), "service"?: bool (a service's features: their axes (when they came as GeoJSON) and paging are checked, and the address is kept as the layer's source), "bounds"?: [west, south, east, north] (with service: the layer's longitude/latitude bounds from its capabilities, which its axes are checked against; without bounds they are left as they come; ignored without service), "coverage"?: bool (a WCS coverage: a GeoTIFF, else the service's error, and its request kept as the layer's source), "geolibreProject"?: bool (internal: downloaded project JSON), "geolibreLayer"?: {"token", "record"} (internal: reserved imported layer and its saved style), "geolibreZarr"?: bool (internal: an imported Zarr slice as GeoTIFF), "connection"?: "as layer.addUrl", "anonymous"?: bool (internal: preserves a plugin-origin download's anonymous reads), "request"?: number (internal: the download a plugin command left to the app, #1202)}"#), - C::new("layer.addEncodedPolyline", "Add Encoded Polyline", "Home › Add", add_encoded_polyline).params( + C::new("layer.addEncodedPolyline", "Add Encoded Polyline", "Home › Add", add_encoded_polyline).roles(&[]).params( r#"{"polyline": "encoded polyline text" | [texts] (one line each), "precision"?: 5 (Google) | 6 (OSRM, Valhalla), "name"?: text}"#, ), - C::new("layer.addDataText", "Add Data (from text)", "Home › Add", add_data_text).params(r#"{"fileName": "name.geojson|.csv|.tsv", "text": string, "name"?: text, "x"?, "y"?, "wkt"?, "crs"?, "delimiter"? (as layer.addData, for delimited text), "layer"?, "paperSpace"? (as layer.addData, for a file of several layers)}"#), - C::new("layer.addSample", "Add Sample Data", "Home › Add › Add Data", add_sample).params(r#"{"name": "countries"|"cities"|"rivers"|"lakes"}"#), - C::new("layer.remove", "Remove", "Layers", remove) + C::new("layer.addDataText", "Add Data (from text)", "Home › Add", add_data_text).roles(&[]).params(r#"{"fileName": "name.geojson|.csv|.tsv", "text": string, "name"?: text, "x"?, "y"?, "wkt"?, "crs"?, "delimiter"? (as layer.addData, for delimited text), "layer"?, "paperSpace"? (as layer.addData, for a file of several layers)}"#), + C::new("layer.addSample", "Add Sample Data", "Home › Add › Add Data", add_sample).roles(&[]).params(r#"{"name": "countries"|"cities"|"rivers"|"lakes"}"#), + C::new("layer.remove", "Remove", "Layers", remove).roles(&[]) .key("Delete") .when(need_layer) .params(r#"{"layer"?: name|id | [name|id] (default: the layer selection)}"#), - C::new("layer.select", "Select Layer", "Layers", select) + C::new("layer.select", "Select Layer", "Layers", select).roles(&[]) .params(r#"{"layer"?: name|id | [name|id] (default: the active layer; the last of a list becomes the active layer), "mode"?: "replace" (only these; the default) | "toggle" (each in or out of the layer selection; the last one out leaves no active layer) | "extend" (added to it)}"#) .view(), C::new("layer.rename", "Rename", "Layers", |s, p| { @@ -102,7 +102,7 @@ pub fn commands() -> Vec { // So do legends that leave it out. legends_follow_layer(s, &old, Some(&n)); Ok(r) - }).params(r#"{"layer"?, "name": string}"#) + }).roles(&[]).params(r#"{"layer"?, "name": string}"#) .when(need_layer), C::new("layer.visibility", "Visibility", "Layers", |s, p| { let ids = layer_ids(s, p)?; @@ -118,11 +118,11 @@ pub fn commands() -> Vec { } } Ok(Value::Null) - }).params(r#"{"layer"?: name|id | [name|id] (default: the layer selection), "visible"?: bool (toggles when omitted: all take the opposite of the active layer's state)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id | [name|id] (default: the layer selection), "visible"?: bool (toggles when omitted: all take the opposite of the active layer's state)}"#) .when(need_layer), - C::new("layer.moveUp", "Move Up", "Layers", |s, p| move_layer(s, p, -1)).params(r#"{"layer"?: name|id (default: the active layer)}"#).when(need_layer), - C::new("layer.moveDown", "Move Down", "Layers", |s, p| move_layer(s, p, 1)).params(r#"{"layer"?: name|id (default: the active layer)}"#).when(need_layer), - C::new("layer.move", "Move", "Layers", move_to) + C::new("layer.moveUp", "Move Up", "Layers", |s, p| move_layer(s, p, -1)).roles(&[]).params(r#"{"layer"?: name|id (default: the active layer)}"#).when(need_layer), + C::new("layer.moveDown", "Move Down", "Layers", |s, p| move_layer(s, p, 1)).roles(&[]).params(r#"{"layer"?: name|id (default: the active layer)}"#).when(need_layer), + C::new("layer.move", "Move", "Layers", move_to).roles(&[]) .when(need_layer) .params(r#"{"layer"?: name|id | [name|id] (default: the layer selection), "to": position (0 = top; past the end, the bottom). Several keep their order, the topmost landing at "to" as one layer would, and go only to the top or the bottom (one in a group, to its group's): with 4 layers and 2 of them, 2 or more is the bottom, 1 is refused}"#), C::new("layer.zoomTo", "Zoom To Layer", "Layers", |s, p| { @@ -133,10 +133,10 @@ pub fn commands() -> Vec { } s.zoom_to(&e); Ok(Value::Null) - }).params(r#"{"layer"?: name|id | [name|id] (default: the layer selection; zooms to all of them)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id | [name|id] (default: the layer selection; zooms to all of them)}"#) .view() .when(need_layer), - C::new("layer.symbology", "Symbology", "Style › Drawing", symbology) + C::new("layer.symbology", "Symbology", "Style › Drawing", symbology).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "type": "single"|"unique"|"graduated"|"proportional", "field"?, "expression"?: text|null (unique: class by this expression's values instead of a field; blank or null: by a field; with neither key, Unique Values keep the layer's expression), "classes"?: 1..12, "method"?: "naturalBreaks"|"equalInterval"|"quantile"|"geometricInterval"|"standardDeviation"|"arithmeticProgression"|"headTailBreaks"|"manual", "breaks"?: [numbers] (manual: the classes' upper bounds, closed by the field's largest value), "ramp"?: string}"#), C::new("layer.ramp", "Color Scheme", "Style › Drawing", |s, p| { @@ -152,9 +152,9 @@ pub fn commands() -> Vec { } } }) - }).params(r#"{"layer"?, "ramp": colour ramp name}"#) + }).roles(&[]).params(r#"{"layer"?, "ramp": colour ramp name}"#) .when(need_layer), - C::new("layer.classStyle", "Class Style", "Style › Drawing", class_style) + C::new("layer.classStyle", "Class Style", "Style › Drawing", class_style).roles(&[]) .when(need_feature_layer) .params( r##"{"layer"?, "value"?: text|number (a Unique Values class), "index"?: number (a Graduated Colors class, 0 = the lowest), "other"?: true (Unique Values' ), "color"?: "#RRGGBB" (preserves polygon translucency, minimum alpha 150)|"#RRGGBBAA" (sets alpha exactly), "outline"?: "#RRGGBB"|"#RRGGBBAA", "label"?: text (the legend label; "" goes back to the value or range)}"##, @@ -185,7 +185,7 @@ pub fn commands() -> Vec { // A fill colour after No Fill is drawn, unless it is transparent itself (#1301). take_no_fill(l); }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r##"{"layer"?, "color": "#RRGGBB"|"#RRGGBBAA" (the fill; a polygon's outline is left as it is, set it with layer.outline)}"##), C::new("layer.noFill", "No Fill", "Style › Drawing", |s, p| { @@ -209,7 +209,7 @@ pub fn commands() -> Vec { l.no_fill = Some(NoFill::Symbols); } }) - }) + }).roles(&[]) .when(need_polygon_features) .params(r#"{"layer"?: name|id (default: the active layer containing polygons; hides polygon fills, keeping outlines and point/line symbols)}"#), C::new("layer.outline", "Outline", "Style › Drawing", |s, p| { @@ -225,15 +225,15 @@ pub fn commands() -> Vec { } }) }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r##"{"layer"?, "color"?: "#RRGGBB"|"#RRGGBBAA", "width"?: number}"##), - C::new("layer.lineStrokes", "Line Strokes", "Style › Drawing", line_strokes) + C::new("layer.lineStrokes", "Line Strokes", "Style › Drawing", line_strokes).roles(&[]) .when(need_lines_or_points) .params( r##"{"layer"?, "strokes": [{"color"?: "#RRGGBB"|"#RRGGBBAA"|null (null: the renderer's line colour), "width"?: points|null (null: the renderer's width), "offset"?: points (positive: right of the line's direction, negative: left), "cap"?: "round"|"flat"|"square", "join"?: "round"|"miter"|"bevel", "level"?: 0..255|null (with layer.strokeLevels: the level it draws at; null: its place in the stack), "dash"?: [points, …]|null (dash, gap, … in turn, at most 8; null: solid), "markers"?: {"mark"?: "circle"|"square"|"triangle"|"diamond"|"star"|"character"|"hash" (a mark across the line), "char"?: text (a character of a font as the marker instead of a shape), "font"?: text (the installed family it is drawn from; none: the label font), "size"?: points (a marker's size, a hash mark's length; default 4), "at"?: "interval"|"vertices"|"innerVertices"|"first"|"last"|"middle"|"segmentMiddles" or a list of them (default "interval"), "interval"?: points (default 12), "offsetAlong"?: points (from the start, back from the end, on from the middle), "rotate"?: bool (turned with the line; default true), "averageAngle"?: points (the stretch of line a marker at an interval turns with; 0: its segment), "angle"?: degrees clockwise (a hash mark at 0 crosses the line square), "outline"?: "#RRGGBB"|"#RRGGBBAA" (the markers' outline, as wide as the stroke)}|null (markers or hash marks along the line in the stroke's colour, instead of a line; cap, join and dash don't apply)}] (drawn bottom first; [] goes back to the single stroke, or for a class to the layer's strokes; on a point layer they are its marks: each stroke's markers drawn at the point in place of the symbol's, "offsetAlong" shifting a mark right and "offset" down, both turned with its "angle", and "rotate" turning it by layer.markerRotation's field), "value"?: text|number (a Unique Values class's own strokes), "index"?: number (a Graduated Colors class's, 0 = the lowest), "other"?: true (Unique Values' )}"##, ), - C::new("layer.fillLayers", "Fill Layers", "Style › Drawing", fill_layers) + C::new("layer.fillLayers", "Fill Layers", "Style › Drawing", fill_layers).roles(&[]) .when(need_polygon_features) .params( r##"{"layer"?, "value"?|"index"?|"other"?: a class of Unique Values (its value, or other: true) or Graduated Colors (its index) to set its own fills (none: the layer's), "fills": [{"color"?: "#RRGGBB"|"#RRGGBBAA"|null (null: the feature's fill colour, so the fills follow the classes), "pattern"?: "solid" (default)|"hatch" (parallel lines)|"dots" (markers on a grid)|"gradient" (from color to color2 across the feature)|"shapeburst" (from color at the edge to color2 inside), "angle"?: degrees anticlockwise from east (hatch), "spacing"?: points between lines (hatch, 1 to 500; default 8) or points across and down between markers, a number or [across, down] (dots), "width"?: a hatch line's width in points (0 to 50; default 1), "offset"?: points across the lines (hatch) or [across, down] (dots), "marker"?: "circle"|"square"|"triangle"|"diamond"|"star" (dots), "char"?: text (dots: a character of a font as the marker instead of a shape), "font"?: text (the installed family it is drawn from; none: the label font), "size"?: a marker's size in points (default 3), "outline"?: a marker's outline colour or null, "outlineWidth"?: points, "displacement"?: [across, down] points every other row and column are shifted by (dots), "color2"?: "#RRGGBB"|"#RRGGBBAA" (gradient, shapeburst: the second colour; default white), "stops"?: [[0..1, "#RRGGBB"|"#RRGGBBAA"], …] (gradient, shapeburst: colours between the two, at most 16), "discrete"?: bool (each colour held until the next stop), "shape"?: "linear" (default)|"radial"|"conical" (gradient), "from"?: [across, down]|"centroid" (gradient: where color is, 0..1 in the feature's box; default [0.5, 0], the top middle), "to"?: [across, down]|"centroid" (where color2 is; default [0.5, 1]), "spread"?: "pad" (default)|"reflect"|"repeat" (gradient: past its points), "angle"?: degrees anticlockwise (gradient: turned round its box's middle), "inView"?: bool (gradient: from and to in the map view's box, not the feature's), "distance"?: points|null (shapeburst: how deep it shades; null or none: to the point deepest inside), "blur"?: 0..17 (shapeburst: pixels), "ignoreRings"?: bool (shapeburst: holes' edges left out)}] (bottom first, at most 8; [] for the symbol's single fill; the outline is drawn over them)}"##, @@ -263,10 +263,10 @@ pub fn commands() -> Vec { } l.marker_rotation = field.map(|field| rustgis_map::MarkerRotation { field, counterclockwise }); Ok(Value::Null) - }) + }).roles(&[]) .when(need_points) .params(r#"{"layer"?, "field": numeric field name|null (each point's marker turned by the angle in it, in degrees; null: not turned), "counterclockwise"?: bool (the angles run counterclockwise, as in mathematics; default clockwise, as on a compass)}"#), - C::new("layer.symbolBy", "Symbol by Expression", "Style › Drawing", symbol_by) + C::new("layer.symbolBy", "Symbol by Expression", "Style › Drawing", symbol_by).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "size"?: expression|null (a point's marker size in points), "fill"?: expression|null (the fill colour: '#RRGGBB', a CSS colour name or 'r,g,b[,a]' as color_rgb gives it), "stroke"?: expression|null (the outline colour, a line's colour), "strokeWidth"?: expression|null (the outline's width in points, a line's width), "angle"?: expression|null (point layers: each marker turned by it, in degrees clockwise, in place of layer.markerRotation's field)} (each expression in the language of queries, per feature, @map_scale the map's scale; a key left out keeps its expression, null or "" removes it; where an expression gives null the symbol's own value is drawn)"#), C::new("layer.strokeLevels", "Stroke Levels", "Style › Drawing", |s, p| { @@ -278,7 +278,7 @@ pub fn commands() -> Vec { } l.stroke_levels = on; Ok(json!({"on": on})) - }) + }).roles(&[]) .when(need_lines) .params(r#"{"layer"?, "on": bool (draw every feature's stacked strokes of one level before the next, so casings join under the cores)}"#), C::new("layer.marker", "Marker", "Style › Drawing", |s, p| { @@ -308,13 +308,13 @@ pub fn commands() -> Vec { } }) }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "shape"?: "circle"|"square"|"triangle"|"diamond"|"star", "char"?: text (a character of a font as the marker, its first; sized by the font's em), "font"?: text (the installed family the character is drawn from; "" or one not installed: the label font), "size"?: number}"#), C::new("layer.transparency", "Transparency", "Style › Effects", |s, p| { let t = f(p, "value").ok_or("value (0–100) is required")?.clamp(0.0, 100.0) as f32; with(s, p, |l| l.transparency = t) - }) + }).roles(&[]) .when(need_layer) .params(r#"{"layer"?, "value": 0..100}"#), C::new("layer.scaleRange", "Visibility Range", "Style › Visibility Range", |s, p| { @@ -324,7 +324,7 @@ pub fn commands() -> Vec { l.min_scale = min; l.max_scale = max; }) - }) + }).roles(&[]) .when(need_layer) .params(r#"{"layer"?, "min"?: scale (out beyond), "max"?: scale (in beyond)}"#), C::new("layer.referenceScale", "Reference Scale", "Style › Drawing", |s, p| { @@ -347,7 +347,7 @@ pub fn commands() -> Vec { } l.reference_scale = scale; Ok(json!({"scale": scale})) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "scale": 1:N the symbols' and labels' sizes are set for (drawn at another scale they grow and shrink with the map)|"current" (the map's scale)|null (sizes stay as set on screen)}"#), C::new("layer.selectable", "Selectable", "Layers", |s, p| { @@ -362,18 +362,18 @@ pub fn commands() -> Vec { } } Ok(Value::Null) - }).params(r#"{"layer"?: name|id | [name|id] (default: the layer selection), "on"?: bool (toggles when omitted: all take the opposite of the active layer's state)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id | [name|id] (default: the layer selection), "on"?: bool (toggles when omitted: all take the opposite of the active layer's state)}"#) .when(need_feature_layer), C::new("layer.expand", "Expand Legend", "Layers", |s, p| { let on = b(p, "on"); with(s, p, |l| l.expanded = on.unwrap_or(!l.expanded)) - }).params(r#"{"layer"?, "on"?: bool (toggles when omitted)}"#) + }).roles(&[]).params(r#"{"layer"?, "on"?: bool (toggles when omitted)}"#) .view() .when(need_layer), C::new("labels.enable", "Label", "Labeling › Labels", |s, p| { let on = b(p, "on"); with(s, p, |l| l.labels.enabled = on.unwrap_or(!l.labels.enabled)) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "on"?: bool}"#), C::new("labels.field", "Label Field", "Labeling › Label Class", |s, p| { @@ -386,7 +386,7 @@ pub fn commands() -> Vec { labels.enabled = true; Ok(()) }) - }).params(r#"{"layer"?, "class"?: name (none: the Default class, by whatever name it has), "field": name}"#) + }).roles(&[]).params(r#"{"layer"?, "class"?: name (none: the Default class, by whatever name it has), "field": name}"#) .when(need_feature_layer), C::new("labels.style", "Text Symbol", "Labeling › Text Symbol", |s, p| { let size = f(p, "size").map(|v| v.clamp(*LABEL_SIZE.start(), *LABEL_SIZE.end()) as f32); @@ -488,7 +488,7 @@ pub fn commands() -> Vec { c.italic = it; } }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r##"{"layer"?, "class"?: name (none: the Default class, by whatever name it has), "size"?: number, "color"?: "#RRGGBB"|"#RRGGBBAA", "halo"?: bool, "haloColor"?: "#RRGGBB"|"#RRGGBBAA"|"" (""/null: white or dark, whichever stands out from the text), "haloWidth"?: points 0-10|null (null: the thin halo), "spacing"?: points between letters (-5 to 50), "bold"?: bool, "italic"?: bool, "font"?: installed family name (labels.fonts lists them; "" is the label font), "background"?: false|null (none) | true (white, a thin dark outline) | {"fill"?: "#RRGGBB"|"#RRGGBBAA"|"" ("": none), "stroke"?: colour|"" (the outline; "": none), "strokeWidth"?: points 0-10, "padding"?: [across, up and down] points 0-50, "radius"?: corner radius in points 0-50} (a rectangle behind the text; what isn't given is kept, or the default's), "wrapChar"?: text (a line break wherever it occurs, left out; a space puts each word on its own line; "" for none), "wrapLength"?: characters (a longest line, broken between words; 0 for none; at most 500), "lineAlign"?: "left"|"center"|"right" (how a label's lines line up)}"##), C::new("layer.definitionQuery", "Definition Query", "Data › Definition Query", |s, p| { @@ -519,20 +519,20 @@ pub fn commands() -> Vec { ); } with(s, p, |l| l.definition_query = if q.is_empty() { None } else { Some(q.clone()) }) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "where": "SQL clause, empty to clear"}"#), C::new("table.open", "Attribute Table", "Data › Table", |s, p| { s.table_layer = Some(s.layer_id(p)?); Ok(Value::Null) - }).params(r#"{"layer"?: name|id (default: the active layer)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id (default: the active layer)}"#) .view() .key("Ctrl+T") .when(need_feature_layer), C::new("table.close", "Close Table", "Data › Table", |s, _| { s.table_layer = None; Ok(Value::Null) - }) + }).roles(&[]) .view(), C::new("table.remove", "Remove Table", "Data › Table", |s, p| { // The position this command took before tables had ids: refused, rather than @@ -543,13 +543,13 @@ pub fn commands() -> Vec { let i = s.table_index(p)?; s.project.tables.remove(i); Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"table": name|id (a standalone table; of tables sharing a name, the name is the first)} (the old position, "at", is refused)"#), C::new("layer.fields", "Fields", "Data › Data Design", |s, p| { let id = s.layer_id(p)?; let l = s.map().layer(id).ok_or("no layer")?; Ok(json!(l.dataset.fields.iter().map(|f| json!({"name": f.name, "type": f.ty.label(), "alias": f.alias})).collect::>())) - }).params(r#"{"layer"?: name|id (default: the active layer)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id (default: the active layer)}"#) .view() .when(need_feature_layer), ] @@ -3631,16 +3631,19 @@ pub(crate) fn add_cog_with(s: &mut Session, p: &Value, url: &str, strict: bool, // A plugin's command in the app doesn't wait (it runs on the frame), as in the browser. let probed = probed && !s.leave_downloads; // A dated address is framed on the file of the date it shows; why its file couldn't be read. - let (url, probed, mut why) = match r.get("url").and_then(Value::as_str) { - Some(dated) => { - // Its header read now (natively) as an undated address's is before it is added. - #[cfg(not(target_arch = "wasm32"))] - let probe = if s.leave_downloads { Ok(false) } else { crate::render::rustgis_tiles::cog::probe_blocking(dated) }; - #[cfg(target_arch = "wasm32")] - let probe: Result = Ok(false); - (dated.to_string(), probe == Ok(true), probe.err()) - } - None => (url.to_string(), probed, None), + let (url, probed, mut why) = if r.get("dated").and_then(Value::as_bool) == Some(true) { + // The public answer is redacted; metadata uses the installed tile's exact address. + let index = + r.get("index").and_then(Value::as_u64).and_then(|i| usize::try_from(i).ok()).ok_or("added GeoTIFF has no tile index")?; + let dated = s.map().tiles.get(index).ok_or("added GeoTIFF tile is missing")?.url.clone(); + // Its header read now (natively) as an undated address's is before it is added. + #[cfg(not(target_arch = "wasm32"))] + let probe = if s.leave_downloads { Ok(false) } else { crate::render::rustgis_tiles::cog::probe_blocking(&dated) }; + #[cfg(target_arch = "wasm32")] + let probe: Result = Ok(false); + (dated, probe == Ok(true), probe.err()) + } else { + (url.to_string(), probed, None) }; #[cfg(not(target_arch = "wasm32"))] let info = if probed { @@ -3689,7 +3692,7 @@ pub(crate) fn add_cog_with(s: &mut Session, p: &Value, url: &str, strict: bool, crate::sources::redact(&url), crate::sources::redact_text(&e) ), - None => format!("{url} couldn't be read yet, so the map wasn't moved to it"), + None => format!("{} couldn't be read yet, so the map wasn't moved to it", crate::sources::redact(&url)), }); } r["inPlace"] = json!(true); diff --git a/crates/engine/src/cmd/layer_copy.rs b/crates/engine/src/cmd/layer_copy.rs index c05c8a58d..e46ce3cc1 100644 --- a/crates/engine/src/cmd/layer_copy.rs +++ b/crates/engine/src/cmd/layer_copy.rs @@ -28,10 +28,10 @@ pub struct LayerClip { pub fn commands() -> Vec { vec![ - C::new("layer.copy", "Copy Layers", "Layers", copy).view().params( + C::new("layer.copy", "Copy Layers", "Layers", copy).roles(&[]).view().params( r#"{"layer"?: name|id | [name|id, …] (default: the active layer, unless a group is given), "group"?: name|id | [name|id, …] (the group with all its layers and subgroups)}"#, ), - C::new("layer.paste", "Paste Layers", "Layers", paste).when(need_copied).params( + C::new("layer.paste", "Paste Layers", "Layers", paste).roles(&[]).when(need_copied).params( r#"{"map"?: name (default: the map in view), "group"?: name|id (into that group, at its top; copied groups go in it too), "to"?: position (0 = top, the default; never inside a group: above it)}"#, ), ] diff --git a/crates/engine/src/cmd/layout.rs b/crates/engine/src/cmd/layout.rs index 132cab180..fa134886b 100644 --- a/crates/engine/src/cmd/layout.rs +++ b/crates/engine/src/cmd/layout.rs @@ -25,26 +25,26 @@ const MAX_LAYOUT: u64 = 64 << 20; pub fn commands() -> Vec { vec![ - C::new("layout.new", "New Layout", "Home › Project", new_layout).params( + C::new("layout.new", "New Layout", "Home › Project", new_layout).roles(&[]).params( r#"{"name"?: string, "page"?: "letter"|"legal"|"tabloid"|"a5"|"a4"|"a3"|"a2"|"a1"|"a0", "orientation"?: "landscape"|"portrait", "width"?: mm, "height"?: mm, "template"?: "map" (frame, title, legend, north arrow, scale bar; the default)|"blank"}"#, ), - C::new("layout.publication", "New Publication Layout", "Publish › Layout", publication).params( + C::new("layout.publication", "New Publication Layout", "Publish › Layout", publication).roles(&[]).params( r#"{"profile"?: "a4" (the default) | "a3" | "a2" (a sheet turned to portrait or landscape, whichever the area's map comes out larger on: its extent east to west against north to south, on the map's projection) | "a4-portrait"|"a4-landscape"|"a3-portrait"|"a3-landscape"|"a2-portrait"|"a2-landscape" (a sheet: frame, title, subtitle, the map, a boxed footer)|"figure-1col" (90 mm)|"figure-1.5col" (140 mm)|"figure-2col" (190 mm)|"thesis-full" (160 × 225 mm)|"thesis-half" (160 × 110 mm) (a figure: the map edge to edge, its title and sources left to the caption), "extent"?: [west, south, east, north] in degrees that must fit (default: the map view), "name"?: string, "title"?: text, "subtitle"?: text, "sources"?: text | [texts], "reference"?: text | [texts] (default: the projection, datum and graticule), "note"?: text, "author"?: text (default: "Made with RustGIS · {date}"; "" or [] for none), "headings"?: [the sources' heading, the reference's heading, an editorial column's heading], "locator"?: "globe" (a globe with the area outlined; the default on sheets) | "inset" (the surrounding region, the area outlined) | "none" (the default on figures), "decimal"?: "." (the default) | "," (graticule, scale bar and colorbar numbers and {scale} with a decimal comma: 0,05 and 1:6.300.000), "style"?: "classic" (the default: the title over the map, the footer under it) | "banner" (the title and subtitle in a band of the accent colour across the top) | "panel" (the map at the left, a panel at the right with the title, locator, legend, sources and reference) | "editorial" (the title large in serif, a column at the right with the note, the locator and the legend) | "modular" (cards on a light page: the map, the locator and the legend each in one, the sources and reference in tinted ones) | "poster" (the page in the accent colour, the title in light text, the map in a light card, the footer in darker blocks) (a sheet's arrangement; figures leave it to the caption), "accent"?: a colour as #RRGGBB (the style's colour: its band, panel tint, headings and rules, a poster's page), "arrangement"?: "best" (the default: the favourite and each arrangement by name laid out, the one moving the least area off its corners kept, then the one leaving the least over the data, the favourite on a tie) | "favourite" (layout.arrangement's) | an arrangement by name or corner by corner, as layout.arrangement takes (the corners the locator, north arrow, legend and scale bar are at home in; one covering what the map draws there still moves)} → a new layout of the active map at a round scale, its texts wrapped to their boxes; {projection}, {datum} and {grid} in the texts are filled from the map"#, ), - C::new("layout.arrangement", "Favourite Arrangement", "Publish › Layout", arrangement).view().params( + C::new("layout.arrangement", "Favourite Arrangement", "Publish › Layout", arrangement).roles(&[]).view().params( r#"{"favourite"?: "classic" (the locator top left, the north arrow top right, the legend bottom left, the scale bar bottom right) | "mirrored" | "legendRight" | "legendTop" | {"locator", "northArrow", "legend", "scaleBar": "top-left"|"top-right"|"bottom-left"|"bottom-right", each a corner of its own}} → the favourite arrangement of publication layouts (kept with the app's preferences), its corners and the arrangements by name; with no favourite given, it is only shown"#, ), C::new("layout.activate", "Open Layout", "Browser", |s, p| { let i = index(s, &by_name(p))?; s.active_layout = Some(i); Ok(json!({"layout": i})) - }) + }).roles(&[]) .view() .params(r#"{"name": string | "layout": name|index}"#), C::new("layout.close", "Show Map", "Browser", |s, _| { s.active_layout = None; Ok(Value::Null) - }) + }).roles(&[]) .view(), C::new("layout.remove", "Delete Layout", "Browser", |s, p| { let i = index(s, &by_name(p))?; @@ -56,7 +56,7 @@ pub fn commands() -> Vec { _ => None, }; Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name"?: string, "layout"?: name|index (instead of name; default: the open layout)}"#), C::new("layout.rename", "Rename Layout", "Browser", |s, p| { let name = ps(p, "newName").map(str::trim).filter(|n| !n.is_empty()).ok_or("newName is required")?; @@ -67,7 +67,7 @@ pub fn commands() -> Vec { l.name = unique.clone(); } Ok(json!({"name": unique})) - }) + }).roles(&[]) .params(r#"{"name"?: string, "layout"?: name|index (instead of name; default: the open layout), "newName": string}"#), // A copy of a layout with all its elements (#288), right after it and open. Element ids // are per layout, so legends and scale bars stay on their frames, which show the same maps. @@ -79,27 +79,27 @@ pub fn commands() -> Vec { s.project.layouts.insert(i + 1, l); s.active_layout = Some(i + 1); Ok(json!({"name": name, "layout": i + 1})) - }) + }).roles(&[]) .params(r#"{"name"?: string, "layout"?: name|index (instead of name; default: the open layout), "newName"?: string (default: " copy")}"#), - C::new("layout.page", "Page Size", "Publish › Page Setup", page).params( + C::new("layout.page", "Page Size", "Publish › Page Setup", page).roles(&[]).params( r#"{"layout"?, "page"?: page id, "orientation"?: "landscape"|"portrait", "width"?: mm, "height"?: mm, "decimal"?: "."|"," (the decimal mark of the numbers its surrounds print)}"#, ), - C::new("layout.addFrame", "Map Frame", "Publish › Map Frames", add_frame).params( + C::new("layout.addFrame", "Map Frame", "Publish › Map Frames", add_frame).roles(&[]).params( r#"{"layout"?, "shape"?: "rectangle"|"circle"|"ellipse"|"polygon"|"lasso", "points"?: [[u, v], …] (polygon/lasso, 0..1 across the frame), "map"?: name, "x"?, "y"?, "w"?, "h"?: mm}"#, ), - C::new("layout.addElement", "Add Element", "Publish › Map Surrounds", add_element).params( + C::new("layout.addElement", "Add Element", "Publish › Map Surrounds", add_element).roles(&[]).params( r##"{"layout"?, "kind": "northArrow"|"scaleBar"|"legend"|"colorbar"|"text"|"dynamicText"|"picture"|"table"|"chart"|"rectangle" (a neatline framing the page unless any of x, y, w, h is given)|"ellipse"|"circle"|"line"|"arrow"|"extentIndicator"|"grid", "frame"?: element id (default: the first map frame), "of"?: frame id (extentIndicator, which also takes "color" and "width" in points, default red and 2), "text"?, "size"?, "color"?: "#RRGGBB"|"#RRGGBBAA" (text, dynamicText, line, arrow, rectangle, ellipse, circle: its outline; extentIndicator; grid; scaleBar; refused for other kinds), "bold"?: bool (text), "italic"?: bool, "align"?: "left"|"center"|"right", "letterSpacing"?: points, -5 to 50 (text, dynamicText), "font"?: "sans"|"serif"|an installed font family's name, as labels.fonts lists them (text, dynamicText; a family not installed here is drawn in its fallback, and a serif where none is installed in the sans, with a warning; refused for other kinds), "fallback"?: "sans"|"serif" (a named font's, where it isn't installed; default its own class, else sans), "title"?: legend title (default "Legend"), "allClasses"?: bool (legend: every class, not only those the frame draws), "hideOther"?: bool (legend: leave out ), "exclude"?: [layer names] (legend: layers it leaves out though the map draws them), "orientation"?: "horizontal"|"vertical" (colorbar), "ticks"?: 2..21 (colorbar: labelled values, ends included), "tickSide"?: "below"|"above"|"right"|"left" (colorbar), "reverse"?: bool (colorbar: high values first), "fill"?: "#RRGGBBAA" (rectangle, ellipse, circle, northArrow; scaleBar: the alternating bar's light divisions, default white; refused for other kinds), "width"?: outline width in points (rectangle, ellipse, circle; 0 for none) or line width in points (line, grid), "radius"?: mm, 0 to 50 (rectangle: rounded corners), "dir"?: "horizontal"|"vertical"|"falling"|"rising" (line), "head"?: "none"|"end"|"start"|"both" (line; default "end" for an arrow), "angle"?: degrees anticlockwise from pointing right, centred on the page unless x or y is given (line, arrow; it sets the box and the direction, so w, h and dir are not used with it), "length"?: mm (line, arrow), "image"?: base64, "path"?: picture file, "layer"?: (table, chart; colorbar: a raster layer, default the top visible one), "field"?, "label"?, "rows"?, "step"?: degrees, or a measured grid's units (metres) (0 picks one), "type"?: "graticule" (latitudes and longitudes, default) | "measured" (eastings and northings in a projected system, as ArcGIS Pro's measured grid), "crs"?: a measured grid's coordinate system: EPSG:25832, WKT or a PROJ string ("" or none: the map's when projected, else the UTM zone at the frame's centre), "short"?: bool (a measured grid's labels as their kilometres' last two digits, 43 for 543000), "labels"?: bool, "style"?: "lines"|"ticks"|"linesAndTicks" (grid) or "classic"|"compass"|"triangle"|"star" (northArrow) or "alternating"|"stepped"|"line" (scaleBar), "tickLength"?: mm, "format"?: "decimal"|"dms", "hemisphere"?: bool (N/S/E/W rather than a sign), "edges"?: ["top","bottom","left","right"] (labelled edges), "labelOrientation"?: "horizontal"|"vertical", "off"?: bool (grid; options not given keep the grid's), "outline"?: "#RRGGBB" (northArrow, or a rectangle's, ellipse's or circle's outline instead of color; refused for other kinds), "units"?: "auto"|"kilometers"|"meters"|"miles"|"feet", "divisions"?: 1..10 (scaleBar, whose "color" is its lines, dark divisions and labels, default #222222), "x"?, "y"?, "w"?, "h"?: mm}"##, ), - C::new("layout.update", "Element Properties", "Publish › Element", update).params( + C::new("layout.update", "Element Properties", "Publish › Element", update).roles(&[]).params( r##"{"layout"?, "element": id, "x"?, "y"?, "w"?, "h"?: mm, "name"?, "visible"?, "locked"?: bool (a locked element takes no other change until unlocked; "locked": false unlocks it and may come with changes), "text"?, "size"?, "color"? (text, line, rectangle, ellipse: its outline; scaleBar: its lines, dark divisions and labels; refused for other elements), "bold"?, "italic"?, "align"?, "letterSpacing"?, "font"?: "sans"|"serif"|an installed font family's name, "fallback"?: "sans"|"serif" (text, as in layout.addElement; refused for other elements), "title"?, "allClasses"?: bool (legend), "hideOther"?: bool (legend), "exclude"?: [layer names] (legend; [] lists every layer again), "orientation"?, "ticks"?, "tickSide"?, "reverse"?: (colorbar), "style"?: (northArrow, scaleBar), "fill"?, "outline"? (northArrow, rectangle, ellipse, and a scaleBar's fill; refused for other elements; for a rectangle or ellipse, color sets it when outline is not given; a scaleBar's fill is the alternating bar's light divisions), "radius"?: mm (rectangle: rounded corners), "width"?: points (rectangle, ellipse, line), "dir"?, "head"?, "angle"?, "length"? (line: turned or sized about its centre, then moved by any x or y; dir does not come with angle or length), "units"?, "divisions"? (scaleBar), "map"?, "scale"?, "center"?: [lon, lat], "bearing"?: degrees clockwise from north, "pitch"?: degrees of tilt (0 to 85), "grid"?: {grid options as in layout.addElement} | false (map frame), "extentIndicator"?: {"color"?, "width"?: points} | false (map frame: its extent indicator's style, or removes it), "layer"?: (colorbar: "" for the top visible raster), "field"?, "rows"?}"##, ), - C::new("layout.guides", "Guides", "Publish › Page Setup", guides).params( + C::new("layout.guides", "Guides", "Publish › Page Setup", guides).roles(&[]).params( r#"{"layout"?, "x"?: [mm] (vertical guides, from the page's left edge), "y"?: [mm] (horizontal guides, from its top edge), "clear"?: bool (remove every guide first)}"#, ), - C::new("layout.duplicateElement", "Duplicate Element", "Publish › Element", duplicate_element).params( + C::new("layout.duplicateElement", "Duplicate Element", "Publish › Element", duplicate_element).roles(&[]).params( r#"{"layout"?, "element": id, "dx"?, "dy"?: mm (offset of the copy, default 5)}"#, ), - C::new("layout.arrange", "Arrange Element", "Publish › Element", arrange).params( + C::new("layout.arrange", "Arrange Element", "Publish › Element", arrange).roles(&[]).params( r#"{"layout"?, "element": id, "to": "front"|"forward"|"backward"|"back" (later elements are drawn over earlier ones: front puts it over every other, forward over the one above it, backward under the one below, back under every other)}"#, ), C::new("layout.removeElement", "Delete Element", "Publish › Element", |s, p| { @@ -110,18 +110,18 @@ pub fn commands() -> Vec { unlocked(l, locked)?; } if l.remove(id) { Ok(Value::Null) } else { Err(format!("no element {id}")) } - }) + }).roles(&[]) .params(r#"{"layout"?, "element": id}"#), - C::new("layout.frameZoom", "Zoom Frame", "Publish › Map Frame", frame_zoom).params( + C::new("layout.frameZoom", "Zoom Frame", "Publish › Map Frame", frame_zoom).roles(&[]).params( r#"{"layout"?, "element"?: frame id, "frame"?: (as element), "to"?: "view" (the map view's extent, bearing and pitch; the default)|"full"|"selection"|"layer", "layer"?}"#, ), - C::new("layout.series", "Map Series", "Publish › Map Frames", series).params( + C::new("layout.series", "Map Series", "Publish › Map Frames", series).roles(&[]).params( r#"{"layout"?, "layer": index layer, "nameField"?: string, "frame"?: id, "margin"?: 0..1, "off"?: bool}"#, ), - C::new("layout.seriesPage", "Map Series Page", "Publish › Map Frames", series_page).params(r#"{"layout"?, "page": number|"next"|"previous"}"#), - C::new("layout.seriesRefresh", "Refresh Map Series", "Publish › Map Frames", |s, p| series_page(s, &json!({"layout": p.get("layout"), "page": "current"}))) + C::new("layout.seriesPage", "Map Series Page", "Publish › Map Frames", series_page).roles(&[]).params(r#"{"layout"?, "page": number|"next"|"previous"}"#), + C::new("layout.seriesRefresh", "Refresh Map Series", "Publish › Map Frames", |s, p| series_page(s, &json!({"layout": p.get("layout"), "page": "current"}))).roles(&[]) .params(r#"{"layout"?}"#), - C::new("layout.export", "Export Layout", "Publish › Output", |s, p| super::exports::write_here(s, p, "layout.export")) + C::new("layout.export", "Export Layout", "Publish › Output", |s, p| super::exports::write_here(s, p, "layout.export")).roles(&[]) .view() .params(r#"{"layout"?, "path": "*.png|*.pdf|*.svg" (PNG/SVG: current page; SVG: each element a group of its own; text and shapes as SVG, map frames and surrounds as vectors, rasters and tiles in them as pictures at dpi), "dpi"?: 72..600, "allPages"?: bool (PDF only; default true: every map series page, up to 500; false: the current page)}"#), C::new("layout.save", "Save Layout File", "Publish › Save As", |s, p| { @@ -130,7 +130,7 @@ pub fn commands() -> Vec { let text = serde_json::to_string_pretty(&l).map_err(|e| e.to_string())?; rustgis_data::io::write_atomic(Path::new(path), text.as_bytes()).map_err(|e| format!("{path}: {e}"))?; Ok(json!({"path": path})) - }) + }).roles(&[]) .view() .params(r#"{"layout"?, "path": "*.rgislayout"}"#), C::new("layout.import", "Import Layout", "Home › Project", |s, p| { @@ -143,7 +143,7 @@ pub fn commands() -> Vec { (None, None) => return Err("path or text is required".into()), }; import(s, &text) - }) + }).roles(&[]) .params(r#"{"path"?: "*.rgislayout", "text"?: layout JSON}"#), ] } diff --git a/crates/engine/src/cmd/navigate.rs b/crates/engine/src/cmd/navigate.rs index 86b2b2456..66406e66a 100644 --- a/crates/engine/src/cmd/navigate.rs +++ b/crates/engine/src/cmd/navigate.rs @@ -11,23 +11,23 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("map.zoomFull", "Full Extent", "Home › Navigate", zoom_full).view(), - C::new("map.zoomIn", "Fixed Zoom In", "Home › Navigate", |s, _| zoom(s, 2.0)).view().key("Mod+="), - C::new("map.zoomOut", "Fixed Zoom Out", "Home › Navigate", |s, _| zoom(s, 0.5)).view().key("Mod+-"), + C::new("map.zoomFull", "Full Extent", "Home › Navigate", zoom_full).roles(&[]).view(), + C::new("map.zoomIn", "Fixed Zoom In", "Home › Navigate", |s, _| zoom(s, 2.0)).roles(&[]).view().key("Mod+="), + C::new("map.zoomOut", "Fixed Zoom Out", "Home › Navigate", |s, _| zoom(s, 0.5)).roles(&[]).view().key("Mod+-"), C::new("map.prevExtent", "Previous Extent", "Home › Navigate", |s, _| { if s.extent_back() { Ok(Value::Null) } else { Err("no previous extent".into()) } - }) + }).roles(&[]) .view() .when(|s| if s.can_extent_back() { None } else { Some("no previous extent") }), C::new("map.nextExtent", "Next Extent", "Home › Navigate", |s, _| { if s.extent_forward() { Ok(Value::Null) } else { Err("no next extent".into()) } - }) + }).roles(&[]) .view() .when(|s| if s.can_extent_forward() { None } else { Some("no next extent") }), C::new("map.zoomWorld", "Zoom To World", "Home › Navigate", |s, _| { s.zoom_to(&Extent::new(-170.0, -58.0, 180.0, 78.0)); Ok(Value::Null) - }) + }).roles(&[]) .view(), C::new("map.extent", "View Extent", "Home › Navigate", |s, _| { let e = if s.map().projection.is_globe() { Extent::new(-180.0, -90.0, 180.0, 90.0) } else { super::workspace::view_extent(s) }; @@ -35,26 +35,26 @@ pub fn commands() -> Vec { return Err("the view shows no part of the Earth".into()); } Ok(json!({"extent": [e.min_x, e.min_y, e.max_x, e.max_y]})) - }) + }).roles(&[]) .view(), - C::new("map.zoomToExtent", "Zoom To Extent", "Home › Navigate", zoom_to_extent) + C::new("map.zoomToExtent", "Zoom To Extent", "Home › Navigate", zoom_to_extent).roles(&[]) .view() .params(r#"{"extent": [minLon, minLat, maxLon, maxLat]}"#), - C::new("map.goToXY", "By Coordinates…", "Home › Navigate › Locate", go_to_xy) + C::new("map.goToXY", "By Coordinates…", "Home › Navigate › Locate", go_to_xy).roles(&[]) .view() .params(r#"{"lon" | "x": number, "lat" | "y": number, "scale"?: number, "bearing"?: degrees clockwise from north, "pitch"?: degrees of tilt (0 to 85)}"#), - C::new("map.goToLink", "Go To Map Link", "Home › Navigate", go_to_link) + C::new("map.goToLink", "Go To Map Link", "Home › Navigate", go_to_link).roles(&[]) .view() .params(r#"{"url": a map link ending in #zoom/lat/lon[/bearing[/pitch]] (as MapLibre writes them; also #map=…), or just zoom/lat/lon[/bearing[/pitch]]}"#), C::new("map.link", "Map Link", "Home › Navigate", |s, _| { let hash = link_of(s.map()).ok_or("the view shows no part of the Earth")?; Ok(json!({"hash": hash})) - }) + }).roles(&[]) .view(), - C::new("map.setBearing", "Rotate Map", "Home › Navigate", set_bearing) + C::new("map.setBearing", "Rotate Map", "Home › Navigate", set_bearing).roles(&[]) .view() .params(r#"{"bearing": degrees clockwise from north (the direction the top of the view faces) | "by": degrees to turn it further}"#), - C::new("map.setPitch", "Tilt Map", "Home › Navigate", set_pitch) + C::new("map.setPitch", "Tilt Map", "Home › Navigate", set_pitch).roles(&[]) .view() .params(r#"{"pitch": degrees of tilt from looking straight down (0 to 85) | "by": degrees to tilt it further}"#), C::new("map.resetNorth", "Reset North", "Home › Navigate", |s, _| { @@ -63,29 +63,29 @@ pub fn commands() -> Vec { m.camera.set_pitch(0.0); s.record_extent(); Ok(Value::Null) - }) + }).roles(&[]) .view(), C::new("map.setScale", "Set Scale", "Status Bar", |s, p| { let sc = f(p, "scale").ok_or("scale is required")?; s.map_mut().set_scale(sc); s.record_extent(); Ok(Value::Null) - }) + }).roles(&[]) .view() .params(r#"{"scale": number}"#), - C::new("map.basemap", "Basemap", "Home › Add", basemap) + C::new("map.basemap", "Basemap", "Home › Add", basemap).roles(&[]) .params(r#"{"name": "paper"|"midnight"|"ocean"|"atlas"|"blank", "keepTiles"?: bool (keeps the web tile basemaps over it)}"#), C::new("map.graticule", "Graticule", "View › Display", |s, p| { let m = s.map_mut(); m.graticule = super::b(p, "on").unwrap_or(!m.graticule); Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"on"?: bool (toggles when omitted)}"#), C::new("map.reloadWebLayers", "Reload Web Layers", "View › Display", |_, _| { // The map view draws again when the tile cache's generation moves. crate::render::rustgis_tiles::cache::clear(); Ok(json!({"generation": crate::render::rustgis_tiles::cache::generation()})) - }) + }).roles(&[]) .view(), C::new("map.projection", "Globe", "View › View", |s, p| { // A preset's parameters, over those of `type` or of the map's own preset. @@ -111,18 +111,18 @@ pub fn commands() -> Vec { super::workspace::fill_cache(s); } Ok(out) - }) + }).roles(&[]) .params(r#"{"type"?: "mercator"|"globe"|preset id (with parameters: "robinson +lon_0=150")|"EPSG:"|"ESRI:" (toggles the globe when omitted), "lon_0"?: degrees (a preset's central meridian), "lat_1"?, "lat_2"?: degrees (a conic preset's standard parallels; these apply to the map's own preset when type is omitted)}"#), C::new("map.projections", "Projections", "View › View", |_, _| { let mut v = vec![json!({"id": "mercator", "label": "Web Mercator"}), json!({"id": "globe", "label": "Globe"})]; v.extend(rustgis_map::projection::PRESETS.iter().map(|(id, label, _)| json!({"id": id, "label": label}))); Ok(Value::Array(v)) - }) + }).roles(&[]) .view(), - C::new("bookmark.add", "New Bookmark", "Home › Navigate › Bookmarks", add_bookmark).params( + C::new("bookmark.add", "New Bookmark", "Home › Navigate › Bookmarks", add_bookmark).roles(&[]).params( r#"{"name"?: string (a number is added when another bookmark has it), "description"?: string, "map"?: name (default: the active map; its current view)}"#, ), - C::new("bookmark.go", "Go To Bookmark", "Home › Navigate › Bookmarks", go_bookmark) + C::new("bookmark.go", "Go To Bookmark", "Home › Navigate › Bookmarks", go_bookmark).roles(&[]) .view() .params(r#"{"name": string, "map"?: name (the bookmark's map), "pan"?: bool (keep the current scale)}"#), C::new("bookmark.remove", "Remove Bookmark", "Home › Navigate › Bookmarks", |s, p| { @@ -131,7 +131,7 @@ pub fn commands() -> Vec { m.bookmarks.remove(bi); } Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name": string, "map"?: name (default: the active map)}"#), C::new("bookmark.rename", "Rename Bookmark", "Bookmarks", |s, p| { let new = ps(p, "newName").map(str::trim).filter(|n| !n.is_empty()).ok_or("newName is required")?; @@ -141,7 +141,7 @@ pub fn commands() -> Vec { b.name = name.clone(); } Ok(json!({"name": name})) - }) + }).roles(&[]) .params(r#"{"name": string, "newName": string (a number is added when another bookmark has it), "map"?: name}"#), C::new("bookmark.update", "Update Bookmark", "Bookmarks", |s, p| { let (mi, bi) = bookmark_at(s, p)?; @@ -156,7 +156,7 @@ pub fn commands() -> Vec { } } Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name": string (it takes the current view; its map must be in view), "map"?: name}"#), C::new("bookmark.describe", "Bookmark Description", "Bookmarks", |s, p| { let text: String = ps(p, "description").ok_or("description is required")?.trim().chars().take(MAX_DESCRIPTION).collect(); @@ -165,7 +165,7 @@ pub fn commands() -> Vec { b.description = text; } Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name": string, "description": string (empty clears it), "map"?: name}"#), C::new("bookmark.move", "Move Bookmark", "Bookmarks", |s, p| { let to = super::uint(p, "to").ok_or("to is required (an index)")?; @@ -176,14 +176,14 @@ pub fn commands() -> Vec { m.bookmarks.insert(to, b); } Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name": string, "to": index (0: first; past the end: last), "map"?: name}"#), - C::new("bookmark.save", "Export Bookmarks", "Home › Navigate › Bookmarks", |s, p| super::items::save_item(s, "bookmarks", p)) + C::new("bookmark.save", "Export Bookmarks", "Home › Navigate › Bookmarks", |s, p| super::items::save_item(s, "bookmarks", p)).roles(&[]) .view() .params(r#"{"path": "*.rgisbookmarks"}"#), - C::new("bookmark.import", "Import Bookmarks", "Home › Navigate › Bookmarks", import_bookmarks) + C::new("bookmark.import", "Import Bookmarks", "Home › Navigate › Bookmarks", import_bookmarks).roles(&[]) .params(r#"{"path"?: "*.rgisbookmarks", "text"?: bookmarks JSON, "data"?: base64 file}"#), - C::new("map.new", "New Map", "Home › Project", new_map).params(r#"{"name"?: string}"#), + C::new("map.new", "New Map", "Home › Project", new_map).roles(&[]).params(r#"{"name"?: string}"#), C::new("map.activate", "Activate Map", "Browser", |s, p| { let n = s_name(p)?; let i = s.project.maps.iter().position(|m| m.name.eq_ignore_ascii_case(n)).ok_or_else(|| format!("no map '{n}'"))?; @@ -191,7 +191,7 @@ pub fn commands() -> Vec { s.active_layer = s.map().layers.first().map(|l| l.id); s.table_layer = None; Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name": string}"#) .view(), C::new("map.rename", "Rename Map", "Layers", |s, p| { @@ -208,9 +208,9 @@ pub fn commands() -> Vec { } follow_renamed_map(s, &old, &name); Ok(json!({"name": name})) - }) + }).roles(&[]) .params(r#"{"name": string (the new name; a number is added when another map has it), "map"?: name (default: the active map)}"#), - C::new("map.duplicate", "Duplicate Map", "Browser", duplicate_map) + C::new("map.duplicate", "Duplicate Map", "Browser", duplicate_map).roles(&[]) .params(r#"{"map"?: name (default: the active map), "name"?: string (default: " copy"; a number is added when another map has it)}"#), C::new("map.remove", "Delete Map", "Browser", |s, p| { if s.project.maps.len() < 2 { @@ -229,7 +229,7 @@ pub fn commands() -> Vec { s.table_layer = None; } Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name": string}"#), ] } diff --git a/crates/engine/src/cmd/nldi.rs b/crates/engine/src/cmd/nldi.rs index 4f7bb71d2..1dd51601d 100644 --- a/crates/engine/src/cmd/nldi.rs +++ b/crates/engine/src/cmd/nldi.rs @@ -13,19 +13,19 @@ const WHERE: &str = "Home › Add › Web Data › United States › USGS NLDI"; pub fn commands() -> Vec { vec![ - C::new("nldi.flowline", "NLDI Flowline", WHERE, flowline).params( + C::new("nldi.flowline", "NLDI Flowline", WHERE, flowline).roles(&[]).params( r#"{"lon": number, "lat": number (a point in the United States), "direction"?: "none" (the whole flowline, default) | "up" (its part upstream of the point) | "down", "name"?: text, "answer"?: "the flowtrace's GeoJSON, or a COMID's flowline, when already fetched", "snapped"?: [lon, lat] (with a COMID's flowline: where the point meets it)} — adds the flowline and the raindrop's path to it; returns its COMID"#, ), - C::new("nldi.comid", "NLDI COMID at a Point", WHERE, comid_cmd) + C::new("nldi.comid", "NLDI COMID at a Point", WHERE, comid_cmd).roles(&[]) .view() .params(r#"{"lon": number, "lat": number} — the nearest NHDPlus flowline's COMID, and where the point snaps to it"#), - C::new("nldi.basin", "NLDI Basin", WHERE, basin).params( + C::new("nldi.basin", "NLDI Basin", WHERE, basin).roles(&[]).params( r#"{"comid"?: number (an NHDPlus flowline's), "lon"?: number, "lat"?: number (or a point: its flowline's COMID), "name"?: text, "answer"?: "the basin's GeoJSON, when already fetched"} — adds the basin draining to the flowline"#, ), - C::new("nldi.sources", "NLDI Navigation Sources", WHERE, sources).view().params( + C::new("nldi.sources", "NLDI Navigation Sources", WHERE, sources).roles(&[]).view().params( r#"{"comid"?: number (an NHDPlus flowline's), "lon"?: number, "lat"?: number (or a point: its flowline's COMID), "mode"?: "UM" (upstreamMain, default) | "UT" (upstreamTributaries) | "DM" (downstreamMain) | "DD" (downstreamDiversions), "answer"?: text} — the linked data a navigation can list (flowlines, nwissite, wqp, huc12pp…)"#, ), - C::new("nldi.navigate", "NLDI Navigation", WHERE, navigate).params( + C::new("nldi.navigate", "NLDI Navigation", WHERE, navigate).roles(&[]).params( r#"{"comid"?: number (an NHDPlus flowline's), "lon"?: number, "lat"?: number (or a point: its flowline's COMID), "mode"?: "UM" (upstreamMain, default) | "UT" | "DM" | "DD", "distance"?: km 1–9999 (500), "source"?: "flowlines" (default) or a source nldi.sources lists (nwissite, wqp…), "name"?: text, "answer"?: "the navigation's GeoJSON, when already fetched"} — adds the flowlines or sites along the way"#, ), ] diff --git a/crates/engine/src/cmd/notebook.rs b/crates/engine/src/cmd/notebook.rs index c82763a71..0f3bbfa62 100644 --- a/crates/engine/src/cmd/notebook.rs +++ b/crates/engine/src/cmd/notebook.rs @@ -131,6 +131,7 @@ pub fn commands() -> Vec { s.trusted_notebooks.insert(name.to_ascii_lowercase()); Ok(r) }) + .roles(&[]) .params(r#"{"name"?}"#), C::new("notebook.open", "Add and Open Notebook", "Analyze › Notebook", |s, p| { let text = super::items::read_text(p)?; @@ -143,12 +144,14 @@ pub fn commands() -> Vec { s.trusted_notebooks.remove(&nb.name.to_ascii_lowercase()); push(s, nb) }) + .roles(&[crate::ParamRole { key: "text", role: crate::Role::Document(crate::Document::Notebook) }]) .params(r#"{"path"?: "*.rgisnb|*.ipynb", "text"?: notebook JSON, "data"?: base64 file}"#), C::new("notebook.activate", "Open Notebook", "Browser", |s, p| { let i = index(s, p)?; s.active_notebook = Some(i); Ok(json!({"notebook": i})) }) + .roles(&[]) .view() .when(need_notebook) .params(r#"{"notebook": name|index}"#), @@ -159,6 +162,7 @@ pub fn commands() -> Vec { s.active_notebook = None; Ok(Value::Null) }) + .roles(&[]) .when(need_notebook) .params(r#"{"notebook"?}"#), C::new("notebook.addCell", "Add Cell", "Notebook", |s, p| { @@ -177,6 +181,7 @@ pub fn commands() -> Vec { nb.cells.insert(at, Cell::new(kind, ps(p, "source").unwrap_or(""))); Ok(json!({"cell": at, "cells": nb.cells.len()})) }) + .roles(&[crate::ParamRole { key: "source", role: crate::Role::Opaque }]) .when(need_notebook) .params(r#"{"notebook"?, "kind"?: "code"|"markdown", "source"?, "at"?: index}"#), C::new("notebook.editCell", "Edit Cell", "Notebook", |s, p| { @@ -186,6 +191,7 @@ pub fn commands() -> Vec { c.source = source.chars().take(MAX_SOURCE).collect(); Ok(json!({"cell": k})) }) + .roles(&[crate::ParamRole { key: "source", role: crate::Role::Opaque }]) .when(need_notebook) .params(r#"{"notebook"?, "cell": index, "source": text}"#), C::new("notebook.removeCell", "Delete Cell", "Notebook", |s, p| { @@ -194,9 +200,11 @@ pub fn commands() -> Vec { nb.cells.remove(k); Ok(json!({"cells": nb.cells.len()})) }) + .roles(&[]) .when(need_notebook) .params(r#"{"notebook"?, "cell": index}"#), C::new("notebook.run", "Run Cells", "Notebook", run) + .roles(&[]) .when(need_notebook) .params(r#"{"notebook"?, "cell"?: index (default: every command cell, stopping at an error)}"#), C::new("notebook.save", "Save Notebook", "Notebook", |s, p| { @@ -206,6 +214,7 @@ pub fn commands() -> Vec { rustgis_data::io::write_atomic(Path::new(path), &bytes).map_err(|e| format!("{path}: {e}"))?; Ok(json!({"path": path})) }) + .roles(&[]) .view() .when(need_notebook) .params(r#"{"notebook"?, "path": "*.rgisnb|*.ipynb"}"#), @@ -278,6 +287,7 @@ fn run(s: &mut Session, p: &Value) -> CmdResult { r } +/// Runs command cells and cleans generated output before applying its retained-size bound. fn run_cells(s: &mut Session, p: &Value) -> CmdResult { let i = index(s, p)?; let cells: Vec = match p.get("cell") { @@ -328,8 +338,9 @@ fn run_cells(s: &mut Session, p: &Value) -> CmdResult { .collect(); if lines.is_empty() { "(nothing to run)".to_string() } else { lines.join("\n") } } - Err(e) => e, + Err(e) => rustgis_data::diagnostics::Redacted::new(&e).into_string(), }; + let text = rustgis_data::address::scrub_addresses(&text).0.into_owned(); // Into the cell that ran, which a command removing or adding cells may have moved: the // cell with its source nearest to where it was. let ran_cell = find(s).and_then(|i| s.project.notebooks.get_mut(i)).and_then(|nb| { @@ -580,8 +591,58 @@ fn jupyter_file(nb: &Notebook) -> Value { } /// The notebook as a RustGIS notebook or (nbformat 4) Jupyter notebook. -pub fn notebook_file(s: &Session, i: usize, jupyter: bool) -> Result, String> { - let nb = s.project.notebooks.get(i).ok_or("no notebook")?; - let v = if jupyter { jupyter_file(nb) } else { serde_json::to_value(nb).map_err(|e| e.to_string())? }; - serde_json::to_vec_pretty(&v).map_err(|e| e.to_string()) +pub fn notebook_file(s: &mut Session, i: usize, jupyter: bool) -> Result, String> { + let nb = s.project.notebooks.get_mut(i).ok_or("no notebook")?; + let (bytes, report) = crate::durable::output(nb, None, |nb| { + let v = if jupyter { jupyter_file(nb) } else { serde_json::to_value(nb).map_err(|e| e.to_string())? }; + serde_json::to_vec_pretty(&v).map_err(|e| e.to_string()) + })?; + crate::durable::notify(s, &report); + Ok(bytes) +} + +impl crate::durable::Durable for Notebook { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut crate::durable::Prepare<'_>) { + use crate::durable::Leaf; + let Self { name: _, cells, jupyter: _ } = self; + let commands = self.language().is_none(); + for (i, Cell { kind, source, output, error: _, jupyter: _ }) in cells.iter().enumerate() { + if commands && *kind == CellKind::Code { + p.script(Leaf::CellSource(i), source); + } + if let Some(output) = output { + p.generated(Leaf::CellOutput(i), output); + } + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[crate::durable::Owner], leaf: &crate::durable::Leaf) -> Option> { + use crate::durable::{FieldMut, Leaf}; + if !owners.is_empty() { + return None; + } + match leaf { + Leaf::CellSource(i) => Some(FieldMut::Text(&mut self.cells.get_mut(*i)?.source)), + Leaf::CellOutput(i) => self.cells.get_mut(*i)?.output.as_mut().map(FieldMut::Text), + _ => None, + } + } +} + +/// Cleans a notebook document on its parsed copy, preserving foreign kernels and opaque metadata. +pub(crate) fn durable_document(text: &str, depth: usize) -> Result<(String, crate::durable::Report), String> { + let nb = parse(text, "Notebook")?; + let jupyter = serde_json::from_str::(text).is_ok_and(|v| v.get("nbformat").is_some()); + crate::durable::output_owned_changed(nb, depth, |nb, changed| { + if !changed { + return Ok(text.into()); + } + if jupyter { + serde_json::to_string(&jupyter_file(nb)).map_err(|e| e.to_string()) + } else { + serde_json::to_string(nb).map_err(|e| e.to_string()) + } + }) } diff --git a/crates/engine/src/cmd/pg.rs b/crates/engine/src/cmd/pg.rs index 56bb4cce2..bf3bc9b13 100644 --- a/crates/engine/src/cmd/pg.rs +++ b/crates/engine/src/cmd/pg.rs @@ -32,18 +32,18 @@ const MAX_LISTED: usize = 5_000; pub fn commands() -> Vec { vec![ - C::new("pg.connect", "Connect PostGIS", "Home › Add › Add Data", connect).view().params( + C::new("pg.connect", "Connect PostGIS", "Home › Add › Add Data", connect).roles(&[]).view().params( r#"{"connection"?: "host=… port=… dbname=… user=…, or a postgresql:// address (instead of the parts below)", "host"?: string, "port"?: number, "database"?: string, "user"?: string, "password"?: "kept for this session only, never saved", "sslmode"?: "disable"|"prefer"|"require"|"verify-ca"|"verify-full"}"#, ), - C::new("pg.disconnect", "Disconnect PostGIS", "Home › Add › Add Data", disconnect).view().when(need_connection), - C::new("pg.status", "PostGIS Connection", "Home › Add › Add Data", status).view(), - C::new("pg.tables", "List PostGIS Tables", "Home › Add › Add Data", tables).view().when(need_connection).params(r#"{"schema"?: string}"#), - C::new("pg.pendingEdits", "PostGIS Edits", "Edit › Manage Edits", pending_edits).view().params(r#"{"layer"?: "a layer added with pg.addLayer (default: the active one)"}"#), - C::new("pg.saveEdits", "Save to PostGIS", "Edit › Manage Edits", save_edits) + C::new("pg.disconnect", "Disconnect PostGIS", "Home › Add › Add Data", disconnect).roles(&[]).view().when(need_connection), + C::new("pg.status", "PostGIS Connection", "Home › Add › Add Data", status).roles(&[]).view(), + C::new("pg.tables", "List PostGIS Tables", "Home › Add › Add Data", tables).roles(&[]).view().when(need_connection).params(r#"{"schema"?: string}"#), + C::new("pg.pendingEdits", "PostGIS Edits", "Edit › Manage Edits", pending_edits).roles(&[]).view().params(r#"{"layer"?: "a layer added with pg.addLayer (default: the active one)"}"#), + C::new("pg.saveEdits", "Save to PostGIS", "Edit › Manage Edits", save_edits).roles(&[]) .params(r#"{"layer"?: "a layer added with pg.addLayer (default: the active one)"}"#), - C::new("pg.discardEdits", "Discard PostGIS Edits", "Edit › Manage Edits", discard_edits) + C::new("pg.discardEdits", "Discard PostGIS Edits", "Edit › Manage Edits", discard_edits).roles(&[]) .params(r#"{"layer"?: "a layer added with pg.addLayer (default: the active one)"}"#), - C::new("pg.addLayer", "Add PostGIS Layer", "Home › Add › Add Data", add_layer).when(need_connection).params( + C::new("pg.addLayer", "Add PostGIS Layer", "Home › Add › Add Data", add_layer).roles(&[]).when(need_connection).params( r#"{"table": "table, schema.table, or table:column for one of several geometry columns", "where"?: "an SQL condition on the table's columns", "limit"?: "the most rows of each layer (default and most: 100000)", "name"?: string, "crs"?: "EPSG:… of a geometry that doesn't say"}"#, ), ] diff --git a/crates/engine/src/cmd/plugins.rs b/crates/engine/src/cmd/plugins.rs index e9e96ca14..591456d72 100644 --- a/crates/engine/src/cmd/plugins.rs +++ b/crates/engine/src/cmd/plugins.rs @@ -19,22 +19,22 @@ pub mod fetch; pub fn commands() -> Vec { vec![ - C::new("plugin.list", "Plugins", "Plugins › Manage", list).view(), - C::new("plugin.install", "Install Plugin", "Plugins › Manage", install) + C::new("plugin.list", "Plugins", "Plugins › Manage", list).roles(&[]).view(), + C::new("plugin.install", "Install Plugin", "Plugins › Manage", install).roles(&[]) .view() .params(r#"{"path": "a .rgplugin package, or a plugin folder (for development)"}"#), - C::new("plugin.uninstall", "Uninstall Plugin", "Plugins › Manage", uninstall).view().params(r#"{"id": plugin id}"#), - C::new("plugin.enable", "Enable Plugin", "Plugins › Manage", enable).view().params(r#"{"id": plugin id, "enabled"?: bool}"#), - C::new("plugin.registry", "Browse Plugins", "Plugins › Manage", browse) + C::new("plugin.uninstall", "Uninstall Plugin", "Plugins › Manage", uninstall).roles(&[]).view().params(r#"{"id": plugin id}"#), + C::new("plugin.enable", "Enable Plugin", "Plugins › Manage", enable).roles(&[]).view().params(r#"{"id": plugin id, "enabled"?: bool}"#), + C::new("plugin.registry", "Browse Plugins", "Plugins › Manage", browse).roles(&[]) .view() .params(r#"{"url"?: "registry.json (default: https://plugins.rustgis.app/registry.json)"}"#), - C::new("plugin.installFromRegistry", "Install Plugin From Registry", "Plugins › Manage", install_from_registry) + C::new("plugin.installFromRegistry", "Install Plugin From Registry", "Plugins › Manage", install_from_registry).roles(&[]) .view() .params(r#"{"id": plugin id, "url"?: registry.json}"#), - C::new("plugin.panel", "Plugin Panel State", "Plugins", panel).view().params( + C::new("plugin.panel", "Plugin Panel State", "Plugins", panel).roles(&[]).view().params( r#"{"id": plugin id, "values"?: {widget id: true/false, a whole number, or a tree node's id (null for none)}, "refresh"?: bool (run the panel's state command)}"#, ), - C::new("plugin.prepare", "Prepare Plugin Environment", "Plugins › Manage", prepare) + C::new("plugin.prepare", "Prepare Plugin Environment", "Plugins › Manage", prepare).roles(&[]) .view() .params(r#"{"id": plugin id (a process plugin that declares "python": downloads its Python and packages)}"#), ] diff --git a/crates/engine/src/cmd/portals.rs b/crates/engine/src/cmd/portals.rs index d2c457dfa..6c8c2d8db 100644 --- a/crates/engine/src/cmd/portals.rs +++ b/crates/engine/src/cmd/portals.rs @@ -11,20 +11,20 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("osm.download", "Download OpenStreetMap Data", "Home › Add › Web Data", osm_download).params( + C::new("osm.download", "Download OpenStreetMap Data", "Home › Add › Web Data", osm_download).roles(&[]).params( r#"{"bbox"?: [west, south, east, north] | "view" (the map's extent, the default), "view"?: bool (false: a bbox must be given), "preset"?: "buildings"|"roads"|"railways"|"water"|"landuse"|"parks"|"pois"|"boundaries", "tags"?: "key=value, key" (any of them), "url"?: "an Overpass server (default https://overpass-api.de/api/interpreter)", "name"?: text, "answer"?: "the server's JSON answer, when already fetched (the app's background request)"}"#, ), - C::new("boundaries.add", "Add Administrative Boundaries", "Home › Add › Web Data", boundaries_add).params( + C::new("boundaries.add", "Add Administrative Boundaries", "Home › Add › Web Data", boundaries_add).roles(&[]).params( r#"{"country": "ISO 3166 alpha-3 code (CHE, KEN…)", "level"?: "ADM0" (the country, default) … "ADM5", "name"?: text, "text"?: "the boundaries' GeoJSON, when already fetched (the app's background request)"}"#, ), - C::new("boundaries.countries", "List geoBoundaries Countries", "Home › Add › Web Data", boundaries_countries).view(), - C::new("portal.search", "Search Open Data", "Home › Add › Web Data", portal_search).view().params( + C::new("boundaries.countries", "List geoBoundaries Countries", "Home › Add › Web Data", boundaries_countries).roles(&[]).view(), + C::new("portal.search", "Search Open Data", "Home › Add › Web Data", portal_search).roles(&[]).view().params( r#"{"query"?: text (needed by ckan, socrata, arcgis-hub and huggingface), "portal"?: "ckan" (data.gov's catalogue, the default, or any CKAN site) | "socrata" | "arcgis-hub" | "arcgis-portal" (ArcGIS Online, or the portal at url) | "earthdata-gis" (NASA's) | "openaerialmap" | "usgs-dem" (3DEP elevation models) | "ign-lidar" (IGN's LiDAR HD tiles, France) | "geolens" | "huggingface" (a dataset's files: portal.files) | "ocean-data" | "dynamical" (weather and climate Zarr stores), "url"?: "the portal: a CKAN site (https://open.canada.ca/data), a Socrata site (data.cityofnewyork.us), an ArcGIS Hub site, an ArcGIS portal, a GeoLens server", "org"?: "an ArcGIS organisation's id: its public items only", "type"?: "an ArcGIS portal's kind of item: all (the default), imagery, maps, features or webmaps", "dataset"?: "usgs-dem's elevation model: 1 meter, 1/9 arc-second (about 3 m), 1/3 arc-second (about 10 m, the default), 1 arc-second (about 30 m), Alaska 5 meter", "bbox"?: [west, south, east, north] | "view", "view"?: bool (true: where the map is; openaerialmap, usgs-dem and ign-lidar search where the map is unless given a bbox), "limit"?: 1–50 (20)}"#, ), - C::new("portal.files", "List a Dataset's Files", "Home › Add › Web Data", portal_files).view().params( + C::new("portal.files", "List a Dataset's Files", "Home › Add › Web Data", portal_files).roles(&[]).view().params( r#"{"dataset": "a Hugging Face dataset, owner/name, or an ArcGIS web map's item id (portal.search lists them)", "portal"?: "huggingface" (the default) | "arcgis-portal" | "earthdata-gis", "url"?: "the ArcGIS portal"}"#, ), - C::new("portal.add", "Add Open Data", "Home › Add › Web Data", portal_add).params( + C::new("portal.add", "Add Open Data", "Home › Add › Web Data", portal_add).roles(&[]).params( r#"{"url": "a resource's address, as portal.search lists it", "use"?: "file" (read by address, the default) | "zarr" (a store: its variable, or the first, at "index"'s steps, as layer.addZarr) | "wms" | "wfs" | "arcgis" (a service), "variable"?: "a Zarr store's variable", "index"?: {"time": 0}, "layer"?: "a service's layer (default: its first)", "name"?: text, "portal"?: "the portal's address the resource came from (an address it gives may not reach into the local network)"}"#, ), ] diff --git a/crates/engine/src/cmd/presentation.rs b/crates/engine/src/cmd/presentation.rs index 8b59ace7f..98b17dfbc 100644 --- a/crates/engine/src/cmd/presentation.rs +++ b/crates/engine/src/cmd/presentation.rs @@ -22,6 +22,7 @@ fn need_presentation(s: &Session) -> Option<&'static str> { if s.project.presentations.is_empty() { Some("add a presentation first (Publish › New Presentation)") } else { None } } +/// Registers presentation actions, with public generated video labels and guarded outputs. pub fn commands() -> Vec { vec![ C::new("presentation.new", "New Presentation", "Publish › Presentation", |s, p| { @@ -42,7 +43,7 @@ pub fn commands() -> Vec { let i = s.project.presentations.len() - 1; s.active_presentation = Some(i); Ok(json!({"presentation": i, "name": name})) - }) + }).roles(&[]) .params(r#"{"name"?, "title"?: the title page's text}"#), C::new("presentation.import", "Import Presentation", "Publish › Presentation", |s, p| { let text = super::items::read_text(p)?; @@ -57,13 +58,13 @@ pub fn commands() -> Vec { let i = s.project.presentations.len() - 1; s.active_presentation = Some(i); Ok(json!({"presentation": i, "name": name})) - }) + }).roles(&[crate::ParamRole { key: "text", role: crate::Role::Document(crate::Document::Presentation) }]) .params(r#"{"path"?: "*.rgispres", "text"?: presentation JSON, "data"?: base64 file}"#), C::new("presentation.activate", "Open Presentation", "Browser", |s, p| { let i = index(s, p)?; s.active_presentation = Some(i); Ok(json!({"presentation": i})) - }) + }).roles(&[]) .view() .when(need_presentation) .params(r#"{"presentation": name|index}"#), @@ -72,7 +73,7 @@ pub fn commands() -> Vec { s.project.presentations.remove(i); s.active_presentation = None; Ok(Value::Null) - }) + }).roles(&[]) .when(need_presentation) .params(r#"{"presentation"?}"#), C::new("presentation.mapPage", "Map Page", "Publish › Page", |s, p| { @@ -87,14 +88,14 @@ pub fn commands() -> Vec { let width_m = f(p, "widthM").unwrap_or(s.view_size.0 * map.ground_res()).clamp(10.0, 4e7); let title = ps(p, "title").unwrap_or(&name).to_string(); add_page(s, p, title, PageKind::Map { map: name, center, width_m }) - }) + }).roles(&[]) .when(need_presentation) .params(r#"{"presentation"?, "map"?: name (default: the active map, at its current view), "lon"?, "lat"?, "widthM"?: ground width shown (m), "title"?, "at"?: page index}"#), C::new("presentation.blankPage", "Blank Page", "Publish › Page", |s, p| { let background = super::layout::color(p, "background", Rgba::rgb(0xFF, 0xFF, 0xFF))?; let text: String = ps(p, "text").unwrap_or("").chars().take(MAX_TEXT).collect(); add_page(s, p, ps(p, "title").unwrap_or("").to_string(), PageKind::Blank { text, background }) - }) + }).roles(&[]) .when(need_presentation) .params(r##"{"presentation"?, "title"?, "text"?, "background"?: "#RRGGBB"|"#RRGGBBAA", "at"?}"##), C::new("presentation.imagePage", "Image Page", "Publish › Page", |s, p| { @@ -105,7 +106,7 @@ pub fn commands() -> Vec { }; let caption: String = ps(p, "caption").unwrap_or("").chars().take(MAX_CAPTION).collect(); add_page(s, p, ps(p, "title").unwrap_or("").to_string(), PageKind::Image { image, caption }) - }) + }).roles(&[]) .when(need_presentation) .params(r#"{"presentation"?, "path"?: PNG/JPEG file, "image"?: base64, "data"?: base64, "title"?, "caption"?, "at"?}"#), C::new("presentation.videoPage", "Video Page", "Publish › Page", |s, p| { @@ -116,12 +117,12 @@ pub fn commands() -> Vec { (None, None) => None, }; let caption: String = ps(p, "caption").unwrap_or("").chars().take(MAX_CAPTION).collect(); - let title = ps(p, "title").map(str::to_string).unwrap_or_else(|| file_name(&source)); + let title = ps(p, "title").map(str::to_string).unwrap_or_else(|| video_label(&source)); add_page(s, p, title, PageKind::Video { source, poster, caption }) - }) + }).roles(&[]) .when(need_presentation) .params(r#"{"presentation"?, "source": video file or URL, "poster"?: PNG/JPEG file, "posterImage"?: base64, "title"?, "caption"?, "at"?}"#), - C::new("presentation.updatePage", "Page Properties", "Presentation", update_page).when(need_presentation).params( + C::new("presentation.updatePage", "Page Properties", "Presentation", update_page).roles(&[]).when(need_presentation).params( r##"{"presentation"?, "page": index, "title"?, "text"?, "caption"?, "notes"?, "seconds"?, "background"?: "#RRGGBB"|"#RRGGBBAA", "moveTo"?: index}"##, ), C::new("presentation.removePage", "Delete Page", "Presentation", |s, p| { @@ -129,7 +130,7 @@ pub fn commands() -> Vec { let pres = s.project.presentations.get_mut(i).ok_or("no presentation")?; pres.pages.remove(k); Ok(json!({"pages": pres.pages.len()})) - }) + }).roles(&[]) .when(need_presentation) .params(r#"{"presentation"?, "page": index}"#), C::new("presentation.save", "Presentation File", "Publish › Save As", |s, p| { @@ -138,7 +139,7 @@ pub fn commands() -> Vec { let bytes = presentation_file(s, i)?; rustgis_data::io::write_atomic(Path::new(path), &bytes).map_err(|e| format!("{path}: {e}"))?; Ok(json!({"path": path})) - }) + }).roles(&[]) .view() .when(need_presentation) .params(r#"{"presentation"?, "path": "*.rgispres"}"#), @@ -159,15 +160,33 @@ pub fn commands() -> Vec { let (bytes, pages) = export_bytes(s, i, format, dpi)?; rustgis_data::io::write_atomic(Path::new(&path), &bytes).map_err(|e| format!("{path}: {e}"))?; Ok(json!({"path": path, "pages": pages, "bytes": bytes.len()})) - }) + }).roles(&[]) .view() .when(need_presentation) .params(r#"{"presentation"?, "path": "*.pdf|*.html (a self-contained slideshow, videos playable)|*.png (the first page)", "dpi"?: 48..300}"#), ] } -fn file_name(source: &str) -> String { - source.rsplit(['/', '\\']).next().filter(|n| !n.is_empty()).unwrap_or(source).chars().take(200).collect() +/// Derives generated labels from the public address before extracting a path or host. User +/// titles/captions and local filenames (including '?' and '#') are data and retain their bytes. +pub(crate) fn video_label(source: &str) -> String { + use crate::durable::{Kept, public_address}; + let clean = match public_address(source) { + Kept::Unchanged => source.to_string(), + Kept::Changed(s) => s, + Kept::Withheld(_) => return "Video".into(), + }; + let inner = crate::sources::template_archive(&clean).map_or(clean.as_str(), |(_, a)| a); + let inner = rustgis_data::s3_address::parse(inner).ok().flatten().map_or(inner, |b| b.url); + let inner = inner.strip_prefix(crate::render::rustgis_tiles::s3::ANONYMOUS).unwrap_or(inner); + let remote = inner.split_once("://").filter(|(s, _)| ["http", "https", "s3"].iter().any(|p| p.eq_ignore_ascii_case(s))); + let name = if let Some((_, rest)) = remote { + let rest = rest.split(['?', '#']).next().unwrap_or(""); + rest.rsplit('/').find(|s| !s.is_empty()).unwrap_or("Video") + } else { + inner.rsplit(['/', '\\']).next().filter(|s| !s.is_empty()).unwrap_or(inner) + }; + name.chars().take(200).collect() } /// The presentation named or numbered by `presentation`, else the active one, else the last. @@ -268,9 +287,11 @@ fn update_page(s: &mut Session, p: &Value) -> CmdResult { } /// A presentation file. -pub fn presentation_file(s: &Session, i: usize) -> Result, String> { - let pres = s.project.presentations.get(i).ok_or("no presentation")?; - serde_json::to_vec_pretty(pres).map_err(|e| e.to_string()) +pub fn presentation_file(s: &mut Session, i: usize) -> Result, String> { + let pres = s.project.presentations.get_mut(i).ok_or("no presentation")?; + let (bytes, report) = crate::durable::output(pres, None, |p| serde_json::to_vec_pretty(p).map_err(|e| e.to_string()))?; + crate::durable::notify(s, &report); + Ok(bytes) } /// A page drawn as a page layout: its title across the top and its content below. @@ -342,7 +363,7 @@ pub fn page_layout(s: &Session, pres: &Presentation, page: &Page) -> Layout { l.add( "Play", (m + bw * 0.3, top + bh / 2.0 - 6.0, bw * 0.4, 12.0), - text(&format!("Video: {}", file_name(source)), 18.0, dark, true), + text(&format!("Video: {}", video_label(source)), 18.0, dark, true), ); } if cap_h > 0.0 { @@ -359,7 +380,7 @@ pub fn page_layout(s: &Session, pres: &Presentation, page: &Page) -> Layout { l.add( "Play", (m + bw * 0.3, top + bh / 2.0 - 6.0, bw * 0.4, 12.0), - text(&format!("Video: {}", file_name(source)), 20.0, Rgba::rgb(0xFF, 0xFF, 0xFF), true), + text(&format!("Video: {}", video_label(source)), 20.0, Rgba::rgb(0xFF, 0xFF, 0xFF), true), ); if !caption.trim().is_empty() { l.add("Caption", (m, h - m - h * 0.07, w - 2.0 * m, h * 0.07), text(caption, 15.0, Rgba::rgb(0x55, 0x5B, 0x66), false)); @@ -430,6 +451,7 @@ pub fn export_bytes(s: &mut Session, i: usize, format: &str, dpi: f64) -> Result } let mut pdf_pages = Vec::new(); let mut slides = String::new(); + let mut output_report = crate::durable::Report::default(); let mut held = 0usize; let (mut frames_pictured, mut pages_pictured) = (Vec::new(), Vec::new()); for k in 0..n { @@ -452,11 +474,25 @@ pub fn export_bytes(s: &mut Session, i: usize, format: &str, dpi: f64) -> Result let body = match &page.kind { // The poster is the drawn page, which already shows the title and caption. The // source is played as given (a file next to the slideshow or a web address). - PageKind::Video { source, .. } => format!( - "", - html_escape(&page.title), - html_escape(playable(source)) - ), + PageKind::Video { source, .. } => { + let source = match crate::durable::public_address(source) { + crate::durable::Kept::Unchanged => source.clone(), + crate::durable::Kept::Changed(s) => { + output_report.videos = output_report.videos.saturating_add(1); + s + } + crate::durable::Kept::Withheld(prefix) => { + output_report.videos = output_report.videos.saturating_add(1); + output_report.withheld = output_report.withheld.saturating_add(1); + prefix.into() + } + }; + format!( + "", + html_escape(&page.title), + html_escape(playable(&source)) + ) + } _ => format!("\"{}\"", html_escape(&page.title)), }; let notes = if page.notes.is_empty() { String::new() } else { format!("", html_escape(&page.notes)) }; @@ -466,6 +502,7 @@ pub fn export_bytes(s: &mut Session, i: usize, format: &str, dpi: f64) -> Result return Err(format!("the export passed {} MB at page {}; export at a lower dpi", BUDGET >> 20, k + 1)); } } + crate::durable::notify(s, &output_report); if format == "pdf" { super::layout::pictured_notices(s, &frames_pictured, &pages_pictured); return Ok((rustgis_render::pdf::write(&pdf_pages, &name), n)); diff --git a/crates/engine/src/cmd/project.rs b/crates/engine/src/cmd/project.rs index 246fc4c40..bc6002175 100644 --- a/crates/engine/src/cmd/project.rs +++ b/crates/engine/src/cmd/project.rs @@ -11,36 +11,36 @@ pub fn commands() -> Vec { vec![ // Replacing or saving the project isn't an undoable edit: these are `.view()` commands // that manage `dirty` and the history themselves. - C::new("project.new", "New Project", "Project", new).key("Mod+N").view().params(r#"{"name"?: string}"#), - C::new("project.sample", "World Atlas Sample", "Project", sample).view(), - C::new("project.open", "Open Project", "Project", open).key("Mod+O").view().params(r#"{"path": string}"#), - C::new("project.openText", "Open Project (from text)", "Project", open_text).view().params(r#"{"text": project JSON, "fileName"?: string, "package"?: "true for package contents: never restore saved layer secrets"}"#), - C::new("project.save", "Save Project", "Project", save).key("Mod+S").view().params(r#"{"path"?: string}"#), - C::new("project.saveAs", "Save Project As", "Project", save_as).key("Mod+Shift+S").view().params(r#"{"path": string}"#), - C::new("project.rename", "Rename Project", "Project", rename).params(r#"{"name": string}"#), - C::new("edit.undo", "Undo", "Quick Access Toolbar", |s, _| if s.undo() { Ok(Value::Null) } else { Err("nothing to undo".into()) }) + C::new("project.new", "New Project", "Project", new).roles(&[]).key("Mod+N").view().params(r#"{"name"?: string}"#), + C::new("project.sample", "World Atlas Sample", "Project", sample).roles(&[]).view(), + C::new("project.open", "Open Project", "Project", open).roles(&[]).key("Mod+O").view().params(r#"{"path": string}"#), + C::new("project.openText", "Open Project (from text)", "Project", open_text).roles(&[crate::ParamRole { key: "text", role: crate::Role::Document(crate::Document::Project) }]).view().params(r#"{"text": project JSON, "fileName"?: string, "package"?: "true for package contents: never restore saved layer secrets"}"#), + C::new("project.save", "Save Project", "Project", save).roles(&[]).key("Mod+S").view().params(r#"{"path"?: string}"#), + C::new("project.saveAs", "Save Project As", "Project", save_as).roles(&[]).key("Mod+Shift+S").view().params(r#"{"path": string}"#), + C::new("project.rename", "Rename Project", "Project", rename).roles(&[]).params(r#"{"name": string}"#), + C::new("edit.undo", "Undo", "Quick Access Toolbar", |s, _| if s.undo() { Ok(Value::Null) } else { Err("nothing to undo".into()) }).roles(&[]) .key("Mod+Z") .view() .when(|s| if s.can_undo() { None } else { Some("nothing to undo") }), - C::new("edit.redo", "Redo", "Quick Access Toolbar", |s, _| if s.redo() { Ok(Value::Null) } else { Err("nothing to redo".into()) }) + C::new("edit.redo", "Redo", "Quick Access Toolbar", |s, _| if s.redo() { Ok(Value::Null) } else { Err("nothing to redo".into()) }).roles(&[]) .key("Mod+Y") .view() .when(|s| if s.can_redo() { None } else { Some("nothing to redo") }), - C::new("tool.set", "Set Map Tool", "Home › Navigate", set_tool) + C::new("tool.set", "Set Map Tool", "Home › Navigate", set_tool).roles(&[]) .view() .params(r#"{"tool": "explore"|"select"|"selectPolygon"|"selectLasso"|"selectRadius"|"zoomIn"|"measure"|"measureArea"|"createPoint"|"createLine"|"createPolygon"|"editVertices"|"cutPolygon"|"reshape"|"graphicPoint"|"graphicText"|"graphicLine"|"graphicCurve"|"graphicFreehand"|"graphicPolygon"|"graphicLasso"|"graphicRectangle"|"graphicCircle"|"graphicEllipse"|"graphicPicture"}"#), - C::new("info.project", "Project Info", "Project", info).view().params(r#"{"nameFilter"?: string, "includeFields"?: bool}"#), - C::new("info.commands", "List Commands", "Help", list).view(), + C::new("info.project", "Project Info", "Project", info).roles(&[]).view().params(r#"{"nameFilter"?: string, "includeFields"?: bool}"#), + C::new("info.commands", "List Commands", "Help", list).roles(&[]).view(), // The session log (#797): read, written out, emptied, turned on or off. It sits outside // the project, so none of these changes the project or takes an undo step. - C::new("project.log", "Session Log", "Project", log).view().params( + C::new("project.log", "Session Log", "Project", log).roles(&[]).view().params( r#"{"since"?: milliseconds since 1970 (UTC), "until"?: milliseconds, "command"?: "a command id, or the start of one (layer.)", "failed"?: bool (true: only those that failed; false: only those that succeeded), "limit"?: entries, a whole number (default 100, the newest; 1 to 1000)} → {"entries": [{"at", "command", "params", "ok", "error"?, "millis"}], "count": matching, "total": in the log, "on": bool}"#, ), - C::new("project.exportLog", "Export Session Log", "Project", export_log) + C::new("project.exportLog", "Export Session Log", "Project", export_log).roles(&[]) .view() .params(r#"{"path": ".csv, .jsonl or .ndjson file", "format"?: "csv"|"jsonl" (for a path with no extension; one given must match the extension)}"#), - C::new("project.clearLog", "Clear Session Log", "Project", clear_log).view(), - C::new("project.logCommands", "Log Commands", "Project", log_commands).view().params(r#"{"on"?: bool (default: turn it the other way)}"#), + C::new("project.clearLog", "Clear Session Log", "Project", clear_log).roles(&[]).view(), + C::new("project.logCommands", "Log Commands", "Project", log_commands).roles(&[]).view().params(r#"{"on"?: bool (default: turn it the other way)}"#), ] } @@ -148,7 +148,9 @@ pub(crate) fn reset_to(s: &mut Session, p: Project) { reset(s, p); } +/// Replaces project authority, clearing the exchange record, item trust and active connections. fn reset(s: &mut Session, p: Project) { + s.exchange = None; super::connections::clear_active(); s.database_logins.clear_activations(); s.database_logins.clear_requests(); @@ -306,13 +308,12 @@ fn open_text(s: &mut Session, p: &Value) -> CmdResult { fn write(s: &mut Session, path: PathBuf) -> CmdResult { let path = if path.extension().is_none() { path.with_extension(crate::project::EXTENSION) } else { path }; let destination = std::path::absolute(&path).map_err(|error| format!("{}: {error}", path.display()))?; - let stripped = s.project.save(&destination)?; + let report = s.project.save(&destination)?; + let stripped = report.layer_addresses(); s.mark_saved(); s.edit_counts.clear(); s.notify(format!("Saved {}", path.display()), false); - if stripped > 0 { - s.notify(format!("{stripped} layer address(es) were saved without their keys, tokens or passwords: after RustGIS closes, add those layers again to give them"), false); - } + crate::durable::notify(s, &report); s.path = Some(destination); Ok(json!({"path": path.display().to_string(), "stripped": stripped})) } diff --git a/crates/engine/src/cmd/qgis.rs b/crates/engine/src/cmd/qgis.rs index b38e74bb1..0b3d9c240 100644 --- a/crates/engine/src/cmd/qgis.rs +++ b/crates/engine/src/cmd/qgis.rs @@ -12,10 +12,10 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("project.importQgis", "Import QGIS Project", "Project › Import", import_project).params( + C::new("project.importQgis", "Import QGIS Project", "Project › Import", import_project).roles(&[]).params( r#"{"path"?: "*.qgs|*.qgz", "text"?: "a .qgs file's text", "data"?: "a .qgs or .qgz file, base64", "dir"?: "the folder its relative paths start from (default: the file's)"} → the layers added and notes on what wasn't carried over"#, ), - C::new("style.importQgis", "Import QGIS Style", "Style › Styles", import_style).when(need_layer).params( + C::new("style.importQgis", "Import QGIS Style", "Style › Styles", import_style).roles(&[]).when(need_layer).params( r#"{"layer"?, "path"?: "*.qml", "text"?: "a .qml file's text", "data"?: "a .qml file, base64"} → notes on what wasn't carried over"#, ), ] diff --git a/crates/engine/src/cmd/raster.rs b/crates/engine/src/cmd/raster.rs index 9c85fed81..8e1f34b80 100644 --- a/crates/engine/src/cmd/raster.rs +++ b/crates/engine/src/cmd/raster.rs @@ -33,12 +33,12 @@ pub(crate) fn need_raster(s: &Session) -> Option<&'static str> { pub fn commands() -> Vec { vec![ - C::new("layer.addDataBytes", "Add Data (from file contents)", "Home › Add", add_bytes) + C::new("layer.addDataBytes", "Add Data (from file contents)", "Home › Add", add_bytes).roles(&[]) .params(r#"{"fileName": "name.tif|.tiff|.asc|.geojson|.csv|.zip|.kml|.kmz|.gpx|.parquet|.fgb|.gpkg|.mbtiles|.jp2|.j2k (JPEG 2000: desktop, through GDAL)", "data": base64, "zoom"?: bool, "layer"?: "one layer of a zip, KML or GPX by name", "name"?: text, "x"?, "y"?, "wkt"?, "crs"?, "delimiter"?, "paperSpace"?, "stream"?, "as"? (as layer.addData)}"#), - C::new("raster.symbology", "Symbology", "Style › Rendering", symbology).when(need_raster).params( + C::new("raster.symbology", "Symbology", "Style › Rendering", symbology).roles(&[]).when(need_raster).params( r#"{"layer"?, "type": "stretch"|"rgb"|"classify"|"unique"|"colormap"|"shadedRelief", "band"?: 1.., "ramp"?, "invert"?: bool, "classes"?: 1..32, "method"?, "azimuth"?, "altitude"?, "zFactor"?, "grey"?: bool (shadedRelief without its colour ramp)}"#, ), - C::new("raster.stretch", "Stretch Type", "Style › Rendering", stretch) + C::new("raster.stretch", "Stretch Type", "Style › Rendering", stretch).roles(&[]) .when(need_raster) .params(r#"{"layer"?, "type": "none"|"minMax"|"custom"|"percentClip"|"standardDeviation"|"histogramEqualize", "low"?: %, "high"?: %, "n"?: deviations, "min"?, "max"?: custom bounds in the data's units, a number for every band or [red, green, blue] on an RGB layer, null for a band's own (a missing one is the range now drawn; the layer's own on one band), "like"?: raster layer (instead of type: that layer's stretch as fixed bounds, so tiles match at their joins)}"#), C::new("raster.dra", "DRA", "Style › Rendering", |s, p| { @@ -47,10 +47,10 @@ pub fn commands() -> Vec { rl.style.dra = on.unwrap_or(!rl.style.dra); Ok(()) }) - }) + }).roles(&[]) .when(need_raster) .params(r#"{"layer"?, "on"?: bool}"#), - C::new("raster.mask", "Mask Values", "Style › Rendering", mask).when(need_raster).params( + C::new("raster.mask", "Mask Values", "Style › Rendering", mask).roles(&[]).when(need_raster).params( r#"{"layer"?, "values"?: [numbers] or "0, 254, 255" (drawn as no data; [] for none), "min"?, "max"?: hide values outside this range (null for no bound), "inside"?: bool (hide those inside it instead), "clear"?: bool}"#, ), C::new("raster.resampling", "Resampling Type", "Style › Rendering", |s, p| { @@ -60,10 +60,10 @@ pub fn commands() -> Vec { rl.style.resampling = r; Ok(()) }) - }) + }).roles(&[]) .when(need_raster) .params(r#"{"layer"?, "type": "nearest"|"bilinear"}"#), - C::new("raster.bands", "Band Combination", "Style › Rendering", bands) + C::new("raster.bands", "Band Combination", "Style › Rendering", bands).roles(&[]) .when(need_raster) .params(r#"{"layer"?, "red"?: 1.., "green"?: 1.., "blue"?: 1.., "preset"?: "natural"|"colorInfrared"}"#), C::new("raster.enhance", "Enhancement", "Style › Enhancement", |s, p| { @@ -80,14 +80,14 @@ pub fn commands() -> Vec { } Ok(()) }) - }) + }).roles(&[]) .when(need_raster) .params(r#"{"layer"?, "brightness"?: -100..100, "contrast"?: -100..100, "gamma"?: 0.1..10}"#), - C::new("raster.setDefaultStyle", "Save as Default Raster Style", "Style › Rendering", set_default_style).view().params( + C::new("raster.setDefaultStyle", "Save as Default Raster Style", "Style › Rendering", set_default_style).roles(&[]).view().params( r#"{"fromLayer"?: bool (the active or named layer's colour ramp stretch; the settings apply to every raster added later that would be a grey stretch, opened files and analysis results alike), "layer"?, "ramp"?, "invert"?: bool, "stretch"?: "none"|"minMax"|"percentClip"|"standardDeviation"|"histogramEqualize", "low"?, "high"?: %, "n"?: deviations, "resampling"?: "nearest"|"bilinear", "reset"?: bool (back to the black-to-white grey stretch)}"#, ), - C::new("raster.info", "Raster Information", "Data › Information", info).view().when(need_raster).params(r#"{"layer"?}"#), - C::new("share.exportRaster", "Export Raster", "Data › Information", |s, p| super::exports::write_here(s, p, "share.exportRaster")).view().when(need_raster).params( + C::new("raster.info", "Raster Information", "Data › Information", info).roles(&[]).view().when(need_raster).params(r#"{"layer"?}"#), + C::new("share.exportRaster", "Export Raster", "Data › Information", |s, p| super::exports::write_here(s, p, "share.exportRaster")).roles(&[]).view().when(need_raster).params( r#"{"layer"?, "path": "*.tif", "band"?: 1.. (rasters that aren't colour ones; a colour raster writes its colour bands), "cog"?: bool (a cloud-optimised GeoTIFF: 512 px tiles, Deflate, overviews; default false, an untiled GeoTIFF)}"#, ), ] diff --git a/crates/engine/src/cmd/repair.rs b/crates/engine/src/cmd/repair.rs index c5b123b31..3905b2cbe 100644 --- a/crates/engine/src/cmd/repair.rs +++ b/crates/engine/src/cmd/repair.rs @@ -7,9 +7,9 @@ use serde_json::json; pub fn commands() -> Vec { vec![ - C::new("layer.repairSource", "Repair Data Source", "Data › Data Source", repair_source).params( + C::new("layer.repairSource", "Repair Data Source", "Data › Data Source", repair_source).roles(&[]).params( r#"{"layer"?: name|id (default: the active layer), "path": "the file or folder (a Zarr store, a File Geodatabase) the layer's data moved to", "table"?: "the dataset in a GeoPackage, GDAL or DuckDB file, or a raster's subdataset (default: the one the layer reads now)"}"#), - C::new("map.repairTiles", "Repair Tile Archive", "Layers", repair_tiles).params( + C::new("map.repairTiles", "Repair Tile Archive", "Layers", repair_tiles).roles(&[]).params( r#"{"name"?: text | "index"?: number (0 = top) | "id"?: catalog id, "path": "the .pmtiles or .mbtiles file the archive moved to"}"#), ] } diff --git a/crates/engine/src/cmd/report.rs b/crates/engine/src/cmd/report.rs index 95376803a..836ec8258 100644 --- a/crates/engine/src/cmd/report.rs +++ b/crates/engine/src/cmd/report.rs @@ -34,17 +34,17 @@ fn need_report(s: &Session) -> Option<&'static str> { pub fn commands() -> Vec { vec![ - C::new("report.new", "New Report", "Publish › Report", new_report).params( + C::new("report.new", "New Report", "Publish › Report", new_report).roles(&[]).params( r#"{"layer"?: name (default: the selected layer), "name"?, "title"?, "fields"?: [names] (default: all), "where"?: SQL, "selectedOnly"?: bool, "groups"?: [fields], "sort"?: field, "descending"?: bool, "page"?: "letter"|"a4"|…, "orientation"?: "portrait"|"landscape"}"#, ), - C::new("report.update", "Report Properties", "Report", update).when(need_report).params( + C::new("report.update", "Report Properties", "Report", update).roles(&[]).when(need_report).params( r#"{"report"?: name|index, "name"?, "title"?, "layer"?, "where"?, "selectedOnly"?, "sort"?: field|null, "descending"?, "page"?, "orientation"?, "width"?, "height"?: mm}"#, ), C::new("report.activate", "Open Report", "Browser", |s, p| { let i = index(s, p)?; s.active_report = Some(i); Ok(json!({"report": i})) - }) + }).roles(&[]) .view() .when(need_report) .params(r#"{"report": name|index}"#), @@ -57,7 +57,7 @@ pub fn commands() -> Vec { } s.active_report = None; Ok(Value::Null) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?}"#), // Report sections. @@ -65,7 +65,7 @@ pub fn commands() -> Vec { let title: String = ps(p, "title").unwrap_or("About this report").chars().take(500).collect(); let text: String = ps(p, "text").unwrap_or("").chars().take(100_000).collect(); add_section(s, p, Section::Page { title, text, end: b(p, "end").unwrap_or(false) }) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "title"?, "text"?, "end"?: bool (after the records; default before)}"#), C::new("report.newTemplate", "New Template", "Publish › Report", |s, p| { @@ -79,7 +79,7 @@ pub fn commands() -> Vec { s.active_layout = before; let layout = r["name"].as_str().unwrap_or_default().to_string(); add_section(s, &json!({"report": i}), Section::Layout { layout, end: b(p, "end").unwrap_or(false) }) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "name"?, "page"?, "orientation"?, "template"?: "map"|"blank", "end"?: bool}"#), C::new("report.existingTemplate", "Existing Template", "Publish › Report", |s, p| { @@ -94,7 +94,7 @@ pub fn commands() -> Vec { r["name"].as_str().unwrap_or_default().to_string() }; add_section(s, &json!({"report": i}), Section::Layout { layout, end: b(p, "end").unwrap_or(false) }) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "layout"?: a layout's name, "path"?: "*.rgislayout", "text"?: layout JSON, "data"?: base64 file, "end"?: bool}"#), C::new("report.relatedReport", "Related Report", "Publish › Report", |s, p| { @@ -106,10 +106,10 @@ pub fn commands() -> Vec { } let report = s.project.reports.get(j).map(|r| r.name.clone()).unwrap_or_default(); add_section(s, &json!({"report": i}), Section::Related { report }) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "related": report name}"#), - C::new("report.subreport", "Subreport", "Publish › Report", subreport) + C::new("report.subreport", "Subreport", "Publish › Report", subreport).roles(&[]) .when(need_report) .params(r#"{"report"?, "relationship": name (Data Design › Relationships), "fields"?: [related fields]}"#), C::new("report.removeSection", "Remove Section", "Report", |s, p| { @@ -121,7 +121,7 @@ pub fn commands() -> Vec { } r.sections.remove(k); Ok(json!({"sections": r.sections.len()})) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "index": section number}"#), // Data. @@ -130,7 +130,7 @@ pub fn commands() -> Vec { let r = s.project.reports.get_mut(i).ok_or("no report")?; toggle(&mut r.fields, field, b(p, "remove").unwrap_or(false)); Ok(json!({"fields": r.fields})) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "field": name, "remove"?: bool}"#), C::new("report.autofill", "Autofill Fields", "Data › Data Design", |s, p| { @@ -139,7 +139,7 @@ pub fn commands() -> Vec { let r = s.project.reports.get_mut(i).ok_or("no report")?; r.fields = names; Ok(json!({"fields": r.fields})) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?}"#), C::new("report.statistics", "Statistics", "Data › Data Design", |s, p| { @@ -157,7 +157,7 @@ pub fn commands() -> Vec { _ => {} } Ok(json!({"stats": r.stats.len()})) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "field": name, "kind"?: "count"|"sum"|"mean"|"min"|"max", "remove"?: bool}"#), C::new("report.group", "Group", "Data › Data Design", |s, p| { @@ -169,7 +169,7 @@ pub fn commands() -> Vec { } toggle(&mut r.groups, field, remove); Ok(json!({"groups": r.groups})) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "field": name, "remove"?: bool}"#), C::new("report.attachments", "Attachments", "Data › Data Design", |s, p| { @@ -177,7 +177,7 @@ pub fn commands() -> Vec { let r = s.project.reports.get_mut(i).ok_or("no report")?; r.attachments_field = field; Ok(json!({"field": r.attachments_field})) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "field": name of a field listing each record's attachments (paths separated by ';')|null}"#), C::new("report.dynamicPicture", "Dynamic Picture", "Data › Data Design", |s, p| { @@ -185,7 +185,7 @@ pub fn commands() -> Vec { let r = s.project.reports.get_mut(i).ok_or("no report")?; r.picture_field = field; Ok(json!({"field": r.picture_field})) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "field": name of a field holding a picture path or base64 PNG/JPEG|null}"#), C::new("report.chart", "Chart Frame", "Data › Data Design", |s, p| { @@ -204,7 +204,7 @@ pub fn commands() -> Vec { let r = s.project.reports.get_mut(i).ok_or("no report")?; r.chart = chart; Ok(json!({"chart": r.chart.is_some()})) - }) + }).roles(&[]) .when(need_report) .params(r#"{"report"?, "category": field|null (no chart), "value"?: field to sum (default: count records)}"#), // Output. @@ -220,19 +220,19 @@ pub fn commands() -> Vec { let (bytes, pages) = report_bytes(s, i, dpi, pdf)?; rustgis_data::io::write_atomic(Path::new(&path), &bytes).map_err(|e| format!("{path}: {e}"))?; Ok(json!({"path": path, "pages": pages, "bytes": bytes.len()})) - }) + }).roles(&[]) .view() .when(need_report) .params(r#"{"report"?, "path": "*.pdf|*.png" (PNG: the first page), "dpi"?: 72..600}"#), - C::new("report.save", "Report File", "Publish › Save As", |s, p| save(s, p, false)) + C::new("report.save", "Report File", "Publish › Save As", |s, p| save(s, p, false)).roles(&[]) .view() .when(need_report) .params(r#"{"report"?, "path": "*.rgisreport"}"#), - C::new("report.saveTemplate", "Report Template", "Publish › Save As", |s, p| save(s, p, true)) + C::new("report.saveTemplate", "Report Template", "Publish › Save As", |s, p| save(s, p, true)).roles(&[]) .view() .when(need_report) .params(r#"{"report"?, "path": "*.rgisrpttemplate"}"#), - C::new("report.open", "Import Report", "Publish › Report", open).params( + C::new("report.open", "Import Report", "Publish › Report", open).roles(&[]).params( r#"{"path"?: "*.rgisreport|*.rgisrpttemplate", "text"?: report JSON, "data"?: base64 file, "layer"?: the layer a template reports on (default: the selected layer)}"#, ), ] diff --git a/crates/engine/src/cmd/s3.rs b/crates/engine/src/cmd/s3.rs index 974cb13d4..5886df63d 100644 --- a/crates/engine/src/cmd/s3.rs +++ b/crates/engine/src/cmd/s3.rs @@ -9,16 +9,16 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("s3.connect", "Connect to S3", "Home › Add › S3", connect).params( + C::new("s3.connect", "Connect to S3", "Home › Add › S3", connect).roles(&[]).params( r#"{"name"?: text (default "default"), "endpoint"?: "https://… of an S3-compatible service (MinIO, Cloudflare R2, Wasabi…); AWS when absent", "region"?: text (default us-east-1), "buckets"?: [names] (the buckets it serves; others are read anonymously unless browsed through it with s3.list), "accessKeyId"?: text, "secretAccessKey"?: text, "sessionToken"?: text, "profile"?: "an AWS profile in ~/.aws/credentials and config (desktop)", "anonymous"?: bool (no credentials), "environment"?: bool (AWS_ACCESS_KEY_ID… from the environment, desktop)} — without keys, a profile or environment, the connection is anonymous; credentials are kept for the session only"#, ) .view(), - C::new("s3.disconnect", "Disconnect From S3", "Home › Add › S3", disconnect).view().params(r#"{"name": text}"#), + C::new("s3.disconnect", "Disconnect From S3", "Home › Add › S3", disconnect).roles(&[]).view().params(r#"{"name": text}"#), C::new("s3.connections", "S3 Connections", "Home › Add › S3", |_s, _p| { Ok(json!({"connections": s3::connections().iter().map(Connection::public).collect::>()})) - }) + }).roles(&[]) .view(), - C::new("s3.list", "List S3 Bucket", "Home › Add › S3", list).view().params( + C::new("s3.list", "List S3 Bucket", "Home › Add › S3", list).roles(&[]).view().params( r#"{"url"?: "s3://bucket/prefix/", "bucket"?: text, "prefix"?: text, "connection"?: "a connection's name: its buckets when no bucket is given; a bucket listed through it is served by it from then on", "limit"?: number (default 1000, at most 10000)}"#, ), ] diff --git a/crates/engine/src/cmd/search.rs b/crates/engine/src/cmd/search.rs index 9c5ec3b90..d0dee851f 100644 --- a/crates/engine/src/cmd/search.rs +++ b/crates/engine/src/cmd/search.rs @@ -15,7 +15,7 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ // Not a view command: going to a feature selects it. - C::new("map.search", "Search", "Map", search_cmd).params( + C::new("map.search", "Search", "Map", search_cmd).roles(&[]).params( r#"{"query": "a latitude and longitude (40.7, -74.0; 40°42'N 74°0'W), a layer, a bookmark or a value in a layer's table", "limit"?: 1–50, "go"?: true (the first hit) | index}"#, ), ] diff --git a/crates/engine/src/cmd/select.rs b/crates/engine/src/cmd/select.rs index c1f80a88f..d1c5eca21 100644 --- a/crates/engine/src/cmd/select.rs +++ b/crates/engine/src/cmd/select.rs @@ -16,16 +16,16 @@ mod info; pub fn commands() -> Vec { let mut commands = vec![ - C::new("select.byAttributes", "Select By Attributes", "Home › Selection", by_attributes) + C::new("select.byAttributes", "Select By Attributes", "Home › Selection", by_attributes).roles(&[]) .when(need_layers) .params(r#"{"layer"?, "where": "SQL clause", "mode"?: "new"|"add"|"remove"|"subset"}"#), - C::new("select.byRect", "Select (rectangle)", "Home › Selection", by_rect) + C::new("select.byRect", "Select (rectangle)", "Home › Selection", by_rect).roles(&[]) .params(r#"{"minLon", "minLat", "maxLon", "maxLat", "mode"?: "new"|"add"|"remove"}"#), - C::new("select.byPolygon", "Select by Polygon", "Home › Selection", by_polygon) + C::new("select.byPolygon", "Select by Polygon", "Home › Selection", by_polygon).roles(&[]) .params(r#"{"points": [[lon,lat],...], "mode"?: "new"|"add"|"remove"}"#), - C::new("select.byRadius", "Select by Radius", "Home › Selection", by_radius) + C::new("select.byRadius", "Select by Radius", "Home › Selection", by_radius).roles(&[]) .params(r#"{"lon", "lat", "radius": metres (0 < radius <= 5000000), "mode"?: "new"|"add"|"remove"}"#), - C::new("select.atPoint", "Select (click)", "Home › Selection", at_point).params(r#"{"lon", "lat", "tolerancePx"?, "mode"?}"#), + C::new("select.atPoint", "Select (click)", "Home › Selection", at_point).roles(&[]).params(r#"{"lon", "lat", "tolerancePx"?, "mode"?}"#), C::new("select.rows", "Select Rows", "Attribute Table", |s, p| { let id = s.layer_id(p)?; let rows: Vec = p @@ -35,19 +35,19 @@ pub fn commands() -> Vec { .unwrap_or_default(); set(s, id, rows, mode(p)); Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"layer"?, "rows": [number], "mode"?}"#), C::new("select.clear", "Clear", "Home › Selection", |s, _| { s.map_mut().clear_selection(); Ok(Value::Null) - }) + }).roles(&[]) .when(need_selection), C::new("select.all", "Select All", "Data › Selection", |s, p| { let id = s.layer_id(p)?; let n = s.map().layer(id).map(|l| l.dataset.features.len()).unwrap_or(0); set(s, id, (0..n).collect(), Mode::New); Ok(Value::Null) - }).params(r#"{"layer"?: name|id (default: the active layer)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id (default: the active layer)}"#) .when(need_layer), C::new("select.switch", "Switch Selection", "Data › Selection", |s, p| { let id = s.layer_id(p)?; @@ -55,7 +55,7 @@ pub fn commands() -> Vec { let rows: Vec = (0..l.dataset.features.len()).filter(|r| !l.selection.contains(r)).collect(); set(s, id, rows, Mode::New); Ok(Value::Null) - }).params(r#"{"layer"?: name|id (default: the active layer)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id (default: the active layer)}"#) .when(need_layer), C::new("select.zoomTo", "Zoom To Selection", "Home › Selection", |s, _| { let e = s.map().selection_extent(); @@ -64,7 +64,7 @@ pub fn commands() -> Vec { } s.zoom_to(&e); Ok(Value::Null) - }) + }).roles(&[]) .view() .when(need_selection), C::new("select.layerFromSelection", "Make Layer From Selected Features", "Data › Selection", |s, p| { @@ -81,12 +81,12 @@ pub fn commands() -> Vec { } let nid = super::layer::add_dataset(s, ds); Ok(json!({"layer": nid})) - }).params(r#"{"layer"?: name|id (default: the active layer, which need not be the layer last selected in)}"#) + }).roles(&[]).params(r#"{"layer"?: name|id (default: the active layer, which need not be the layer last selected in)}"#) .when(need_selection), - C::new("identify", "Identify", "Home › Inspect", identify).view().params( + C::new("identify", "Identify", "Home › Inspect", identify).roles(&[]).view().params( r#"{"lon", "lat", "tolerancePx"?, "mode"?: "topmost"|"visible"|"selectable"|"selectedInContents"|"none" (the map's pop-up mode when omitted; the mode picks the layers, and every feature under the point in them, up to 100, is in "results"; topmost and visible take vector tile layers' features too, with "tileLayer" and "sourceLayer", from the tiles drawn there: one not loaded yet is asked for, and its features come on a later identify; and they list in "featureInfo" the WMS layers drawn there to ask, each {tileLayer, url} (its GetFeatureInfo request) or {tileLayer, capabilities} (its service's, when the layer doesn't know its request): identify.featureInfo reads what they answer; selectedInContents does both for the tile layer picked in the Layers pane (map.selectTiles) alone, whichever it is, a basemap or MapLibre style too)}"#, ), - C::new("map.popupMode", "Pop-up Mode", "Home › Navigate", popup_mode) + C::new("map.popupMode", "Pop-up Mode", "Home › Navigate", popup_mode).roles(&[]) .params(r#"{"mode": "topmost"|"visible"|"selectable"|"selectedInContents"|"none"}"#), ]; commands.extend(info::commands()); diff --git a/crates/engine/src/cmd/select_info.rs b/crates/engine/src/cmd/select_info.rs index 2c831aa6c..739bdc5a2 100644 --- a/crates/engine/src/cmd/select_info.rs +++ b/crates/engine/src/cmd/select_info.rs @@ -22,7 +22,7 @@ const MAX_TEXT: usize = 2 * ogc::MAX_INFO_BYTES; pub(super) fn commands() -> Vec { vec![ - C::new("identify.featureInfo", "Identify WMS Layer", "Home › Inspect", feature_info).view().params( + C::new("identify.featureInfo", "Identify WMS Layer", "Home › Inspect", feature_info).roles(&[]).view().params( r#"{"layer": "a WMS tile layer's name", "lon"?, "lat"?: the point (for a request), "text"?: "the GetFeatureInfo answer, when already fetched (identify lists the requests)", "url"?: "the request text answers: with lon and lat, the service's error in its format gives the request in the next format the layer offers ({url}), and features answered in a later format keep the layer asking in it", "capabilities"?: "the service's capabilities document, when the layer doesn't know its request (identify then asks for it)", "request"?: bool (only say what to fetch next: {url} or {capabilities})} → results, as identify's (with "tileLayer"), or with request or after an error, what to fetch"#, ), ] diff --git a/crates/engine/src/cmd/service.rs b/crates/engine/src/cmd/service.rs index b9a2bf1e0..eb131c158 100644 --- a/crates/engine/src/cmd/service.rs +++ b/crates/engine/src/cmd/service.rs @@ -22,23 +22,29 @@ mod group; /// The total feature response bytes accepted by a grouped import or refresh. pub const MAX_GROUP_BYTES: usize = 64 << 20; +const DOCUMENT_ROLES: &[crate::ParamRole] = &[ + crate::ParamRole { key: "text", role: crate::Role::Opaque }, + crate::ParamRole { key: "documents", role: crate::Role::Opaque }, + crate::ParamRole { key: "describe", role: crate::Role::Opaque }, +]; + pub fn commands() -> Vec { let mut commands = vec![ - C::new("service.layers", "Service Layers", "Home › Add", layers).view().params( + C::new("service.layers", "Service Layers", "Home › Add", layers).roles(&[]).view().params( r#"{"url": "a WMS, WMTS or WFS address (GetCapabilities is added), an OGC API collections address or landing page (its map tiles and linked collections), or an ArcGIS FeatureServer, MapServer, ImageServer or VectorTileServer, or an ArcGIS services directory or folder (its folders and services, each listed by its own url)", "type"?: "wms"|"wmts"|"wfs"|"ogcapi"|"arcgis" (guessed from the address), "text"?: "the capabilities document, when already fetched", "documents"?: [{"url": "a discovered document address", "text": "its fetched JSON"}], "connection"?: "the saved web-service connection id whose login the requests use"} → layers with source_url, and next when supplied documents need another fetch"#, ), - C::new("service.add", "Add Service Layer", "Home › Add", add).params( + C::new("service.add", "Add Service Layer", "Home › Add", add).roles(&[]).params( r#"{"url": "as service.layers", "type"?, "layer": "the layer's name (service.layers lists them)", "name"?: text, "time"?: "a time-enabled WMS or WMTS layer: by default (or \"slider\", true) it follows the time slider, turned on with the layer's times when it is off; or one of its dates (2024-06-01), or \"default\" (false): its service's default", "text"?: "the capabilities document, when already fetched", "documents"?: "additional fetched documents, as service.layers", "source_url"?: "the listed layer's source_url", "describe"?: "a WCS coverage's DescribeCoverage document, when already fetched", "tiles"?: "an OGC API layer's tile template, when already read from its tileset (the app's)", "bbox"?: [west, south, east, north] | "view" | "full": a WCS coverage's area (the view by default); for a WFS, OGC API or ArcGIS feature layer, only the features that cross it (the whole layer, "full", by default), "connection"?: "the id of the saved web-service connection it comes from (connection.browse gives it): the layer records it, and is read with its login"}"#, ), - C::new("service.coverage", "WCS Coverage Request", "Home › Add", coverage).view().params( - r#"{"url": "a WCS address", "type"?: "wcs", "layer": "the coverage's name", "text"?: "the capabilities document, when already fetched", "documents"?: "additional fetched documents, as service.layers", "source_url"?: "the listed coverage's source_url", "describe"?: "its DescribeCoverage answer, when already fetched", "bbox"?: [west, south, east, north] | "view" (the default) | "full", "connection"?: "the saved web-service connection id whose login the request uses"} → the GetCoverage address of a GeoTIFF (layer.addUrl adds it)"#, + C::new("service.coverage", "WCS Coverage Request", "Home › Add", coverage).roles(&[]).view().params( + r#"{"url": "a WCS address", "type"?: "wcs", "layer": "the coverage's name", "text"?: "the capabilities document, when already fetched", "documents"?: "additional fetched documents, as service.layers", "source_url"?: "the listed coverage's source_url", "describe"?: "its DescribeCoverage answer, when already fetched", "bbox"?: [west, south, east, north] | "view" (the default) | "full", "connection"?: "the saved web-service connection id whose login the request uses", "anonymous"?: boolean (never use a saved login for the coverage or its description)} → the GetCoverage address of a GeoTIFF (layer.addUrl adds it)"#, ), - C::new("catalog.search", "Search Catalogue", "Home › Add", catalog_search).view().params( + C::new("catalog.search", "Search Catalogue", "Home › Add", catalog_search).roles(&[]).view().params( r#"{"url": "a catalogue's CSW address", "query"?: "words anywhere in the records", "bbox"?: [west, south, east, north] | "view" (none by default), "start"?: "the first record's position (a page's next)", "text"?: "the GetRecords answer, when already fetched", "connection"?: "as service.layers"} → records with their service and download links (service.add or layer.addUrl add them)"#, ), ]; commands.extend(group::commands()); - commands + commands.into_iter().map(|command| command.roles(DOCUMENT_ROLES)).collect() } fn url_of(p: &Value) -> Result<&str, String> { @@ -60,9 +66,9 @@ fn same_address(a: &str, b: &str) -> bool { pub(crate) fn without_token(p: &Value) -> Result, String> { use crate::render::rustgis_tiles::arcgis_auth; let Some(url) = ps(p, "url").map(str::trim) else { return Ok(None) }; - let has_token = url - .split_once('?') - .is_some_and(|(_, q)| q.split('&').any(|kv| kv.split('=').next().is_some_and(|k| k.eq_ignore_ascii_case("token")))); + let has_token = url.split_once('?').is_some_and(|(_, q)| { + q.split('&').any(|kv| kv.split('=').next().is_some_and(|k| rustgis_data::address::normalized_name(k).as_deref() == Some("token"))) + }); // ArcGIS REST addresses only: another kind of service's `token` stays in its address. let path = url.split(['?', '#']).next().unwrap_or(url).to_ascii_lowercase(); let arcgis = match ps(p, "type") { @@ -83,12 +89,18 @@ pub(crate) fn without_token(p: &Value) -> Result, String> { Ok(Some(owned)) } +/// One layer's exact capabilities document and the origin history used to reach it. +struct CapabilitySource { + address: String, + crossed_origin: bool, +} + /// Capability discovery is shared by supplied-document GUI requests and native commands. /// Sources stay alongside layers so a landing page and a collection may use the same name. #[derive(Default)] struct Discovery { service: Service, - sources: Vec, + sources: Vec, next: Option, error: Option, excluded_layers: bool, @@ -97,8 +109,9 @@ struct Discovery { } impl Discovery { - fn failed(mut self, error: String) -> Result { + fn failed(mut self, s: &mut Session, error: String) -> Result { if self.service.layers.is_empty() { + notify_withheld(s, self.service.withheld_addresses); Err(error) } else { self.error = Some(error.clone()); @@ -111,7 +124,7 @@ impl Discovery { const MAX_SERVICE_DOCUMENTS: usize = 4; -fn capabilities(p: &Value) -> Result { +fn capabilities(s: &mut Session, p: &Value) -> Result { let original = url_of(p)?; let mut address = crate::render::rustgis_tiles::logins::inherit(original, &rustgis_data::ogc::capabilities_url(original, ps(p, "type"))); @@ -130,15 +143,17 @@ fn capabilities(p: &Value) -> Result { let mut result = Discovery::default(); let mut seen = std::collections::HashSet::new(); let mut parent = original.to_string(); + let mut crossed_origin = false; loop { + crossed_origin |= crosses_origin(&parent, &address); if let Err(e) = crate::render::rustgis_tiles::address::check(&address, Some(&parent)) { - return result.failed(e); + return result.failed(s, e); } if seen.contains(&address) { - return result.failed("the service's collections links form a loop".into()); + return result.failed(s, "the service's collections links form a loop".into()); } if seen.len() >= MAX_SERVICE_DOCUMENTS { - return result.failed("the service's collections discovery exceeds four documents".into()); + return result.failed(s, "the service's collections discovery exceeds four documents".into()); } let parsed = if let Some(root) = supplied { let text = if seen.is_empty() { @@ -164,7 +179,7 @@ fn capabilities(p: &Value) -> Result { result.excluded_layers |= vetted.layers.len() != before; vetted } - Err(e) => return result.failed(e), + Err(e) => return result.failed(s, e), }; seen.insert(address.clone()); if result.service.kind.is_empty() { @@ -175,8 +190,9 @@ fn capabilities(p: &Value) -> Result { // Distinct offers may share a name (including generated tile names and collection ids). for layer in service.layers { result.service.layers.push(layer); - result.sources.push(address.clone()); + result.sources.push(CapabilitySource { address: address.clone(), crossed_origin }); } + result.service.withheld_addresses = result.service.withheld_addresses.saturating_add(service.withheld_addresses); result.service.notes.extend(service.notes); let features = result.service.layers.iter().any(|l| matches!(l.offer, Offer::Features { .. })); let tiles = result.service.layers.iter().any(|l| matches!(l.offer, Offer::Tiles { .. } | Offer::VectorTiles { .. })); @@ -249,9 +265,14 @@ pub(crate) fn mark_service_source(s: &mut Session, r: &Value, url: &str) { .filter_map(Value::as_u64) .filter_map(|i| u32::try_from(i).ok()) .collect(); + let anonymous = s.web_logins.is_anonymous(); for id in ids { if let Some(l) = s.map_mut().layer_mut(id) { l.dataset.source = Some(url.to_string()); + if anonymous { + l.anonymous = true; + l.connection.clear(); + } } } } @@ -352,30 +373,114 @@ fn checked_template(service: &str, t: &str) -> Result { return Err(format!("{t}: not a tile template")); } crate::render::rustgis_tiles::address::check(t, Some(service))?; - Ok(crate::render::rustgis_tiles::logins::inherit(service, t)) + // The resolver already chose the destination's login, including its absence after a + // foreign hop. The original service is only the address-policy context here. + Ok(t.to_string()) +} + +#[derive(Default)] +struct AdvertisedTemplate { + template: Option, + crossed_origin: bool, +} + +/// Compares canonical origins without treating a runtime login wrapper as part of the origin. +fn crosses_origin(from: &str, to: &str) -> bool { + crate::render::rustgis_tiles::logins::origin(from) != crate::render::rustgis_tiles::logins::origin(to) } /// The tile template an OGC API tileset list advertises (its WebMercatorQuad tileset's), read now -/// (the list, then the tileset when the list doesn't carry it). `None` when it can't be read. +/// (the list, then the tileset when the list doesn't carry it). Failed discovery still keeps +/// any foreign hop: a fallback must not recover the original connection either. #[cfg(not(target_arch = "wasm32"))] -fn advertised_template(list: &str, vector: bool) -> Option { +fn advertised_template(source: &str, list: &str, vector: bool, withheld: &mut u32) -> AdvertisedTemplate { + advertised_template_with(source, list, vector, withheld, crate::render::rustgis_tiles::cache::download_blocking) +} + +/// Resolves at most two advertised documents while retaining every foreign hop, even on failure. +#[cfg(not(target_arch = "wasm32"))] +fn advertised_template_with( + source: &str, + list: &str, + vector: bool, + withheld: &mut u32, + mut read: impl FnMut(&str) -> Result, String>, +) -> AdvertisedTemplate { use rustgis_data::ogc::{TilesStep, resolve_tiles}; + let mut out = AdvertisedTemplate { template: None, crossed_origin: crosses_origin(source, list) }; let mut doc = list.to_string(); for _ in 0..2 { - crate::render::rustgis_tiles::address::check(&doc, Some(list)).ok()?; - let body = crate::render::rustgis_tiles::cache::download_blocking(&doc).ok()?; - match resolve_tiles(list, &doc, &body, vector)? { - TilesStep::Template(t) => return checked_template(list, &t).ok(), - TilesStep::Next(next) => doc = crate::render::rustgis_tiles::logins::inherit(&doc, &next), + if crate::render::rustgis_tiles::address::check(&doc, Some(list)).is_err() { + break; + } + let Ok(body) = read(&doc) else { break }; + let Some(resolved) = resolve_tiles(list, &doc, &body, vector) else { break }; + *withheld = withheld.saturating_add(resolved.withheld_addresses); + match resolved.step { + TilesStep::Template(t) => { + out.crossed_origin |= crosses_origin(&doc, &t); + out.template = checked_template(list, &t).ok(); + break; + } + TilesStep::Next(next) => { + out.crossed_origin |= crosses_origin(&doc, &next); + // Resolve against the document actually fetched, never the original login. + doc = next; + } } } - None + out } /// In the browser nothing waits: the app reads the template itself and passes it as `"tiles"`. #[cfg(target_arch = "wasm32")] -fn advertised_template(_: &str, _: bool) -> Option { - None +fn advertised_template(source: &str, list: &str, _: bool, _: &mut u32) -> AdvertisedTemplate { + AdvertisedTemplate { template: None, crossed_origin: crosses_origin(source, list) } +} + +/// Adds a template resolved off the command thread through the ordinary admitted command. +/// `crossed_origin` is sticky runtime provenance, not a test for an absent login: a saved public +/// connection on a same-origin chain must remain available for a later login. +pub fn add_resolved_tiles(s: &mut Session, p: &Value, template: &str, crossed_origin: bool) -> CmdResult { + let mut p = p.clone(); + let params = p.as_object_mut().ok_or("service parameters must be an object")?; + params.insert("tiles".into(), json!(if crossed_origin { crate::render::rustgis_tiles::logins::inner(template) } else { template })); + if !crossed_origin { + return s.run("service.add", &p); + } + params.remove("connection"); + match crate::service_logins::anonymously(s, true, |s| s.run("service.add", &p)) { + Ok(r) => r, + Err(panic) => std::panic::resume_unwind(panic), + } +} + +/// Native discovery happens inside `service.add`, whose outer hook still holds its original +/// connection. Narrow the actual tile creation and tag its anonymous provenance before that +/// hook can consider login adoption. No second command admission or default User scope. +fn add_tiles(s: &mut Session, p: &Value, crossed_origin: bool) -> CmdResult { + if !crossed_origin { + return super::tiles::add(s, p); + } + let mut p = p.clone(); + if let Some(params) = p.as_object_mut() { + params.remove("connection"); + } + let result = crate::service_logins::anonymously(s, true, |s| { + let before = crate::service_logins::before(s, "map.addTiles", &p)?; + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| super::tiles::add(s, &p))); + if let Some(before) = before { + crate::service_logins::tag(s, &p, before, result.as_ref().ok().and_then(|r| r.as_ref().ok()).unwrap_or(&Value::Null))?; + } + match result { + Ok(r) => r, + Err(panic) => std::panic::resume_unwind(panic), + } + }); + match result { + Ok(r) => r, + Err(panic) => std::panic::resume_unwind(panic), + } } /// Of a layer's times, those `service.layers` lists from each end of a long list. @@ -412,7 +517,11 @@ pub fn service_json(svc: &Service) -> Value { layer }) .collect(); - json!({"kind": svc.kind, "title": svc.title, "count": layers.len(), "layers": layers, "notes": svc.notes}) + let mut notes = svc.notes.clone(); + if svc.withheld_addresses > 0 { + notes.push(rustgis_data::ogc::QUERY_WITHHELD_NOTICE.replace("{n}", &svc.withheld_addresses.to_string())); + } + json!({"kind": svc.kind, "title": svc.title, "count": layers.len(), "layers": layers, "notes": notes}) } pub(crate) fn layers(s: &mut Session, p: &Value) -> CmdResult { @@ -420,11 +529,12 @@ pub(crate) fn layers(s: &mut Session, p: &Value) -> CmdResult { let p = owned.as_ref().unwrap_or(p); let bound = crate::service_logins::request_params(s, p)?; let p = bound.as_ref(); - let found = capabilities(p)?; + let found = capabilities(s, p)?; + notify_withheld(s, found.service.withheld_addresses); let mut out = service_json(&found.service); if let Some(layers) = out.get_mut("layers").and_then(Value::as_array_mut) { for (layer, source) in layers.iter_mut().zip(found.sources) { - layer["source_url"] = json!(source); + layer_source(layer, source); } } if let Some(next) = found.next { @@ -433,6 +543,28 @@ pub(crate) fn layers(s: &mut Session, p: &Value) -> CmdResult { Ok(out) } +/// Publishes the selected document and its already-resolved feature login provenance. +fn layer_source(layer: &mut Value, source: CapabilitySource) { + let foreign_features = layer.get("offer").is_some_and(|offer| { + offer.get("type").and_then(Value::as_str) == Some("features") + && ps(offer, "url").is_some_and(|url| crosses_origin(&source.address, url)) + }); + layer["source_url"] = json!(source.address); + if source.crossed_origin || foreign_features { + layer["anonymous"] = json!(true); + if let Some(url) = layer.get_mut("offer").and_then(|o| o.get_mut("url")) + && let Some(value) = url.as_str() + { + *url = json!(crate::render::rustgis_tiles::logins::inner(value)); + } + } +} + +/// Counts destinations, not pairs; clean discovery emits no credential-forwarding notice. +fn notify_withheld(s: &mut Session, count: u32) { + s.notify_loss(rustgis_data::ogc::QUERY_WITHHELD_NOTICE, count); +} + /// Adds one of a service's layers: map layers as tiles (an overlay), a style as a style, features /// downloaded. pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { @@ -442,9 +574,23 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { let p = bound.as_ref(); let url = url_of(p)?.to_string(); let want = ps(p, "layer").map(str::trim).filter(|l| !l.is_empty()).ok_or("layer is required")?; - let found = capabilities(p)?; + let found = capabilities(s, p)?; + let mut withheld = found.service.withheld_addresses; + let result = add_discovered(s, p, &url, want, &found, &mut withheld); + notify_withheld(s, withheld); + result +} + +/// Adds from an already-read listing, accumulating resolver losses for `add`'s single notice +/// epilogue. Validation and execution errors return unchanged through that epilogue too. +fn add_discovered(s: &mut Session, p: &Value, url: &str, want: &str, found: &Discovery, withheld: &mut u32) -> CmdResult { let candidates = || { - found.service.layers.iter().zip(&found.sources).filter(|(_, source)| ps(p, "source_url").is_none_or(|s| same_address(s, source))) + found + .service + .layers + .iter() + .zip(&found.sources) + .filter(|(_, source)| ps(p, "source_url").is_none_or(|s| same_address(s, &source.address))) }; let (layer, source) = candidates() .find(|(l, _)| l.name == want) @@ -458,6 +604,8 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { None => format!("{url}: no layer '{want}' (service.layers lists them)"), } })?; + let capability_crossed = source.crossed_origin; + let source = source.address.as_str(); // A time-enabled layer follows the time slider, at each of its steps, unless a time is asked // for: one of its times (the layer named with it), or "default" (`false`), the service's. // (`"time": true`, the slider, as for a Zarr variable, `layer.addZarr`'s.) @@ -515,13 +663,17 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { // An OGC API layer's tiles: the template its tileset advertises (given, or read now), else // the usual path. let advertised = match (&layer.tilesets, ps(p, "tiles")) { - (Some(_), Some(t)) => Some(checked_template(source, t)?), - (Some(list), None) => advertised_template(list, matches!(layer.offer, Offer::VectorTiles { .. })), - _ => None, + (Some(list), Some(t)) => AdvertisedTemplate { + template: Some(checked_template(source, t)?), + crossed_origin: crosses_origin(source, list) || crosses_origin(list, t), + }, + (Some(list), None) => advertised_template(source, list, matches!(layer.offer, Offer::VectorTiles { .. }), withheld), + _ => AdvertisedTemplate::default(), }; match &layer.offer { Offer::Tiles { template } => { - let template = advertised.as_deref().unwrap_or(template); + let template = advertised.template.as_deref().unwrap_or(template); + let crossed_origin = capability_crossed || advertised.crossed_origin || crosses_origin(source, template); let template = match (&time, &slider) { // Its time written as the layer writes its times, filled at each step. (_, Some(st)) => template.replace("{Time}", &format!("{{date:{}}}", st.format)), @@ -561,17 +713,22 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { if !legends.is_empty() { add["legend"] = json!(legends); } - let mut r = super::tiles::add(s, &add)?; + let mut r = add_tiles(s, &add, crossed_origin)?; // A WMS layer's GetFeatureInfo request, for identify. let added = r["index"].as_u64().and_then(|i| usize::try_from(i).ok()); if let Some(t) = added.and_then(|i| s.map_mut().tiles.get_mut(i)) { t.feature_info = layer.feature_info.clone(); + if t.anonymous + && let Some(info) = &mut t.feature_info + { + info.url = crate::render::rustgis_tiles::logins::inner(&info.url).to_string(); + } } if let Some(at) = slider.as_ref().filter(|_| fresh).and_then(|st| st.at) { super::time::set(s, &json!({"date": at}))?; // Its address at the step the slider went to. if let Some(t) = added.and_then(|i| s.map().tiles.get(i)) { - r["url"] = json!(t.url); + r["url"] = json!(crate::sources::redact(&t.url)); } } // A slider already on keeps its steps: the service may have no map for some of them. @@ -590,10 +747,12 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { Ok(r) } Offer::VectorTiles { template, layers } => { - let template = advertised.as_deref().unwrap_or(template); - let r = super::tiles::add( + let template = advertised.template.as_deref().unwrap_or(template); + let crossed_origin = capability_crossed || advertised.crossed_origin || crosses_origin(source, template); + let r = add_tiles( s, &json!({"url": template, "kind": "vector", "name": name, "layers": layers, "overlay": true, "bounds": layer.bbox, "connection":p.get("connection")}), + crossed_origin, )?; zoom(s); Ok(r) @@ -602,9 +761,10 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { // and does not count as content (see `super::map_is_empty`): adding a layer after it frames // the map again. Documented in docs/user-guide/data-formats.md. Offer::Style { url: style } => { - let r = super::tiles::add( + let r = add_tiles( s, &json!({"url": style, "kind": "style", "name": name, "bounds": layer.bbox, "connection":p.get("connection")}), + capability_crossed || crosses_origin(source, style), )?; zoom(s); Ok(r) @@ -613,7 +773,8 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { Err(format!("{url}: '{want}' is a folder or service the directory lists: list its layers (service.layers with url {open})")) } Offer::Coverage(c) => { - let (get, _) = coverage_request(s, p, c)?; + let anonymous = capability_crossed || crosses_origin(source, &c.endpoint) || s.web_logins.is_anonymous(); + let (get, _, _) = coverage_request(s, p, c, anonymous)?; #[cfg(target_arch = "wasm32")] { let _ = get; @@ -621,11 +782,14 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { } #[cfg(not(target_arch = "wasm32"))] { - let bytes = crate::render::rustgis_tiles::cache::download_blocking(&get).map_err(|e| format!("{name}: {e}"))?; - coverage_check(&name, &bytes)?; - let r = super::raster::add_named_bytes(s, &json!({"name": name}), format!("{name}.tif"), bytes, None)?; - mark_service_source(s, &r, &get); - Ok(r) + crate::service_logins::anonymously(s, anonymous, |s| { + let bytes = crate::render::rustgis_tiles::cache::download_blocking(&get).map_err(|e| format!("{name}: {e}"))?; + coverage_check(&name, &bytes)?; + let r = super::raster::add_named_bytes(s, &json!({"name": name}), format!("{name}.tif"), bytes, None)?; + mark_service_source(s, &r, &get); + Ok(r) + }) + .unwrap_or_else(|e| std::panic::resume_unwind(e)) } } Offer::Features { url: items } => { @@ -647,29 +811,35 @@ pub(crate) fn add(s: &mut Session, p: &Value) -> CmdResult { } let max = crate::render::rustgis_tiles::cache::MAX_DOWNLOAD; let from = asked.as_deref().unwrap_or(items); - // An area request keeps the view until the requested box is applied below. - let keep_view = asked.is_some().then_some(false); - let mut r = match download_layer(s, &name, from, max, true).map_err(|e| slow_whole_layer(e, given))? { - Downloaded::Bytes(bytes) => { - let bytes = features_answer(s, &name, bytes, layer.bbox); - super::raster::add_named_bytes(s, &json!({"name": name}), format!("{name}.geojson"), bytes, keep_view)? + let crossed = capability_crossed || crosses_origin(source, from); + let from = if crossed { crate::render::rustgis_tiles::logins::inner(from) } else { from }; + let anonymous = crossed || s.web_logins.is_anonymous(); + crate::service_logins::anonymously(s, anonymous, |s| { + // An area request keeps the view until the requested box is applied below. + let keep_view = asked.is_some().then_some(false); + let mut r = match download_layer(s, &name, from, max, true).map_err(|e| slow_whole_layer(e, given))? { + Downloaded::Bytes(bytes) => { + let bytes = features_answer(s, &name, bytes, layer.bbox); + super::raster::add_named_bytes(s, &json!({"name": name}), format!("{name}.geojson"), bytes, keep_view)? + } + // Read in protocol buffer pages: their positions as the service's system + // places them, so there are no axes to check. + Downloaded::Read(read) => super::layer::add_read(s, &json!({"name": name}), *read, keep_view)?, + }; + mark_service_source(s, &r, from); + // Over an area, the features crossing it: the view stays, or goes to a box given + // (the whole layer's bounds would show a country to fill one state). + match (asked.is_some(), area, p.get("bbox")) { + (true, Some([w, so, e, n]), Some(Value::Array(_))) => s.zoom_to(&rustgis_geom::Extent::new(w, so, e, n)), + (true, ..) => {} + _ => zoom(s), } - // Read in protocol buffer pages: their positions as the service's system - // places them, so there are no axes to check. - Downloaded::Read(read) => super::layer::add_read(s, &json!({"name": name}), *read, keep_view)?, - }; - mark_service_source(s, &r, from); - // Over an area, the features crossing it: the view stays, or goes to a box given - // (the whole layer's bounds would show a country to fill one state). - match (asked.is_some(), area, p.get("bbox")) { - (true, Some([w, so, e, n]), Some(Value::Array(_))) => s.zoom_to(&rustgis_geom::Extent::new(w, so, e, n)), - (true, ..) => {} - _ => zoom(s), - } - if let (Some(_), Some(out)) = (&asked, r.as_object_mut()) { - out.insert("area".into(), json!(area)); - } - Ok(r) + if let (Some(_), Some(out)) = (&asked, r.as_object_mut()) { + out.insert("area".into(), json!(area)); + } + Ok(r) + }) + .unwrap_or_else(|e| std::panic::resume_unwind(e)) } } } @@ -714,7 +884,15 @@ fn coverage_area(s: &Session, p: &Value, info: &rustgis_data::ogc::CoverageInfo, /// A coverage's GetCoverage address and the area it covers: its description from `"describe"`, or /// fetched (waiting for it). -fn coverage_request(s: &Session, p: &Value, c: &Coverage) -> Result<(String, [f64; 4]), String> { +fn coverage_request(s: &Session, p: &Value, c: &Coverage, anonymous: bool) -> Result<(String, [f64; 4], String), String> { + let c = if anonymous { + let mut plain = c.clone(); + plain.endpoint = crate::render::rustgis_tiles::logins::inner(&c.endpoint).to_string(); + std::borrow::Cow::Owned(plain) + } else { + std::borrow::Cow::Borrowed(c) + }; + let c = c.as_ref(); let describe = crate::render::rustgis_tiles::logins::inherit(&c.endpoint, &rustgis_data::ogc::describe_coverage_url(c)); let info = match ps(p, "describe") { Some(text) => rustgis_data::ogc::parse_describe(c, text.as_bytes())?, @@ -723,7 +901,7 @@ fn coverage_request(s: &Session, p: &Value, c: &Coverage) -> Result<(String, [f6 let area = coverage_area(s, p, &info, &c.id)?; let get = crate::render::rustgis_tiles::logins::inherit(&c.endpoint, &rustgis_data::ogc::get_coverage_url(c, &info, area)); crate::render::rustgis_tiles::address::check(&get, Some(&c.endpoint))?; - Ok((get, area)) + Ok((get, area, describe)) } #[cfg(not(target_arch = "wasm32"))] @@ -744,19 +922,27 @@ fn coverage(s: &mut Session, p: &Value) -> CmdResult { let p = bound.as_ref(); let url = url_of(p)?.to_string(); let want = ps(p, "layer").map(str::trim).filter(|l| !l.is_empty()).ok_or("layer is required")?; - let found = capabilities(p)?; - let layer = found + let found = capabilities(s, p)?; + notify_withheld(s, found.service.withheld_addresses); + let (layer, source) = found .service .layers .iter() .zip(&found.sources) - .filter(|(_, source)| ps(p, "source_url").is_none_or(|s| same_address(s, source))) - .map(|(layer, _)| layer) - .find(|l| l.name == want || l.name.eq_ignore_ascii_case(want)) + .filter(|(_, source)| ps(p, "source_url").is_none_or(|s| same_address(s, &source.address))) + .find(|(l, _)| l.name == want || l.name.eq_ignore_ascii_case(want)) .ok_or_else(|| format!("{url}: no coverage '{want}' (service.layers lists them)"))?; let Offer::Coverage(c) = &layer.offer else { return Err(format!("{url}: '{want}' isn't a WCS coverage")) }; - let (get, area) = coverage_request(s, p, c)?; - Ok(json!({"url": get, "name": layer.title, "bbox": area, "describe": rustgis_data::ogc::describe_coverage_url(c)})) + let anonymous = source.crossed_origin + || crosses_origin(&source.address, &c.endpoint) + || s.web_logins.is_anonymous() + || p.get("anonymous").and_then(Value::as_bool) == Some(true); + let (get, area, describe) = coverage_request(s, p, c, anonymous)?; + let mut out = json!({"url": get, "name": layer.title, "bbox": area, "describe": describe}); + if anonymous { + out["anonymous"] = json!(true); + } + Ok(out) } /// The GetRecords request `catalog.search` makes of its parameters (the app fetches it first). @@ -832,7 +1018,83 @@ fn fetch_records(url: &str, _: &str) -> Result list B -> document A, and list A -> document B -> template A. + // The third case fails after the foreign hop and uses A's original bound fallback. + for (case, (list_origin, next_origin, fail)) in [(b, a, false), (a, b, false), (a, b, true)].into_iter().enumerate() { + let list = tiles::logins::inherit(&source, &format!("{list_origin}/tiles?f=json")); + let next = format!("{next_origin}/returned"); + let template = format!("{a}/paint/{{z}}/{{x}}/{{y}}.png"); + let mut reads = Vec::new(); + let mut withheld = 0; + let resolved = advertised_template_with(&source, &list, false, &mut withheld, |url| { + let first = reads.is_empty(); + let header = tiles::logins::apply(url).unwrap().and_then(|a| a.header); + assert_eq!(header.is_some(), first && list_origin == a); + reads.push(tiles::logins::inner(url).to_string()); + if first { + Ok(serde_json::to_vec( + &json!({"tilesets":[{"tileMatrixSetId":"WebMercatorQuad","links":[{"rel":"self","href":next}]}]}), + ) + .unwrap()) + } else if fail { + Err("fixture tileset unavailable".into()) + } else { + Ok(serde_json::to_vec(&json!({"links":[{"rel":"item","type":"image/png","href":format!("{a}/paint/{{tileMatrix}}/{{tileCol}}/{{tileRow}}.png")}]})).unwrap()) + } + }); + assert_eq!(reads, [format!("{list_origin}/tiles?f=json"), format!("{next}?f=json")]); + assert!(resolved.crossed_origin); + assert_eq!(withheld, 0); + let chosen = resolved.template.unwrap_or_else(|| tiles::logins::inherit(&source, &template)); + if !fail { + assert_eq!(chosen, template, "returning to A must not recover its binding"); + } + let p = json!({"url":chosen,"kind":if case == 1 {"vector"} else {"raster"},"name":format!("Hop {case}"),"connection":ids[0]}); + // The real native tile consumer must narrow before its outer service.add hook. + let before = web::before(&s, "service.add", &p).unwrap().unwrap(); + let out = add_tiles(&mut s, &p, resolved.crossed_origin).unwrap(); + web::tag(&mut s, &p, before, &out).unwrap(); + let tile = &s.map().tiles[out["index"].as_u64().unwrap() as usize]; + assert!(tile.anonymous && tile.connection.is_empty()); + assert_eq!(tile.url, template); + } + s.run("connection.login", &json!({"id":ids[0],"apiKey":"RGSYNTH_tiles_rotated_r16"})).unwrap(); + for tile in &s.map().tiles { + assert!(tile.anonymous && tile.connection.is_empty()); + assert!(tiles::logins::apply(&tile.url).unwrap().is_none(), "a later login must not adopt the returned template"); + } + let text = s.project.saved_text(None).unwrap().0; + let mut reopened = Session::new(Project::default()); + reopened.connections = crate::connections::Connections::from_json(&s.connections.to_json()); + reopened.run("project.openText", &json!({"text":text})).unwrap(); + reopened.run("connection.login", &json!({"id":ids[0],"apiKey":"RGSYNTH_tiles_reopened_r16"})).unwrap(); + assert!(!reopened.map().tiles.is_empty()); + for tile in &reopened.map().tiles { + assert!(tile.anonymous && tile.connection.is_empty()); + let url = tiles::template::expand(&tile.url, tiles::template::TileId { z: 0, x: 0, y: 0 }, 256, false, &[]); + assert!(tiles::logins::apply(&url).unwrap().is_none(), "rendered and reopened tiles stay anonymous"); + } + } + /// A whole layer past the wait is pointed to `bbox`, unless a `bbox` was given (one the request /// couldn't carry, a WFS 1.0's or across the antimeridian); other errors are left as they are. #[test] diff --git a/crates/engine/src/cmd/service_group.rs b/crates/engine/src/cmd/service_group.rs index 0ef03a88b..f1025a6bb 100644 --- a/crates/engine/src/cmd/service_group.rs +++ b/crates/engine/src/cmd/service_group.rs @@ -9,16 +9,16 @@ use super::{C, CmdResult, MAX_GROUP_BYTES, Session, capabilities, ps, service_js pub fn commands() -> Vec { vec![ - C::new("service.import", "Import Service", "Home › Add", import).params( + C::new("service.import", "Import Service", "Home › Add", import).roles(&[]).params( r#"{"url": "an OGC API landing page or collections address", "type"?, "name"?: "group name (default My OGC Layer)", "text"?, "documents"?: "as service.layers", "responses"?: [{"url": "features URL", "text": "downloaded GeoJSON"}], "connection"?: "as service.add", "anonymous"?: bool (internal: preserve plugin-selected service provenance)} ("group" and "layer" are rejected: use service.refresh)"#, ), - C::new("service.refresh", "Refresh", "Layers", refresh).params( + C::new("service.refresh", "Refresh", "Layers", refresh).roles(&[]).params( r#"{"group"?: id, "layer"?: "one child layer id or name", "text"?, "documents"?, "responses"?: "as service.import", "connection"?: "as service.add (default: the one its layers record)", "anonymous"?: bool (internal: do not borrow a saved login)}"#, ), - C::new("service.groupPlan", "Service Import Requests", "Layers", plan).view().params( + C::new("service.groupPlan", "Service Import Requests", "Layers", plan).roles(&[]).view().params( r#"{"url"?, "type"?, "name"?, "group"?: id, "layer"?: "one child layer", "sourceOnly"?: "return the connection without fetching", "text"?, "documents"?, "connection"?: "as service.add", "anonymous"?: bool (internal: preserve plugin-selected service provenance)} → request, layers, and next while discovery is incomplete"#, ), - C::new("service.group", "Service Group", "Layers", set_group).params( + C::new("service.group", "Service Group", "Layers", set_group).roles(&[]).params( r#"{"group"?: id, "layer"?: "a child identifying its group", "name"?: text, "expanded"?: bool, "visible"?: "set all children", "remove"?: "remove the group and its children"}"#, ), ] @@ -81,8 +81,8 @@ fn valid_connection(s: &Session, id: &str, url: &str) -> bool { }) } -fn record_connection(s: &mut Session, layer: u32, request: &Value) { - if request.get("anonymous").and_then(Value::as_bool) == Some(true) { +fn record_connection(s: &mut Session, layer: u32, request: &Value, anonymous: bool) { + if anonymous || request.get("anonymous").and_then(Value::as_bool) == Some(true) { if let Some(layer) = s.map_mut().layer_mut(layer) { layer.anonymous = true; layer.connection.clear(); @@ -107,7 +107,8 @@ fn plan(s: &mut Session, p: &Value) -> CmdResult { fn listing(s: &mut Session, p: &Value) -> CmdResult { let q = request(s, p)?; - let mut found = capabilities(&q)?; + let mut found = capabilities(s, &q)?; + super::notify_withheld(s, found.service.withheld_addresses); if let Some(error) = found.error { return Err(error); } @@ -122,14 +123,24 @@ fn listing(s: &mut Session, p: &Value) -> CmdResult { "the service lists collections RustGIS cannot read (more than 2000, or unreadable addresses); the group was not changed".into(), ); } - let child = target(s, p)?.and_then(|(g, id)| id.and_then(|id| g.children.into_iter().find(|c| c.layer == id))); + let previous = target(s, p)?; + let child = previous.as_ref().and_then(|(g, id)| id.and_then(|id| g.children.iter().find(|c| c.layer == id).cloned())); + let anonymous: std::collections::HashSet<&str> = previous + .as_ref() + .into_iter() + .flat_map(|(g, _)| &g.children) + .filter(|c| s.map().layer(c.layer).is_some_and(|l| l.anonymous)) + .map(|c| c.collection.as_str()) + .collect(); let mut identities = std::collections::HashSet::new(); let mut offered = Vec::new(); for (layer, source) in found.service.layers.into_iter().zip(found.sources) { + let mut source = source; if !matches!(layer.offer, Offer::Features { .. }) { continue; } - if !identities.insert((source.clone(), layer.name.clone())) { + source.crossed_origin |= anonymous.contains(layer.name.as_str()); + if !identities.insert((source.address.clone(), layer.name.clone())) { return Err("the service advertises duplicate collection identifiers".into()); } offered.push((layer, source)); @@ -140,7 +151,7 @@ fn listing(s: &mut Session, p: &Value) -> CmdResult { // One child: its collection at its saved source, or else the one collection with its id (a // rotated query or a moved collections document keeps the child's identity). if let Some(c) = &child { - let exact = offered.iter().position(|(l, src)| l.name == c.collection && super::same_address(src, &c.source_url)); + let exact = offered.iter().position(|(l, src)| l.name == c.collection && super::same_address(&src.address, &c.source_url)); let pos = match exact { Some(i) => Some(i), None => { @@ -162,7 +173,7 @@ fn listing(s: &mut Session, p: &Value) -> CmdResult { let mut out = service_json(&found.service); if let Some(layers) = out.get_mut("layers").and_then(Value::as_array_mut) { for (layer, source) in layers.iter_mut().zip(sources) { - layer["source_url"] = json!(source); + super::layer_source(layer, source); } } if let Some(next) = found.next { @@ -187,7 +198,7 @@ fn download(_: &mut Session, _: &str, _: &str, _: usize) -> Result, Stri Err("the browser must supply downloaded feature responses".into()) } -fn read(s: &mut Session, p: &Value) -> Result<(Value, Vec<(ServiceChild, Dataset)>), String> { +fn read(s: &mut Session, p: &Value) -> Result<(Value, Vec<(ServiceChild, Dataset, bool)>), String> { let listing = listing(s, p)?; if listing.get("next").is_some() { return Err("complete service discovery before importing or refreshing".into()); @@ -240,6 +251,7 @@ fn read(s: &mut Session, p: &Value) -> Result<(Value, Vec<(ServiceChild, Dataset source_url: ps(layer, "source_url").unwrap_or_default().to_string(), }, dataset, + layer.get("anonymous").and_then(Value::as_bool) == Some(true), )); } Ok((listing, reads)) @@ -266,10 +278,10 @@ fn import(s: &mut Session, p: &Value) -> CmdResult { } let mut children = Vec::new(); let mut extent = rustgis_geom::Extent::EMPTY; - for (mut child, dataset) in reads { + for (mut child, dataset, anonymous) in reads { child.layer = crate::cmd::layer::add_dataset(s, dataset); if let Some(request) = listing.get("request") { - record_connection(s, child.layer, request); + record_connection(s, child.layer, request, anonymous); } if let Some(layer) = s.map().layer(child.layer) { extent = extent.union(&layer.extent()); @@ -305,7 +317,7 @@ fn refresh(s: &mut Session, p: &Value) -> CmdResult { if !s.map().service_groups.iter().any(|g| g.id == old.id) { return Err("the group was removed".into()); } - let read_count = |name: &str| reads.iter().filter(|(c, _)| c.collection == name).count(); + let read_count = |name: &str| reads.iter().filter(|(c, _, _)| c.collection == name).count(); let old_count = |name: &str| old.children.iter().filter(|c| c.collection == name).count(); // A child refresh updates that child. A group refresh keeps a child whose collection is at the // same source, or else whose collection id is unique on both sides (the source address may @@ -314,7 +326,7 @@ fn refresh(s: &mut Session, p: &Value) -> CmdResult { Some(id) => reads.iter().map(|_| Some(id)).collect(), None => reads .iter() - .map(|(child, _)| { + .map(|(child, _, _)| { old.children .iter() .find(|c| c.collection == child.collection && super::same_address(&c.source_url, &child.source_url)) @@ -332,7 +344,7 @@ fn refresh(s: &mut Session, p: &Value) -> CmdResult { let mut notes = Vec::new(); let mut singled = Vec::new(); let mut moved = 0; - for ((mut child, mut dataset), existing) in reads.into_iter().zip(matches) { + for ((mut child, mut dataset, anonymous), existing) in reads.into_iter().zip(matches) { let existing = existing.and_then(|id| s.map_mut().layer_mut(id)); if let Some(layer) = existing { if dataset.geometry_type == rustgis_data::GeometryType::None { @@ -403,12 +415,12 @@ fn refresh(s: &mut Session, p: &Value) -> CmdResult { lock.retain(|((id, _, _, _), _)| *id != child.layer); } if let Some(request) = listing.get("request") { - record_connection(s, child.layer, request); + record_connection(s, child.layer, request, anonymous); } } else { child.layer = crate::cmd::layer::add_dataset(s, dataset); if let Some(request) = listing.get("request") { - record_connection(s, child.layer, request); + record_connection(s, child.layer, request, anonymous); } if hidden && let Some(layer) = s.map_mut().layer_mut(child.layer) { layer.visible = false; diff --git a/crates/engine/src/cmd/share.rs b/crates/engine/src/cmd/share.rs index efa96122f..344e5817e 100644 --- a/crates/engine/src/cmd/share.rs +++ b/crates/engine/src/cmd/share.rs @@ -19,10 +19,10 @@ pub fn commands() -> Vec { // tiles. The app runs it here, not on a thread, while one loads. super::tiles::follow_style_view(s, web_time::Duration::from_secs(30)); super::exports::write_here(s, p, "share.exportMap") - }) + }).roles(&[]) .view() .params(r#"{"path": "*.png", "width"?: px, "height"?: px, "dpi"?: 96..600}"#), - C::new("share.exportLayer", "Export Features", "Publish › Output", |s, p| super::exports::write_here(s, p, "share.exportLayer")) + C::new("share.exportLayer", "Export Features", "Publish › Output", |s, p| super::exports::write_here(s, p, "share.exportLayer")).roles(&[]) .view() .when(need_layer) .params(r#"{"layer"?, "path": "*.geojson|*.csv|*.fgb|*.gpkg|*.parquet|*.shp|*.zip|*.kml|*.kmz|*.gpx", "crs"?: "EPSG:… (or WKT) for a GeoPackage, Shapefile or FlatGeobuf (default: the system the layer was read in, else EPSG:4326)", "mode"?: "newFile (default: refuses an existing file) | addTable | replaceTable, for a GeoPackage", "tableName"?: "the GeoPackage table (default: the layer's name)"}"#), @@ -49,7 +49,7 @@ pub fn commands() -> Vec { let mut out = json!({"path": path}); report_table(s, path, &gpkg, table, &mut out); Ok(out) - }) + }).roles(&[]) .params( r#"{"table": name|id (a standalone table), "path": "*.geojson|*.csv|*.fgb|*.gpkg|*.parquet|*.shp|*.zip|*.kml|*.kmz|*.gpx", "mode"?: "newFile (default: refuses an existing file) | addTable | replaceTable, for a GeoPackage", "tableName"?: "the GeoPackage table (default: the table's name)"}"#, ) diff --git a/crates/engine/src/cmd/sharing.rs b/crates/engine/src/cmd/sharing.rs index 2dd654c9c..ae63bbab7 100644 --- a/crates/engine/src/cmd/sharing.rs +++ b/crates/engine/src/cmd/sharing.rs @@ -129,7 +129,7 @@ const MANAGE: &str = r#"{"item"?: id|name (default: the last of its kind), "map" /// Publish shared items and import models, checking file compatibility before creating a model. pub fn commands() -> Vec { let share_as = |id: &'static str, label: &'static str, run: fn(&mut Session, &Value) -> CmdResult, params: &'static str| { - C::new(id, label, "Publish › Share As", run).params(params) + C::new(id, label, "Publish › Share As", run).roles(&[]).params(params) }; vec![ share_as("share.webMap", "Web Map", web_map, r#"{"map"?: name (default: the active map), "path"?: "*.html" (default: the first data store)}"#), @@ -143,19 +143,19 @@ pub fn commands() -> Vec { C::new("share.jobs", "Jobs", "Publish › Share As", |s, _| { let items: Vec = s.project.shared.iter().rev().map(|i| serde_json::to_value(i).unwrap_or(Value::Null)).collect(); Ok(json!({"items": items, "stores": s.project.data_stores})) - }) + }).roles(&[]) .view(), - C::new("share.saveWebMap", "Save Web Map", "Publish › Manage", |s, p| update_item(s, p, "webMap", ps(p, "map"))).when(need_items).params(MANAGE), - C::new("share.replaceWebLayer", "Replace Web Layer", "Publish › Manage", |s, p| update_item(s, p, "webLayer", ps(p, "layer"))) + C::new("share.saveWebMap", "Save Web Map", "Publish › Manage", |s, p| update_item(s, p, "webMap", ps(p, "map"))).roles(&[]).when(need_items).params(MANAGE), + C::new("share.replaceWebLayer", "Replace Web Layer", "Publish › Manage", |s, p| update_item(s, p, "webLayer", ps(p, "layer"))).roles(&[]) .when(need_items) .params(r#"{"item"?: id|name (default: the last web layer), "layer"?: the layer replacing it (default: the one it was made from)}"#), - C::new("share.dataStore", "Data Stores", "Publish › Manage", data_store).params(r#"{"name"?, "path"?: folder, "remove"?: name (no parameters: list them)}"#), - C::new("share.upload", "Upload File", "Publish › Manage", upload).params(r#"{"path": file, "store"?: name (default: the first data store)}"#), - C::new("share.saveWebScene", "Save Web Scene", "Publish › Manage", |s, p| update_item(s, p, "webScene", ps(p, "map"))).when(need_items).params(MANAGE), - C::new("share.saveWebLinkChart", "Save Web Link Chart", "Publish › Manage", |s, p| update_item(s, p, "linkChart", ps(p, "relationship"))) + C::new("share.dataStore", "Data Stores", "Publish › Manage", data_store).roles(&[]).params(r#"{"name"?, "path"?: folder, "remove"?: name (no parameters: list them)}"#), + C::new("share.upload", "Upload File", "Publish › Manage", upload).roles(&[]).params(r#"{"path": file, "store"?: name (default: the first data store)}"#), + C::new("share.saveWebScene", "Save Web Scene", "Publish › Manage", |s, p| update_item(s, p, "webScene", ps(p, "map"))).roles(&[]).when(need_items).params(MANAGE), + C::new("share.saveWebLinkChart", "Save Web Link Chart", "Publish › Manage", |s, p| update_item(s, p, "linkChart", ps(p, "relationship"))).roles(&[]) .when(need_items) .params(r#"{"item"?: id|name (default: the last link chart), "relationship"?: re-share another relationship}"#), - C::new("locator.find", "Find Place", "Home › Inspect", find).params(r#"{"locator"?: shared item name or *.rgisloc path (default: the last locator), "text": search text, "limit"?: 1..100, "zoom"?: bool (go to the best match)}"#), + C::new("locator.find", "Find Place", "Home › Inspect", find).roles(&[]).params(r#"{"locator"?: shared item name or *.rgisloc path (default: the last locator), "text": search text, "limit"?: 1..100, "zoom"?: bool (go to the best match)}"#), C::new("model.import", "Import Model", "Analyze › Geoprocessing", |s, p| { let text = super::items::read_text(p)?; let v: Value = serde_json::from_str(&text).map_err(|e| format!("not a RustGIS web tool: {e}"))?; @@ -172,7 +172,7 @@ pub fn commands() -> Vec { s.notify("1 model name was trimmed, shortened, replaced or made unique during import.", false); } Ok(imported) - }) + }).roles(&[crate::ParamRole { key: "text", role: crate::Role::Document(crate::Document::Model) }]) .params(r#"{"path"?: "*.model.json" | "*.rgistool", "text"?: file JSON, "data"?: base64 file}"#), ] } @@ -211,25 +211,30 @@ pub fn share_bytes(s: &mut Session, kind: &str, p: &Value) -> Result<(String, Ve Ok((m.name.clone(), ogc_zip(&m.name, &layers, &[])?, m.name.clone())) } "webTool" => { - let ms = &s.project.models; + let ms = &mut s.project.models; let m = match ps(p, "model") { - Some(n) => ms.iter().find(|m| m.name.eq_ignore_ascii_case(n)).ok_or_else(|| format!("no model '{n}'"))?, - None => ms.last().ok_or("make a model first (ModelBuilder)")?, + Some(n) => ms.iter_mut().find(|m| m.name.eq_ignore_ascii_case(n)).ok_or_else(|| format!("no model '{n}'"))?, + None => ms.last_mut().ok_or("make a model first (ModelBuilder)")?, }; - let params: Vec = m - .steps - .iter() - .enumerate() - .flat_map(|(i, st)| { - st.params - .as_object() - .into_iter() - .flatten() - .map(move |(k, v)| json!({"step": i + 1, "tool": st.tool, "name": k, "default": v})) - }) - .collect(); - let tool = json!({"format": "RustGIS web tool", "version": crate::model::WEB_TOOL_VERSION, "name": m.name, "steps": m.steps, "parameters": params}); - Ok((m.name.clone(), serde_json::to_vec_pretty(&tool).map_err(|e| e.to_string())?, m.name.clone())) + let name = m.name.clone(); + let (bytes, report) = crate::durable::output(m, None, |m| { + let params: Vec = m + .steps + .iter() + .enumerate() + .flat_map(|(i, st)| { + st.params + .as_object() + .into_iter() + .flatten() + .map(move |(k, v)| json!({"step": i + 1, "tool": st.tool, "name": k, "default": v})) + }) + .collect(); + let tool = json!({"format": "RustGIS web tool", "version": crate::model::WEB_TOOL_VERSION, "name": m.name, "steps": m.steps, "parameters": params}); + serde_json::to_vec_pretty(&tool).map_err(|e| e.to_string()) + })?; + crate::durable::notify(s, &report); + Ok((name.clone(), bytes, name)) } "webScene" => { let mut m = map_of(s, p)?; diff --git a/crates/engine/src/cmd/sources.rs b/crates/engine/src/cmd/sources.rs index e0632eea3..6848ac603 100644 --- a/crates/engine/src/cmd/sources.rs +++ b/crates/engine/src/cmd/sources.rs @@ -10,14 +10,14 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("sources.list", "List Data Sources", "Home › Add › Data Sources", list).view(), - C::new("sources.save", "Save Data Source", "Home › Add › Data Sources", |s, p| keep(s, p, "saved")).view().params( + C::new("sources.list", "List Data Sources", "Home › Add › Data Sources", list).roles(&[]).view(), + C::new("sources.save", "Save Data Source", "Home › Add › Data Sources", |s, p| keep(s, p, "saved")).roles(&[]).view().params( r#"{"kind": "service"|"database"|"stac"|"file"|"url"|"tiles"|"zarr"|"s3", "address": "its path, web address or connection (never with a password)", "name"?: text, "extra"?: {"type"?: service type, "layer"?: text}}"#, ), - C::new("sources.favourite", "Favourite Data Source", "Home › Add › Data Sources", favourite) + C::new("sources.favourite", "Favourite Data Source", "Home › Add › Data Sources", favourite).roles(&[]) .view() .params(r#"{"kind": as sources.save, "address": text, "name"?: text, "extra"?: object, "on"?: bool (false: no longer a favourite)}"#), - C::new("sources.remove", "Remove Data Source", "Home › Add › Data Sources", remove) + C::new("sources.remove", "Remove Data Source", "Home › Add › Data Sources", remove).roles(&[]) .view() .params(r#"{"list": "saved"|"favourites"|"recent", "kind": text, "address": text}"#), ] diff --git a/crates/engine/src/cmd/sql.rs b/crates/engine/src/cmd/sql.rs index 89f133d47..b8b2f7507 100644 --- a/crates/engine/src/cmd/sql.rs +++ b/crates/engine/src/cmd/sql.rs @@ -12,10 +12,10 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("sql.query", "Run SQL Query", "Analyze › SQL", query).view().params( + C::new("sql.query", "Run SQL Query", "Analyze › SQL", query).roles(&[crate::ParamRole { key: "sql", role: crate::Role::Sql }, crate::ParamRole { key: "query", role: crate::Role::Sql }]).view().params( r#"{"sql": "one DuckDB SELECT (DuckDB Spatial's ST_ functions, read_parquet('…'), web addresses)", "database"?: "a .duckdb file or a PostgreSQL connection (postgres:host=… dbname=…) whose tables the query names", "password"?: "the connection's, kept for this session only", "limit"?: "rows returned, up to 1000 (default 100)"}"#, ), - C::new("sql.schema", "List Database Columns", "Analyze › SQL", schema) + C::new("sql.schema", "List Database Columns", "Analyze › SQL", schema).roles(&[crate::ParamRole { key: "sql", role: crate::Role::Sql }, crate::ParamRole { key: "query", role: crate::Role::Sql }]) .view() .params(r#"{"database": "a .duckdb file or a PostgreSQL connection", "password"?: string}"#), ] diff --git a/crates/engine/src/cmd/stac.rs b/crates/engine/src/cmd/stac.rs index 58aaec07f..e2bb89de1 100644 --- a/crates/engine/src/cmd/stac.rs +++ b/crates/engine/src/cmd/stac.rs @@ -11,18 +11,18 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("stac.collections", "List STAC Collections", "Home › Add › Web Data › STAC Search", collections) + C::new("stac.collections", "List STAC Collections", "Home › Add › Web Data › STAC Search", collections).roles(&[]) .view() .params(r#"{"url"?: "a STAC API (default: Earth Search, https://earth-search.aws.element84.com/v1; Planetary Computer: https://planetarycomputer.microsoft.com/api/stac/v1), or a static catalog's catalog.json or collection.json (its child catalogs and collections are listed)", "connection"?: "as service.add: the saved web-service connection whose login the request uses"}"#), - C::new("stac.search", "STAC Search", "Home › Add › Web Data › STAC Search", search).params( + C::new("stac.search", "STAC Search", "Home › Add › Web Data › STAC Search", search).roles(&[]).params( r#"{"url"?: "a STAC API (default: Earth Search), or a static catalog's catalog.json or collection.json (walked from there, at most 400 documents)", "collections"?: id | [ids] (APIs), "bbox"?: [west, south, east, north] | "view" (the map's extent), "datetime"?: "2024-06-01" | "2024-06-01/2024-06-30", "limit"?: 1–200 (default 10), "maxCloud"?: percent, "sortby"?: "eo:cloud_cover" | "-datetime" (APIs that sort), "phase"?: "pre" | "post" (static catalogs of events), "properties"?: bool (every property of each item, not only its date and cloud cover; default false), "footprints"?: bool (adds the items' footprints as a layer, default true), "connection"?: "as service.add: the saved web-service connection whose login the request uses"}"#, ), - C::new("stac.addFootprints", "Add STAC Footprints", "Home › Add › Web Data › STAC Search", add_footprints_cmd) + C::new("stac.addFootprints", "Add STAC Footprints", "Home › Add › Web Data › STAC Search", add_footprints_cmd).roles(&[]) .params(r#"{"footprints": GeoJSON FeatureCollection (items' footprints, as STAC Search finds them), "name"?: text}"#), - C::new("stac.addCube", "Add STAC Data Cube", "Home › Add › Web Data › STAC Search", add_cube).params( + C::new("stac.addCube", "Add STAC Data Cube", "Home › Add › Web Data › STAC Search", add_cube).roles(&[]).params( r#"{"url"?: a STAC API (default: Earth Search), "collection": id, "asset": key (a band, such as NDVI), "bbox"?: [west, south, east, north] | "view" (default: the map's extent), "start", "end": dates, "maxCloud"?: percent, "range"?: [low, high] (the values shown from black to white, in the asset's own units after its raster:bands scale and offset, the same for every date and tile; default: each file's 2nd–98th percentile), "name"?: text, "connection"?: "as service.add: the saved web-service connection whose login the request uses"} → a tile layer per tile of the grid the items cover, each a file per date following the time slider, which turns on over their dates when it is off (desktop)"#, ), - C::new("stac.addAsset", "Add STAC Asset", "Home › Add › Web Data › STAC Search", add_asset).params( + C::new("stac.addAsset", "Add STAC Asset", "Home › Add › Web Data › STAC Search", add_asset).roles(&[]).params( r#"{"item": a STAC item (as stac.search returns it) | "url": an item's address (Planetary Computer assets are signed), "asset"?: key (default: visual, else the first GeoTIFF, else the first data asset), "name"?: text, "connection"?: "as service.add"}"#, ), ] diff --git a/crates/engine/src/cmd/tiles.rs b/crates/engine/src/cmd/tiles.rs index 8d7193152..8c41e9a58 100644 --- a/crates/engine/src/cmd/tiles.rs +++ b/crates/engine/src/cmd/tiles.rs @@ -64,15 +64,15 @@ pub const NETWORK: &[&str] = &[ pub fn commands() -> Vec { vec![ - C::new("map.addTiles", "Add Tile Layer", "Home › Add", add).params( + C::new("map.addTiles", "Add Tile Layer", "Home › Add", add).roles(&[]).params( r#"{"url": "https://…/{z}/{x}/{y}.png" | vector tiles ".pbf" or TileJSON | MapLibre style.json | a PMTiles archive (web address or file path) | an MBTiles file path, "kind"?: "raster"|"vector"|"style"|"pmtiles"|"mbtiles" (guessed from the url), "name"?: text, "bounds"?: [west, south, east, north] (longitude/latitude coverage, when known), "attribution"?: text, "minZoom"?: number, "maxZoom"?: number, "tileSize"?: 256|512 (1024 too for a WMS or map service asked for in Web Mercator by its tiles' box: their pictures are asked for that size), "singleImage"?: bool (a WMS or map service: one picture of the view instead of tiles, so symbols and labels aren't cut at tile edges; tiles on the globe, a tilted map or a projection the service doesn't offer; by default on for one, unless tileSize is given), "tms"?: bool, "subdomains"?: [text], "layers"?: [vector tile layer names, drawn in colours of their own], "styleLayers"?: [MapLibre style layers drawing vector tiles instead (their source is the tiles')], "opacity"?: 0–1, "blend"?: "normal"|"multiply"|"screen"|"overlay"|"darken"|"lighten"|"difference" (how it combines with what is beneath), "overlay"?: bool (on top of the map's layers, as every new layer goes; without it a basemap, under them; one with the same address and role is replaced when it has the same name or the same look), "top"?: bool (on top of the map's layers, as a layer added as data goes; by default an overlay's place), "replace"?: bool (removes the other tile basemaps, not overlays), "id"?: catalog id, "view"?: bool (a MapLibre style: go to the place, zoom, bearing and pitch its root properties give, once it loads), "start"?, "end"?: dates a url with date placeholders ({date:YYYY-MM-DD}, {YYYY}) spans, turning the time slider on (or, when it is on, setting its span), "unit"?, "interval"?, "dates"?: its steps (as in time.set), "legend"?: "the web address of its legend picture (PNG or JPEG)" | [address | {"url": address, "minScale"?: scale denominator, "maxScale"?: one (the scales it is for, from minScale up to below maxScale)}] (at most 16; the Layers pane shows the one for the map's scale under the layer; service.add gives a WMS or WMTS layer its service's), "connection"?: "the id of the saved web-service connection it comes from (as service.add): the tile layer records it, and its tiles are read with its login"}"#, ), - C::new("map.removeTiles", "Remove Tile Layer", "Home › Add", remove) + C::new("map.removeTiles", "Remove Tile Layer", "Home › Add", remove).roles(&[]) .params(r#"{"name"?: text, "index"?: number, "id"?: catalog id} (removes every tile layer when none is given)"#), - C::new("map.setTiles", "Tile Layer Properties", "Home › Add", set).params( + C::new("map.setTiles", "Tile Layer Properties", "Home › Add", set).roles(&[]).params( r#"{"name"?: text | "index"?: number (0 = top) | "id"?: catalog id, "visible"?: bool, "opacity"?: 0–1, "blend"?: "normal"|"multiply"|"screen"|"overlay"|"darken"|"lighten"|"difference", "move"?: "up"|"down"|"top"|"bottom" (past the next layer in the drawing order of all the map's layers, a feature or raster layer too), "to"?: number (its place in that order among the other layers, 0 = top; the Layers pane lists it), "rename"?: text, "bands"?: [red, green, blue] | [band] (a cloud-optimised GeoTIFF's bands shown, from 1; [] for its own), "range"?: [low, high] (the values shown from black to full colour; null: each band's 2nd–98th percentile), "dequantize"?: "aef" (AlphaEarth's quantised embeddings) | "none", "legend"?: as map.addTiles's, or null for none, "hiddenLayers"?: [source layer name] (vector tiles' source layers not drawn or identified, as map.tiles's sourceLayers names them; replaces the list, [] or null for none), "singleImage"?: bool (as map.addTiles's), "tileSize"?: 256|512|1024 (a WMS or map service asked for in Web Mercator: its tiles' size)}"#, ), - C::new("map.tiles", "Tile Layers", "Home › Add", |s, _| Ok(list(s, false))).view(), + C::new("map.tiles", "Tile Layers", "Home › Add", |s, _| Ok(list(s, false))).roles(&[]).view(), // As `layer.expand` for a layer's classes: what the pane shows, not an edit to undo. C::new("map.expandTiles", "Expand Tile Legend", "Layers", |s, p| { let i = find(s, p)?.ok_or("name or index is required")?; @@ -80,20 +80,20 @@ pub fn commands() -> Vec { let t = s.map_mut().tiles.get_mut(i).ok_or("no tile layer at that index")?; t.expanded = on.unwrap_or(!t.expanded); Ok(json!({"index": i, "expanded": t.expanded})) - }) + }).roles(&[]) .params(r#"{"name"?: text | "index"?: number (0 = top) | "id"?: catalog id, "on"?: bool (toggles when omitted)} — shows or hides a tile layer's legend and source layers under it in the Layers pane"#) .view(), // As `layer.select` for a map layer: what the pane shows, not an edit to undo. - C::new("map.selectTiles", "Select Tile Layer", "Layers", select_tiles) + C::new("map.selectTiles", "Select Tile Layer", "Layers", select_tiles).roles(&[]) .params(r#"{"name"?: text | "index"?: number (0 = top) | "id"?: catalog id} — picks a tile layer in the Layers pane, leaving no map layer active (picking one ends it): the Selected in Layers pop-up mode identifies it"#) .view(), - C::new("map.zoomToTiles", "Zoom To Tile Layer", "Home › Add", zoom_to) + C::new("map.zoomToTiles", "Zoom To Tile Layer", "Home › Add", zoom_to).roles(&[]) .params(r#"{"name"?: text | "index"?: number (0 = top) | "id"?: catalog id} (advertised service bounds, or the bounds in a PMTiles archive or cloud-optimised GeoTIFF header)"#) .view(), - C::new("map.body", "Celestial Body", "Home › Add", body).params( + C::new("map.body", "Celestial Body", "Home › Add", body).roles(&[]).params( r#"{"name"?: "earth"|"moon"|"mars"|"mercury"|"venus"|"io"|"europa"|"ganymede"|"callisto"|"titan"|"pluto"|"charon" (map.bodies lists them; default: the basemap's body), "basemap"?: a planetary basemap's id (default: the body's first; USGS Imagery on the Earth) | false (adds none, keeping the web basemaps but other bodies')}"#, ), - C::new("map.bodies", "Celestial Bodies", "Home › Add", |s, _| Ok(bodies(s))).view(), + C::new("map.bodies", "Celestial Bodies", "Home › Add", |s, _| Ok(bodies(s))).roles(&[]).view(), ] } @@ -321,7 +321,7 @@ pub(crate) fn add_with_metadata_wait(s: &mut Session, p: &Value, metadata_wait: let at = if top { 0 } else { m.tiles.iter().position(|o| !o.overlay && o.over.is_none()).unwrap_or(m.tiles.len()) }; let mut out = json!({"name": t.name, "kind": t.kind.code(), "index": at}); if t.time_url.is_some() { - out["url"] = json!(t.url); + out["url"] = json!(crate::sources::redact(&t.url)); out["dated"] = json!(true); } // A file added as data (a catalog's archive, kept where the user put it, excepted) frames the diff --git a/crates/engine/src/cmd/time.rs b/crates/engine/src/cmd/time.rs index a0f5acb9a..a2f471a13 100644 --- a/crates/engine/src/cmd/time.rs +++ b/crates/engine/src/cmd/time.rs @@ -25,7 +25,7 @@ fn need_slider(s: &Session) -> Option<&'static str> { pub fn commands() -> Vec { vec![ - C::new("time.bind", "Bind to Time Slider", "View › Time", bind) + C::new("time.bind", "Bind to Time Slider", "View › Time", bind).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?, "field"?: date field (default: the one most of whose values read as dates), "start"?, "end"?: dates (default: the field's span, when the slider is new), "unit"?: "hour"|"day"|"month"|"year", "interval"?: number, "show"?: "step"|"wide"|"wider"|"cumulative"}"#), C::new("time.unbind", "Unbind from Time Slider", "View › Time", |s, p| { @@ -34,7 +34,7 @@ pub fn commands() -> Vec { let was = l.time.take().map(|t| t.field); super::zarr::follow_time(s); Ok(json!({"unbound": was})) - }) + }).roles(&[]) .when(need_feature_layer) .params(r#"{"layer"?}"#), C::new("time.fields", "Time Fields", "View › Time", |s, p| { @@ -42,11 +42,11 @@ pub fn commands() -> Vec { let l = s.map().layer(id).ok_or("no such layer")?; let fields: Vec = time_fields(&l.dataset).into_iter().map(|(n, share)| json!({"field": n, "share": share})).collect(); Ok(json!({"fields": fields, "bound": l.time.as_ref().map(|t| t.field.clone())})) - }) + }).roles(&[]) .view() .when(need_feature_layer) .params(r#"{"layer"?}"#), - C::new("time.set", "Time Slider Settings", "View › Time", set).params( + C::new("time.set", "Time Slider Settings", "View › Time", set).roles(&[]).params( r#"{"start"?, "end"?: dates (text, a year or milliseconds since 1970), "date"?: the current step, "unit"?: "hour"|"day"|"month"|"year", "interval"?: steps of that many units, "dates"?: [dates] (irregular steps; [] for regular ones), "show"?: "step"|"wide"|"wider"|"cumulative", "before"?: steps before the current one the window takes in, "after"?: steps from the current one (at least 1), "cumulative"?: bool, "speed"?: milliseconds a step shows for, "loop"?: bool, "format"?: date format (YYYY, MM, MMM, DD, HH, mm), "off"?: bool (turns the slider off, every feature showing)}"#, ), C::new("time.goTo", "Go to Date", "View › Time", |s, p| { @@ -64,7 +64,7 @@ pub fn commands() -> Vec { m.apply_time(); super::zarr::follow_time(s); Ok(info(s)) - }) + }).roles(&[]) .view() .when(need_slider) .params(r#"{"date": date | "step": number (0 = the first)}"#), @@ -76,12 +76,12 @@ pub fn commands() -> Vec { m.apply_time(); super::zarr::follow_time(s); Ok(info(s)) - }) + }).roles(&[]) .view() .when(need_slider) .params(r#"{"by"?: steps (default 1; negative goes back), "wrap"?: bool (past the end starts over)}"#), - C::new("time.info", "Time Slider", "View › Time", |s, _| Ok(info(s))).view(), - C::new("time.export", "Export Time Animation", "View › Time", export) + C::new("time.info", "Time Slider", "View › Time", |s, _| Ok(info(s))).roles(&[]).view(), + C::new("time.export", "Export Time Animation", "View › Time", export).roles(&[]) .view() .when(need_slider) .params(r#"{"path": "*.gif" | "*.zip" (PNG frames), "width"?, "height"?: px, "speed"?: milliseconds a frame shows for (default: the slider's), "from"?, "to"?: dates (default: the whole span)}"#), diff --git a/crates/engine/src/cmd/view.rs b/crates/engine/src/cmd/view.rs index ce1ae6f8c..eb4306b56 100644 --- a/crates/engine/src/cmd/view.rs +++ b/crates/engine/src/cmd/view.rs @@ -11,7 +11,7 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("map.effects", "View Effects", "View › Effects", effects).params( + C::new("map.effects", "View Effects", "View › Effects", effects).roles(&[]).params( r#"{"sunTime"?: seconds since 1970|"now"|null (illumination), "weather"?: "none"|"haze"|"fog"|"rain"|"snow", "vignette"?, "edges"?, "stars"?: bool (visual effects; stars around the globe), "tiltShift"?, "grain"?: bool (camera effects), "brightness"?, "contrast"?, "saturation"?: -100..100, "sepia"?: 0..100 (post-processing), "reset"?: bool}"#, ), C::new("map.colorVision", "Color Vision Simulator", "View › Accessibility", |s, p| { @@ -19,13 +19,13 @@ pub fn commands() -> Vec { let cv = ColorVision::from_code(m).ok_or_else(|| format!("unknown colour vision '{m}'"))?; s.map_mut().view.color_vision = cv; Ok(json!({"mode": cv.code()})) - }) + }).roles(&[]) .view() .params(r#"{"mode": "normal"|"protanopia"|"deuteranopia"|"tritanopia"|"achromatopsia"}"#), C::new("layer.priority", "Depth Priority", "View › Scene", |s, p| { let v = p.get("value").and_then(Value::as_i64).ok_or("value is required (higher draws on top)")?.clamp(-1000, 1000) as i32; with(s, p, |l| l.priority = v) - }) + }).roles(&[]) .when(need_layer) .params(r#"{"layer"?, "value": integer (higher draws on top)}"#), C::new("map.drawingMode", "Drawing Mode", "View › Scene", |s, p| { @@ -36,7 +36,7 @@ pub fn commands() -> Vec { }; s.map_mut().view.wireframe = w; Ok(json!({"wireframe": w})) - }) + }).roles(&[]) .params(r#"{"mode": "standard"|"wireframe"}"#), C::new("map.clipping", "View Clipping", "View › View Clipping", |s, p| { let d = match ps(p, "mode").unwrap_or("distance") { @@ -46,9 +46,9 @@ pub fn commands() -> Vec { }; s.map_mut().view.clip_distance = d; Ok(json!({"distance": d})) - }) + }).roles(&[]) .params(r#"{"mode": "none"|"distance", "distance"?: metres from the view's centre}"#), - C::new("map.terrain", "Terrain", "View › Scene", terrain).params( + C::new("map.terrain", "Terrain", "View › Scene", terrain).roles(&[]).params( r#"{"layer"?: a DEM raster layer (heights in metres; without layer or tiles, and no terrain yet, the active layer when it is a raster) | "tiles"?: elevation tile URL with {z}/{x}/{y}, "encoding"?: "terrarium"|"mapbox" (tiles; default terrarium), "exaggeration"?: 0 to 20 (default 1, or the terrain's), "off"?: true}"#, ), C::new("map.displayType", "Display Type", "View › Options", |s, p| { @@ -59,17 +59,17 @@ pub fn commands() -> Vec { }; s.map_mut().view.fast = fast; Ok(json!({"mode": if fast { "speed" } else { "quality" }})) - }) + }).roles(&[]) .view() .params(r#"{"mode": "quality"|"speed"}"#), C::new("map.navigator", "Navigator", "View › Navigation", |s, p| { let v = &mut s.map_mut().view; v.navigator = b(p, "on").unwrap_or(!v.navigator); Ok(json!({"navigator": v.navigator})) - }) + }).roles(&[]) .view() .params(r#"{"on"?: bool}"#), - C::new("map.camera", "Camera", "View › Navigation", camera).view().params(r#"{"lon"?, "lat"?, "scale"? (the camera's position when omitted)}"#), + C::new("map.camera", "Camera", "View › Navigation", camera).roles(&[]).view().params(r#"{"lon"?, "lat"?, "scale"? (the camera's position when omitted)}"#), C::new("map.navigation", "Navigation Mode", "Home › Navigate", |s, p| { let fp = match ps(p, "mode").ok_or("mode is required")? { "planar" => false, @@ -83,7 +83,7 @@ pub fn commands() -> Vec { m.set_projection(rustgis_map::Projection::WebMercator); } Ok(json!({"mode": if fp { "firstPerson" } else { "planar" }})) - }) + }).roles(&[]) .params(r#"{"mode": "planar"|"firstPerson"}"#), C::new("map.viewMode", "View", "View › View", |s, p| { let next = match ps(p, "mode").ok_or("mode is required")? { @@ -100,9 +100,9 @@ pub fn commands() -> Vec { let m = s.map_mut(); m.set_projection(next); Ok(json!({"projection": m.projection.code(), "label": m.projection.label()})) - }) + }).roles(&[]) .params(r#"{"mode": "global" (a globe)|"local" (a flat view centred here; not in the web app)}"#), - C::new("map.walk", "Walk", "Home › Navigate", walk).view().params(r#"{"forward"?: steps, "right"?: steps, "up"?: steps (first-person navigation)}"#), + C::new("map.walk", "Walk", "Home › Navigate", walk).roles(&[]).view().params(r#"{"forward"?: steps, "right"?: steps, "up"?: steps (first-person navigation)}"#), ] } diff --git a/crates/engine/src/cmd/webdata.rs b/crates/engine/src/cmd/webdata.rs index a045b9757..0dd8a77cd 100644 --- a/crates/engine/src/cmd/webdata.rs +++ b/crates/engine/src/cmd/webdata.rs @@ -13,12 +13,12 @@ pub fn commands() -> Vec { vec![ C::new("nationalMap.services", "List The National Map Services", "Home › Add › Web Data › United States", |_, _| { Ok(national_map_list()) - }) + }).roles(&[]) .view(), - C::new("nationalMap.add", "Add The National Map Service", "Home › Add › Web Data › United States", national_map_add).params( + C::new("nationalMap.add", "Add The National Map Service", "Home › Add › Web Data › United States", national_map_add).roles(&[]).params( r#"{"service": "a service's id or title (nationalMap.services lists them), e.g. \"basemap/USGSTopo\"", "name"?: text}"#, ), - C::new("fieldsOfTheWorld.add", "Add Fields of the World", "Home › Add › Web Data", fields_add).params( + C::new("fieldsOfTheWorld.add", "Add Fields of the World", "Home › Add › Web Data", fields_add).roles(&[]).params( r#"{"year"?: 2017–2025 (2025), "draw"?: "both"|"fields"|"density" (both: field density zoomed out, fields from zoom 9), "minScore"?: 0–100 (fields scoring below are hidden; 0)}"#, ), ] diff --git a/crates/engine/src/cmd/workflows.rs b/crates/engine/src/cmd/workflows.rs index acb182653..e116142c6 100644 --- a/crates/engine/src/cmd/workflows.rs +++ b/crates/engine/src/cmd/workflows.rs @@ -14,25 +14,25 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("gp.env", "Environments", "Analyze › Geoprocessing", environments).params( + C::new("gp.env", "Environments", "Analyze › Geoprocessing", environments).roles(&[]).params( r#"{"extent"?: "view"|"full"|layer name|[minLon, minLat, maxLon, maxLat]|null, "cellSize"?: degrees|null}"#, ), - C::new("gp.gallery", "Tool Gallery", "Analyze › Portal", gallery).view().params(r#"{"kind": "ready"|"feature"|"raster"}"#), - C::new("model.new", "ModelBuilder", "Analyze › Geoprocessing", new_model).params( + C::new("gp.gallery", "Tool Gallery", "Analyze › Portal", gallery).roles(&[]).view().params(r#"{"kind": "ready"|"feature"|"raster"}"#), + C::new("model.new", "ModelBuilder", "Analyze › Geoprocessing", new_model).roles(&[crate::ParamRole { key: "steps", role: crate::Role::ModelSteps }, crate::ParamRole { key: "graph", role: crate::Role::ModelGraph }]).params( r#"{"name"?: string, "graph"?: {"nodes": [{"id", "kind": "input"|"tool"|"output", "x", "y" (optional), an input's "layer", a tool's "tool" and "params", an output's "name"}], "edges": [{"from", "fromPort": "out", "to", "toPort": a tool's layer parameter | "in" (output)}]}, "steps"?: [{"tool": id, "params": {…, "%1" for step 1's output}}], "fromHistory"?: bool, "imported"?: bool (from a file: its tools may not touch files)}"#, ), - C::new("model.save", "Save Model", "Analyze › Geoprocessing", save_model) + C::new("model.save", "Save Model", "Analyze › Geoprocessing", save_model).roles(&[crate::ParamRole { key: "graph", role: crate::Role::ModelGraph }]) .params(r#"{"name": model (made if new), "graph": graph (as model.new), "rename"?: new name, "imported"?: bool (from a file: its tools may not touch files; a model from a file stays one)}"#), - C::new("model.run", "Run Model", "Analyze › Geoprocessing", run_model) + C::new("model.run", "Run Model", "Analyze › Geoprocessing", run_model).roles(&[crate::ParamRole { key: "graph", role: crate::Role::ModelGraph }]) .params(r#"{"name"?: model (default: the last), "graph"?: run this graph instead (as model.new)}"#), C::new("model.validate", "Validate Model", "Analyze › Geoprocessing", |s, p| { let (_, g, _) = model_graph(s, p)?; let problems = crate::model::validate(s, &g); Ok(json!({"ok": problems.is_empty(), "problems": problems})) - }) + }).roles(&[crate::ParamRole { key: "graph", role: crate::Role::ModelGraph }]) .view() .params(r#"{"name"?: model (default: the last), "graph"?: graph}"#), - C::new("model.export", "Export Model", "Analyze › Geoprocessing", export_model) + C::new("model.export", "Export Model", "Analyze › Geoprocessing", export_model).roles(&[crate::ParamRole { key: "graph", role: crate::Role::ModelGraph }]) .view() .params(r#"{"name"?: model (default: the last), "graph"?: export this graph instead, "path"?: "*.model.json" (else the file's text is returned)}"#), C::new("model.remove", "Delete Model", "Browser", |s, p| { @@ -41,33 +41,33 @@ pub fn commands() -> Vec { s.project.models.retain(|m| !m.name.eq_ignore_ascii_case(n)); s.trusted_models.remove(&n.to_ascii_lowercase()); if s.project.models.len() == before { Err(format!("no model '{n}'")) } else { Ok(Value::Null) } - }) + }).roles(&[]) .params(r#"{"name": model}"#), - C::new("script.run", "Run Script", "Analyze › Geoprocessing", run_script) + C::new("script.run", "Run Script", "Analyze › Geoprocessing", run_script).roles(&[crate::ParamRole { key: "script", role: crate::Role::Script }]) .params(r#"{"script": "one command per line: id {json}; # comments"}"#), - C::new("raster.function", "Raster Functions", "Analyze › Raster", raster_function).params( + C::new("raster.function", "Raster Functions", "Analyze › Raster", raster_function).roles(&[]).params( r#"{"layer": raster, "function": "slope"|"aspect"|"hillshade"|"ndvi"|"bandMath"|"remap"|"threshold"|"resample", "band"?: 1-based, "expression"?: "(b4-b3)/(b4+b3)", "ranges"?: [[lo, hi, value]], "value"?: number, "keep"?: bool (remap: keep unmatched cells), "nir"?, "red"?: band numbers (ndvi; default 4 and 3), "azimuth"?, "altitude"?, "zFactor"?, "target"?: raster (resample onto its grid), "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#, ), - C::new("raster.functionChain", "Function Editor", "Analyze › Raster", function_chain) + C::new("raster.functionChain", "Function Editor", "Analyze › Raster", function_chain).roles(&[]) .params(r#"{"layer": raster, "steps": [{"function", …}], "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#), - C::new("analysis.suitability", "Suitability Modeler", "Analyze › Workflows", suitability).params( + C::new("analysis.suitability", "Suitability Modeler", "Analyze › Workflows", suitability).roles(&[]).params( r#"{"criteria": [{"layer": raster, "band"?: 1-based, "ranges": [[lo, hi, score]], "weight": number}], "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#, ), - C::new("analysis.viewshed", "Visibility Analysis", "Analyze › Workflows", viewshed) + C::new("analysis.viewshed", "Visibility Analysis", "Analyze › Workflows", viewshed).roles(&[]) .params(r#"{"layer": elevation raster, "lon", "lat", "band"?: 1-based, "height"?: observer metres, "radius"?: metres, "name"?: output, "ran"?: (internal: a viewshed the app has worked out)}"#), - C::new("analysis.lineOfSight", "Line Of Sight", "Analyze › Workflows", line_of_sight).params( + C::new("analysis.lineOfSight", "Line Of Sight", "Analyze › Workflows", line_of_sight).roles(&[]).params( r#"{"layer": elevation raster, "from": [lon, lat], "to": [lon, lat], "band"?: 1-based, "observerHeight"?: m, "targetHeight"?: m, "name"?: output}"#, ), - C::new("analysis.interpolate", "Geostatistical Wizard", "Analyze › Workflows", interpolate).params( + C::new("analysis.interpolate", "Geostatistical Wizard", "Analyze › Workflows", interpolate).roles(&[]).params( r#"{"layer": points, "field": numeric field, "cellSize"?: degrees, "power"?: 2, "neighbors"?: 12, "name"?: output, "ran"?: (internal: a surface the app has worked out)}"#, ), - C::new("analysis.route", "Network Analysis", "Analyze › Workflows", route).params( + C::new("analysis.route", "Network Analysis", "Analyze › Workflows", route).roles(&[]).params( r#"{"layer": line network, "from": [lon, lat], "to": [lon, lat], "costField"?: field, "snapDistance"?: metres, "name"?: output}"#, ), - C::new("analysis.neighborhood", "Neighborhood Explorer", "Analyze › Workflows", neighborhood) + C::new("analysis.neighborhood", "Neighborhood Explorer", "Analyze › Workflows", neighborhood).roles(&[]) .view() .params(r#"{"lon", "lat", "radius": metres, "layers"?: [names], "select"?: bool}"#), - C::new("analysis.simulate", "Simulation", "Analyze › Workflows", simulate).params( + C::new("analysis.simulate", "Simulation", "Analyze › Workflows", simulate).roles(&[]).params( r#"{"layer": polygons, "count"?: points per polygon, "seed"?: number, "field"?: numeric field (count per polygon from it, scaled by "rate"), "rate"?: number, "name"?: output}"#, ), C::new("data.profile", "Data Engineering", "Analyze › Workflows", |s, p| { @@ -77,15 +77,15 @@ pub fn commands() -> Vec { .map(|f| json!({"field": f.name, "type": f.ty, "count": f.count, "nulls": f.nulls, "distinct": f.distinct, "truncated": f.truncated, "min": f.min, "max": f.max, "mean": f.mean, "top": f.top})) .collect(); Ok(json!({"layer": ds.name, "features": ds.features.len(), "fields": rows})) - }) + }).roles(&[]) .view() .params(r#"{"layer"?}"#), - C::new("data.explore", "Exploratory Analysis", "Analyze › Workflows", explore) + C::new("data.explore", "Exploratory Analysis", "Analyze › Workflows", explore).roles(&[]) .view() .params(r#"{"layer"?, "fields"?: [numeric fields], "bins"?: number}"#), - C::new("data.similar", "Embeddings", "Analyze › Workflows", similar) + C::new("data.similar", "Embeddings", "Analyze › Workflows", similar).roles(&[]) .params(r#"{"layer"?, "row"?: number (the selected feature when omitted), "fields"?: [numeric fields], "count"?: number, "select"?: bool}"#), - C::new("data.convert", "Data Interop", "Analyze › Workflows", convert) + C::new("data.convert", "Data Interop", "Analyze › Workflows", convert).roles(&[]) .view() .params(r#"{"layer"?, "format": "geojson"|"csv"|"kml"|"gpx"|"wkt", "path"?: file (the text is returned without one), "nameField"?: field}"#), ] @@ -632,17 +632,10 @@ fn save_model(s: &mut Session, p: &Value) -> CmdResult { /// A model as a `*.model.json` file: written to `path`, or its text returned. fn export_model(s: &mut Session, p: &Value) -> CmdResult { - let (name, mut g, _) = model_graph(s, p)?; - // A file outlives the session: its steps keep their addresses, not this session's logins. - let mut addresses = Vec::new(); - for node in &mut g.nodes { - crate::project::recorded_addresses(&mut node.params, &mut addresses); - if let Some(layer) = &mut node.layer { - addresses.push(layer); - } - } - crate::project::clear_login_bindings(addresses); - let text = serde_json::to_string_pretty(&crate::model::to_file(&name, &g)).map_err(|e| e.to_string())?; + let (name, g, _) = model_graph(s, p)?; + let (text, report) = + crate::durable::output_owned(g, |g| serde_json::to_string_pretty(&crate::model::to_file(&name, g)).map_err(|e| e.to_string()))?; + crate::durable::notify(s, &report); let file: String = name.chars().map(|c| if c.is_alphanumeric() || c == '-' || c == '_' { c } else { '_' }).collect(); match ps(p, "path") { Some(path) => { @@ -774,21 +767,10 @@ pub fn refused_in_script(id: &str) -> bool { pub fn script_commands(script: &str) -> Result, String> { let mut out = Vec::new(); for (n, line) in script.lines().enumerate().take(MAX_SCRIPT_LINES) { - let line = line.trim(); - if line.is_empty() || line.starts_with('#') { - continue; - } - let (id, rest) = line.split_once(char::is_whitespace).unwrap_or((line, "")); - let id = id.trim_end_matches('='); - let params: Value = if rest.trim().is_empty() { - Value::Null - } else { - serde_json::from_str(rest.trim()).map_err(|e| format!("line {}: {e}", n + 1))? - }; - if refused_in_script(id) { - return Err(format!("line {}: '{id}' can't run in a script", n + 1)); + if let Some(parsed) = crate::script_line(line) { + let (id, params) = parsed.map_err(|e| format!("line {}: {e}", n + 1))?; + out.push((n + 1, id, params)); } - out.push((n + 1, id.to_string(), params)); } Ok(out) } diff --git a/crates/engine/src/cmd/workspace.rs b/crates/engine/src/cmd/workspace.rs index 5b707550d..e237cf576 100644 --- a/crates/engine/src/cmd/workspace.rs +++ b/crates/engine/src/cmd/workspace.rs @@ -71,19 +71,19 @@ pub fn commands() -> Vec { s.sync_linked_views(); } Ok(json!({"linked": s.project.link_views})) - }) + }).roles(&[]) .params(r#"{"on"?: bool}"#), C::new("project.thumbnail", "Create Thumbnail", "Publish › Thumbnail", |s, p| { let t = thumbnail(s, p)?; s.project.thumbnail = t; Ok(json!({"thumbnail": s.project.thumbnail.is_some()})) - }) + }).roles(&[]) .params(r#"{"create"?: bool (from the map view), "image"?: base64 PNG/JPEG, "path"?: image file, "clear"?: bool}"#), C::new("map.thumbnail", "Map Thumbnail", "Publish › Thumbnail", |s, p| { let t = thumbnail(s, p)?; s.map_mut().thumbnail = t; Ok(json!({"thumbnail": s.map().thumbnail.is_some()})) - }) + }).roles(&[]) .params(r#"{"create"?: bool, "image"?: base64 PNG/JPEG, "path"?: image file, "clear"?: bool}"#), // Workflow Manager. C::new("job.new", "Workflow Manager", "View › Windows", |s, p| { @@ -105,7 +105,7 @@ pub fn commands() -> Vec { updated: t, }); Ok(json!({"job": id})) - }) + }).roles(&[]) .params(r#"{"name": string, "description"?, "assigned"?, "viewExtent"?: bool (the job covers the map view)}"#), C::new("job.update", "Update Job", "View › Windows", |s, p| { let id = p.get("job").and_then(Value::as_u64).ok_or("job is required")?; @@ -125,14 +125,14 @@ pub fn commands() -> Vec { } j.updated = now(); Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"job": id, "status"?: "new"|"in progress"|"on hold"|"done", "assigned"?, "description"?}"#), C::new("job.remove", "Delete Job", "View › Windows", |s, p| { let id = p.get("job").and_then(Value::as_u64).ok_or("job is required")?; let before = s.project.jobs.len(); s.project.jobs.retain(|j| u64::from(j.id) != id); if s.project.jobs.len() == before { Err(format!("no job {id}")) } else { Ok(Value::Null) } - }) + }).roles(&[]) .params(r#"{"job": id}"#), // Animation. C::new("animation.add", "Add Keyframe", "View › Animation", |s, p| { @@ -144,7 +144,7 @@ pub fn commands() -> Vec { } m.animation.push(k); Ok(json!({"keyframes": m.animation.len()})) - }) + }).roles(&[]) .params(r#"{"seconds"?: time from the previous keyframe}"#), C::new("animation.remove", "Remove Keyframe", "View › Animation", |s, p| { let m = s.map_mut(); @@ -159,21 +159,21 @@ pub fn commands() -> Vec { } } Ok(json!({"keyframes": m.animation.len()})) - }) + }).roles(&[]) .params(r#"{"index"?: number (the last one when omitted), "all"?: bool}"#), C::new("animation.export", "Export Animation", "View › Animation", |s, p| { let path = ps(p, "path").ok_or("path is required (a .zip of PNG frames)")?.to_string(); let bytes = animation_zip(s, p)?; rustgis_data::io::write_atomic(std::path::Path::new(&path), &bytes).map_err(|e| format!("{path}: {e}"))?; Ok(json!({"path": path, "bytes": bytes.len()})) - }) + }).roles(&[]) .view() .params(r#"{"path": "*.zip", "fps"?: 1..30, "width"?, "height"?: px}"#), // Profiles. C::new("profile.create", "Create Profile", "View › Profile Viewing", |s, p| { let pts = points(p)?; profile(s, p, &pts) - }) + }).roles(&[]) .view() .params(r#"{"layer": elevation raster, "points": [[lon, lat], …], "samples"?: number}"#), C::new("profile.autoComplete", "Auto-complete Profile", "View › Profile Viewing", |s, p| { @@ -191,7 +191,7 @@ pub fn commands() -> Vec { t *= 1.1; } profile(s, p, &[a, Coord::new(a.x + dx * t, a.y + dy * t)]) - }) + }).roles(&[]) .view() .params(r#"{"layer": elevation raster, "samples"?: number (default 200), "points": [[lon, lat], [lon, lat]] (a start and a direction)}"#), C::new("profile.crossSection", "Cross Section", "View › Profile Viewing", |s, p| { @@ -207,7 +207,7 @@ pub fn commands() -> Vec { let half_deg = f(p, "width").unwrap_or(2000.0).clamp(1.0, 1e6) / 2.0 / 111_320.0; let (nx, ny) = (-dy / len * half_deg / k, dx / len * half_deg); profile(s, p, &[Coord::new(mid.x - nx, mid.y - ny), Coord::new(mid.x + nx, mid.y + ny)]) - }) + }).roles(&[]) .view() .params( r#"{"layer": elevation raster, "samples"?: number (default 200), "points": [[lon, lat], [lon, lat]] (the line crossed), "at"?: 0..1 along it, "width"?: metres}"#, @@ -216,25 +216,25 @@ pub fn commands() -> Vec { let lines = selected_lines(s, p, false)?; let first = lines.into_iter().next().ok_or("no line")?; profile(s, p, &first) - }) + }).roles(&[]) .view() .params(r#"{"layer": elevation raster, "samples"?: number (default 200), "lines": line layer, "row"?: number (the first selected when omitted)}"#), C::new("profile.alongSelected", "Profile Along Selected Lines", "View › Profile Viewing", |s, p| { let lines = selected_lines(s, p, true)?; let out: Vec = lines.iter().take(50).map(|l| profile(s, p, l)).collect::>()?; Ok(Value::Array(out)) - }) + }).roles(&[]) .view() .params(r#"{"layer": elevation raster, "samples"?: number (default 200), "lines": line layer}"#), // Infographics and diagnostics. - C::new("info.infographic", "Infographics", "Home › Inspect", infographic) + C::new("info.infographic", "Infographics", "Home › Inspect", infographic).roles(&[]) .view() .params(r#"{"layer"? (every visible feature layer when omitted)}"#), - C::new("info.diagnostics", "Diagnostic Monitor", "Help › Performance", |s, _| Ok(diagnostics(s))).view(), + C::new("info.diagnostics", "Diagnostic Monitor", "Help › Performance", |s, _| Ok(diagnostics(s))).roles(&[]).view(), // Offline areas. - C::new("offline.download", "Download Map", "Data › Offline", offline_download) + C::new("offline.download", "Download Map", "Data › Offline", offline_download).roles(&[]) .params(r#"{"name"?: string, "extent"?: [minLon, minLat, maxLon, maxLat] (the map view when omitted)}"#), - C::new("offline.sync", "Sync", "Data › Offline", offline_sync).params(r#"{"name": offline area}"#), + C::new("offline.sync", "Sync", "Data › Offline", offline_sync).roles(&[]).params(r#"{"name": offline area}"#), C::new("offline.remove", "Remove", "Data › Offline", |s, p| { let n = ps(p, "name").ok_or("name is required")?; let before = s.project.offline_areas.len(); @@ -255,7 +255,7 @@ pub fn commands() -> Vec { } } Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name": offline area}"#), C::new("offline.open", "Open Offline Map", "Data › Offline", |s, p| { let n = ps(p, "name").ok_or("name is required")?; @@ -282,13 +282,13 @@ pub fn commands() -> Vec { let e = s.project.offline_areas.iter().find(|a| a.name.eq_ignore_ascii_case(n)).map(|a| a.extent).unwrap_or(Extent::EMPTY); s.zoom_to(&e); Ok(Value::Null) - }) + }).roles(&[]) .params(r#"{"name": offline area}"#), // Feature cache. C::new("cache.fill", "Fill Cache", "Data › Feature Cache", |s, _| { let n = fill_cache(s); Ok(json!({"cached": n})) - }) + }).roles(&[]) .view(), C::new("cache.empty", "Empty Cache", "Data › Feature Cache", |s, _| { let mut n = 0; @@ -299,17 +299,17 @@ pub fn commands() -> Vec { } } Ok(json!({"emptied": n})) - }) + }).roles(&[]) .view(), // The Shapefile index cache on disk, shared by every project (#1259). Command search, // scripts and the CLI run it; it has no ribbon button. - C::new("cache.emptyIndexes", "Empty Index Cache", "Data › Shapefile Index Cache", empty_indexes).view(), + C::new("cache.emptyIndexes", "Empty Index Cache", "Data › Shapefile Index Cache", empty_indexes).roles(&[]).view(), // A fill runs to the end in one go (it's quick); cancelling stops the automatic fills // that follow projection changes. C::new("cache.cancel", "Cancel Caching", "Data › Feature Cache", |s, _| { s.auto_cache = false; Ok(Value::Null) - }) + }).roles(&[]) .view(), C::new("cache.auto", "Auto Cache", "Data › Feature Cache", |s, p| { s.auto_cache = b(p, "on").unwrap_or(!s.auto_cache); @@ -317,7 +317,7 @@ pub fn commands() -> Vec { fill_cache(s); } Ok(json!({"autoCache": s.auto_cache})) - }) + }).roles(&[]) .view() .params(r#"{"on"?: bool}"#), ] diff --git a/crates/engine/src/cmd/zarr.rs b/crates/engine/src/cmd/zarr.rs index d1b796649..aa318964d 100644 --- a/crates/engine/src/cmd/zarr.rs +++ b/crates/engine/src/cmd/zarr.rs @@ -18,16 +18,16 @@ use crate::{CmdResult, CommandSpec as C, Session}; pub fn commands() -> Vec { vec![ - C::new("raster.zarrInfo", "Zarr Variables", "Home › Add › Add Data › Zarr", info) + C::new("raster.zarrInfo", "Zarr Variables", "Home › Add › Add Data › Zarr", info).roles(&[]) .view() .params(r#"{"path": "a Zarr store's folder" | "url": "https://… a store with consolidated metadata"}"#), - C::new("zarr.series", "Time Series at a Point", "View › Time Series", series).view().params( + C::new("zarr.series", "Time Series at a Point", "View › Time Series", series).roles(&[]).view().params( r#"{"lon", "lat", "layer"? (a Zarr layer: its store, variable and steps; default the active layer) | "path": folder | "url": "https://…", "variable"?, "index"?: {"level": 0} (the other dimensions' steps), "start"?, "end"?: dates (default: the layer's period, else the whole axis), "crs"?: for a projected store that doesn't say} → dates, values (null for no data) and labels along the variable's time dimension at the cell holding the point (desktop)"#, ), - C::new("zarr.reduce", "Summarise Zarr Over Time", "View › Time Series", reduce).params( + C::new("zarr.reduce", "Summarise Zarr Over Time", "View › Time Series", reduce).roles(&[]).params( r#"{"layer"? (a Zarr layer: its store, variable and steps; default the active layer) | "path": folder | "url": "https://…", "variable"?, "index"?, "stat": "mean"|"min"|"max"|"sum"|"std" (the population standard deviation, n)|"count"|"countAbove", "above"?: number (countAbove's threshold), "start"?, "end"?: dates (default: the layer's period, else the whole axis), "months"?: [1–12] (a season across years: [12, 1, 2]), "minValid"?: 0–1 (a cell with a smaller share of values is left empty; default 0), "name"?: text, "crs"?} → a raster layer of the statistic (desktop)"#, ), - C::new("layer.addZarr", "Add Zarr Variable", "Home › Add › Add Data", add).params( + C::new("layer.addZarr", "Add Zarr Variable", "Home › Add › Add Data", add).roles(&[]).params( r#"{"path": folder | "url": "https://…", "variable"?: "its path in the store (default: the first)", "index"?: {"time": 3, "level": 0} (other dimensions' steps, default 0), "crs"?: "EPSG:… for a store that doesn't say", "name"?: text, "time"?: bool (follow the time slider: the layer shows the step the slider is at, and turns the slider on over the variable's dates when it is off; desktop), "start"?, "end"?: dates (with time: the period to take, for an axis longer than the slider's 20,000 steps), "ran"?: (internal: a slice the app has read), "summary"?: bool (internal: with ran, a summary over time rather than a slice)}"#, ), ] @@ -671,6 +671,9 @@ fn follow(s: &mut Session, wait: bool) { #[cfg(not(target_arch = "wasm32"))] { let mut errors = Vec::new(); + // A worker may finish between layers of the same slice. Match every layer against + // one completion batch so the failure is reported once with all affected layers. + let failures = if wait { Vec::new() } else { failed_reads().lock().map(|f| f.clone()).unwrap_or_default() }; let mut ahead: Vec<(ZarrTime, usize)> = Vec::new(); let mut said_failed: Vec = Vec::new(); // The steps layers wait for off the frame: a failure kept for any other is dropped, so @@ -707,7 +710,7 @@ fn follow(s: &mut Session, wait: bool) { continue; } // Said for every layer on that step, then dropped (below). - let failed = failed_reads().lock().ok().and_then(|f| f.iter().find(|(k, _)| *k == key).map(|(_, e)| e.clone())); + let failed = failures.iter().find(|(k, _)| *k == key).map(|(_, e)| e.clone()); if failed.is_some() { said_failed.push(key.clone()); } diff --git a/crates/engine/src/durable.rs b/crates/engine/src/durable.rs new file mode 100644 index 000000000..939022141 --- /dev/null +++ b/crates/engine/src/durable.rs @@ -0,0 +1,827 @@ +//! Controlled, synchronous durable output. Preparation owns every replacement before the first +//! mutation. The edit list is also the rollback journal; applying and undoing it only swaps +//! disjoint whole fields at fixed locations. No observer, policy planner or callback may see the +//! temporary projection. Formatters are internal, borrow only the projection and never reenter. + +use std::collections::{HashSet, hash_map::RandomState}; +use std::hash::BuildHasher; +use std::sync::{Mutex, OnceLock}; + +use serde_json::Value; + +/// Counted losses at the output boundary. Generated display text is cleaned without a notice. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct Report { + pub(crate) addresses: u32, + pub(crate) videos: u32, + pub(crate) steps: u32, + pub(crate) lines: u32, + pub(crate) withheld: u32, + pub(crate) unchecked: u32, + pub(crate) not_remembered: bool, + pub(crate) lost: u32, + pub(crate) not_retained: u32, + pub(crate) executable_changes: u32, + pub(crate) reconnect: u32, +} + +impl Report { + /// The existing `project.save` result's `stripped` count, not a count of unique URLs or layers. + /// Changed address fields, including streamed provenance and source paths, count separately. + /// Explicit saves also add layers whose runtime login has no recorded connection to restore it. + pub fn layer_addresses(&self) -> u32 { + self.addresses + } + + /// Addresses whose keys the receiving session lost during peer replacement. + pub fn lost(&self) -> u32 { + self.lost + } + + /// One notice for each kind of loss, and none for a clean output. + pub fn notices(&self) -> Vec { + let mut out = Vec::new(); + for (n, text) in [ + ( + self.addresses, + "layer address(es) were saved without their keys, tokens, passwords or URL fragments; after RustGIS closes, add those layers again to restore missing values", + ), + (self.videos, "video address(es) were written without their keys, tokens, passwords or URL fragments"), + (self.steps, "executable step(s) were written without credentials or URL fragments; supply missing values again before replay"), + (self.lines, "command line(s) were written without credentials or URL fragments; supply missing values again before replay"), + (self.withheld, "credential-bearing slot(s) could not be read and were withheld"), + (self.unchecked, "step(s) contain opaque parameters that may hold credentials and were kept unchecked"), + (self.not_retained, "address(es) exceeded the bridge retention limit; their keys will not survive a round trip"), + (self.lost, "address(es) lost their keys when a peer sent back its project; add them again in the receiving session"), + (self.reconnect, "database source(s) from a peer require an explicit reconnect before reading"), + ( + self.executable_changes, + "credential-bearing executable field(s) changed when a peer sent back its project; supply missing values before replay", + ), + ] { + if n > 0 { + out.push(format!("{n} {text}")); + } + } + if self.not_remembered { + out.push("This output's credential restoration exceeds the memory limit; reopening it will not bring its keys back".into()); + } + out + } + + /// Other loss notices for a consumer that presents the layer-address count separately. + /// Suppresses that kind explicitly, independent of the ordering of notices. + pub fn notices_without_layer_addresses(&self) -> Vec { + let mut report = self.clone(); + report.addresses = 0; + report.notices() + } + + /// Adds the independently prepared losses of a nested typed document. + pub(crate) fn merge(&mut self, other: &Self) { + self.addresses = self.addresses.saturating_add(other.addresses); + self.videos = self.videos.saturating_add(other.videos); + self.steps = self.steps.saturating_add(other.steps); + self.lines = self.lines.saturating_add(other.lines); + self.withheld = self.withheld.saturating_add(other.withheld); + self.unchecked = self.unchecked.saturating_add(other.unchecked); + self.not_remembered |= other.not_remembered; + self.lost = self.lost.saturating_add(other.lost); + self.not_retained = self.not_retained.saturating_add(other.not_retained); + self.executable_changes = self.executable_changes.saturating_add(other.executable_changes); + self.reconnect = self.reconnect.saturating_add(other.reconnect); + } +} + +/// A child owner in a typed location, independent of any field's current contents. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub(crate) enum Owner { + Map(usize), + Area(usize), + Table(usize), + Task(usize), + Model(usize), + History(usize), + Notebook(usize), + Presentation(usize), + Graph, +} + +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub(crate) enum Leaf { + Map(rustgis_map::SlotPos), + Layer(rustgis_map::LayerRole), + Table, + Task(usize), + TaskCommand(usize), + ModelStep(usize), + ModelStepTool(usize), + ModelNode(usize), + ModelNodeTool(usize), + HistoryParams, + HistoryId, + HistoryCommand, + HistoryCommandId, + Message(usize), + CellSource(usize), + CellOutput(usize), + Video(usize), +} + +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub(crate) struct At { + pub(crate) owners: Vec, + pub(crate) leaf: Leaf, +} + +#[derive(Clone, Debug, PartialEq)] +enum Payload { + Text(String), + Json(Value), +} + +pub(crate) enum FieldMut<'a> { + Text(&'a mut String), + Json(&'a mut Value), +} + +impl FieldMut<'_> { + /// Type-checks before swapping; after the swap the caller immediately advances its journal. + fn swap(self, held: &mut Payload) -> bool { + match (self, held) { + (Self::Text(field), Payload::Text(value)) => std::mem::swap(field, value), + (Self::Json(field), Payload::Json(value)) => std::mem::swap(field, value), + _ => return false, + } + true + } + + /// Clones only one changed owning field, after the live projection has been rolled back. + fn payload(&self) -> Payload { + match self { + Self::Text(v) => Payload::Text((**v).clone()), + Self::Json(v) => Payload::Json((**v).clone()), + } + } + + /// Compares the actual parsed field to the recorded written value without policy dispatch. + fn equals(&self, value: &Payload) -> bool { + match (self, value) { + (Self::Text(a), Payload::Text(b)) => **a == *b, + (Self::Json(a), Payload::Json(b)) => **a == *b, + _ => false, + } + } + + /// Counts retained text/JSON bytes before copying into the bounded reopen memory. + fn bytes(&self) -> usize { + match self { + Self::Text(v) => v.len(), + Self::Json(v) => json_bytes(v), + } + } +} + +/// Counts payload without allocating a serialized copy, including JSON punctuation/escaping. +fn json_bytes(v: &Value) -> usize { + struct Counter(usize); + impl std::io::Write for Counter { + /// Counts the serializer's bytes with saturating arithmetic, retaining no payload. + fn write(&mut self, b: &[u8]) -> std::io::Result { + self.0 = self.0.saturating_add(b.len()); + Ok(b.len()) + } + /// The counting writer has no buffered bytes to flush. + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + let mut c = Counter(0); + if serde_json::to_writer(&mut c, v).is_err() { + return usize::MAX; + } + c.0 +} + +pub(crate) trait Durable { + /// Preparation is read-only, exhaustive over owning fields, and offers each leaf once. + fn prepare(&self, p: &mut Prepare<'_>); + /// Resolves recorded indices only, through plain owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option>; +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Kind { + Runtime, + Credential, + Path, + Generated, +} +struct Edit { + at: At, + held: Payload, + kind: Kind, +} + +pub(crate) struct Prepare<'a> { + owners: Vec, + edits: Vec, + pub(crate) report: Report, + relative_to: Option<&'a str>, + pub(crate) depth: usize, +} + +impl<'a> Prepare<'a> { + /// Makes every replacement and journal allocation before the live borrow is mutated. + fn new(relative_to: Option<&'a str>, depth: usize) -> Self { + Self { owners: Vec::new(), edits: Vec::new(), report: Report::default(), relative_to, depth } + } + + /// Prepares one child under a fixed, typed owning location. + pub(crate) fn child(&mut self, owner: Owner, value: &impl Durable) { + self.owners.push(owner); + value.prepare(self); + self.owners.pop(); + } + + /// Records a single replacement of a whole owning field, never a subfield of an edit. + fn edit(&mut self, leaf: Leaf, held: Payload, kind: Kind) { + self.edits.push(Edit { at: At { owners: self.owners.clone(), leaf }, held, kind }); + } + + /// Cleans an address and then makes its local path relative, in one journal entry. + pub(crate) fn string(&mut self, leaf: Leaf, value: &str, address: bool, local_path: bool) { + let mut replacement = None; + let mut kind = Kind::Path; + if address { + replacement = match public_address(value) { + Kept::Unchanged => None, + Kept::Changed(s) => Some(s), + Kept::Withheld(prefix) if prefix != value => { + self.report.withheld = self.report.withheld.saturating_add(1); + Some(prefix.into()) + } + Kept::Withheld(_) => None, + }; + if replacement.is_some() { + kind = Kind::Credential; + let binding_only = + crate::project::without_login_binding(value).is_some_and(|public| matches!(public_address(&public), Kept::Unchanged)); + if binding_only { + kind = Kind::Runtime; + } + if !binding_only || matches!(leaf, Leaf::Video(_)) { + if matches!(leaf, Leaf::Video(_)) { + self.report.videos = self.report.videos.saturating_add(1); + } else { + self.report.addresses = self.report.addresses.saturating_add(1); + } + } + } + } + let current = replacement.as_deref().unwrap_or(value); + if local_path + && let Some(dir) = self.relative_to + && rustgis_map::paths::is_absolute_local(current) + && let Some(relative) = rustgis_map::paths::relative_to(dir, current).filter(|r| rustgis_map::paths::is_local(r)) + { + replacement = Some(relative); + } + if let Some(s) = replacement.filter(|s| s != value) { + self.edit(leaf, Payload::Text(s), kind); + } + } + + /// Prepares both fields of one replayable command, counting the owning step once. + pub(crate) fn command(&mut self, leaf: Leaf, id_leaf: Leaf, id: &str, value: &Value) { + self.executable((leaf, id_leaf), id, value, crate::roles::command(id, value, self.depth)); + } + + /// Model and history tool params use the same common roles as `gp.run` params. + pub(crate) fn tool(&mut self, leaf: Leaf, id_leaf: Leaf, id: &str, value: &Value) { + self.executable((leaf, id_leaf), id, value, crate::roles::tool(id, value, self.depth)); + } + + /// Journals the identifier and params independently while reporting their shared owner. + fn executable(&mut self, (leaf, id_leaf): (Leaf, Leaf), id: &str, value: &Value, (clean, report): (Option, Report)) { + let clean = match clean { + Some(clean) => Some(crate::project::public_login_params(&clean).unwrap_or(clean)), + None => crate::project::public_login_params(value), + }; + self.report.merge(&report); + let clean_id = crate::roles::identifier(id, &mut self.report); + if clean.is_some() || clean_id.is_some() { + self.report.steps = self.report.steps.saturating_add(1); + } + if let Some(clean) = clean { + self.edit(leaf, Payload::Json(clean), Kind::Credential); + } + if let Some(clean) = clean_id { + self.edit(id_leaf, Payload::Text(clean), Kind::Credential); + } + } + + /// Rewrites only changed command lines; comments, data and foreign-language cells stay exact. + pub(crate) fn script(&mut self, leaf: Leaf, source: &str) { + let (clean, report) = crate::roles::script(source, self.depth); + let clean = match clean { + Some(clean) => Some(crate::project::without_recorded_login_bindings(&clean).unwrap_or(clean)), + None => crate::project::without_recorded_login_bindings(source), + }; + self.report.merge(&report); + if let Some(clean) = clean { + self.edit(leaf, Payload::Text(clean), Kind::Credential); + } + } + + /// Cleans only addresses in text RustGIS generated, without claiming opaque text detection. + pub(crate) fn generated(&mut self, leaf: Leaf, source: &str) { + let clean = generated_text(source); + if clean != source { + self.edit(leaf, Payload::Text(clean), Kind::Generated); + } + } +} + +/// Generated display text only: remove capability headers structurally, sanitize addresses +/// while their syntax is intact, then mask active secrets. Never use as execution or approval. +pub(crate) fn generated_text(source: &str) -> String { + let public = crate::project::without_recorded_login_bindings(source); + let source = public.as_deref().unwrap_or(source); + let (clean, _) = rustgis_data::address::scrub_addresses(source); + crate::render::rustgis_tiles::logins::scrub(&clean) +} + +struct Transaction<'a, T: Durable> { + live: &'a mut T, + edits: Vec, + applied: usize, + defect: bool, +} + +impl Transaction<'_, T> { + /// Guard is already armed. The original becomes journal-owned at each infallible swap. + fn apply(&mut self) -> Result<(), String> { + for edit in &mut self.edits { + let field = self.live.field_mut(&edit.at.owners, &edit.at.leaf).ok_or("internal output location changed")?; + if !field.swap(&mut edit.held) { + return Err("internal output field type changed".into()); + } + self.applied += 1; + } + Ok(()) + } + + /// Restores the applied prefix in reverse, with no discovery, parsing or allocation on the + /// valid path. A resolver defect is an internal error, never a successful partial output. + fn rollback(&mut self) { + while self.applied > 0 { + self.applied -= 1; + if let Some(edit) = self.edits.get_mut(self.applied) { + match self.live.field_mut(&edit.at.owners, &edit.at.leaf) { + Some(field) => self.defect |= !field.swap(&mut edit.held), + None => self.defect = true, + } + } else { + self.defect = true; + } + } + } +} + +impl Drop for Transaction<'_, T> { + /// Restores the applied journal before releasing the live borrow, including native unwinding. + fn drop(&mut self) { + self.rollback(); + } +} + +/// Rejects repeated locations before arming, so one edit can never overwrite another's original. +fn prepared<'a>(value: &impl Durable, relative_to: Option<&'a str>, depth: usize) -> Result, String> { + let mut p = Prepare::new(relative_to, depth); + value.prepare(&mut p); + let mut seen = HashSet::new(); + if p.edits.iter().any(|e| !seen.insert(&e.at)) { + return Err("internal duplicate output location".into()); + } + drop(seen); + Ok(p) +} + +/// No whole-project clone: every live field is restored on success, error and native unwind. +/// Abort/OOM/process death cannot run Drop; no temporarily edited state is persisted by them. +pub(crate) fn output( + value: &mut T, + relative_to: Option<&str>, + f: impl FnOnce(&T) -> Result, +) -> Result<(R, Report), String> { + let p = prepared(value, relative_to, 0)?; + let mut t = Transaction { live: value, edits: p.edits, applied: 0, defect: false }; + t.apply()?; + let result = f(t.live); + t.rollback(); + if t.defect { + return Err("internal output rollback failed".into()); + } + result.map(|r| (r, p.report)) +} + +/// Consumes a draft the writer already owns, after structural edits; no extra project clone. +pub(crate) fn output_owned(value: T, f: impl FnOnce(&T) -> Result) -> Result<(R, Report), String> { + output_owned_at(value, 0, f) +} + +/// Nested documents share the executable nesting budget, instead of starting a fresh one. +pub(crate) fn output_owned_at( + value: T, + depth: usize, + f: impl FnOnce(&T) -> Result, +) -> Result<(R, Report), String> { + output_owned_changed(value, depth, |value, _| f(value)) +} + +/// Typed documents serialize only when an actual field changed, independently of notices. +pub(crate) fn output_owned_changed( + mut value: T, + depth: usize, + f: impl FnOnce(&T, bool) -> Result, +) -> Result<(R, Report), String> { + let p = prepared(&value, None, depth)?; + let changed = !p.edits.is_empty(); + for mut edit in p.edits { + if !value.field_mut(&edit.at.owners, &edit.at.leaf).ok_or("internal output location changed")?.swap(&mut edit.held) { + return Err("internal output field type changed".into()); + } + } + f(&value, changed).map(|r| (r, p.report)) +} + +struct Remembered { + at: At, + written: Payload, + original: Payload, +} +struct Written { + len: usize, + hash: u64, + bytes: usize, + edits: Vec, +} +const MAX_ENTRY: usize = 1 << 20; +const MAX_TOTAL: usize = 8 << 20; +const MAX_ENTRIES: usize = 32; + +/// Process-local identity, never a public hash or a peer's authority. +fn identity(text: &str) -> (usize, u64) { + static KEY: OnceLock = OnceLock::new(); + (text.len(), KEY.get_or_init(RandomState::new).hash_one(text.as_bytes())) +} + +/// The explicit-save cache is independent of the output rollback journal. +fn kept() -> std::sync::MutexGuard<'static, Vec> { + static KEPT: OnceLock>> = OnceLock::new(); + KEPT.get_or_init(Default::default).lock().unwrap_or_else(|e| e.into_inner()) +} + +/// Remembers only credential edits after rollback, with exact written-value equality on reopen. +pub(crate) fn output_remembered( + value: &mut T, + dir: Option<&str>, + f: impl FnOnce(&T) -> Result, +) -> Result<(String, Report), String> { + let p = prepared(value, dir, 0)?; + let mut t = Transaction { live: value, edits: p.edits, applied: 0, defect: false }; + t.apply()?; + let result = f(t.live); + t.rollback(); + if t.defect { + return Err("internal output rollback failed".into()); + } + let text = result?; + let mut report = p.report; + let (len, hash) = identity(&text); + let mut bytes = 0usize; + for e in t.edits.iter().filter(|e| e.kind == Kind::Credential) { + let original = t.live.field_mut(&e.at.owners, &e.at.leaf).ok_or("internal output location changed")?; + let written = match &e.held { + Payload::Text(v) => v.len(), + Payload::Json(v) => json_bytes(v), + }; + bytes = bytes + .saturating_add(original.bytes()) + .saturating_add(written) + .saturating_add(std::mem::size_of::()) + .saturating_add(e.at.owners.len().saturating_mul(std::mem::size_of::())); + } + let mut cache = kept(); + cache.retain(|e| (e.len, e.hash) != (len, hash)); + if bytes > MAX_ENTRY { + report.not_remembered = true; + } else if bytes > 0 { + let mut edits = Vec::new(); + for e in std::mem::take(&mut t.edits).into_iter().filter(|e| e.kind == Kind::Credential) { + let mut original = t.live.field_mut(&e.at.owners, &e.at.leaf).ok_or("internal output location changed")?.payload(); + match &mut original { + Payload::Text(text) => crate::project::clear_login_bindings([text]), + Payload::Json(value) => { + if let Some(public) = crate::project::public_login_params(value) { + *value = public; + } + } + } + edits.push(Remembered { at: e.at, written: e.held, original }); + } + insert(&mut cache, Written { len, hash, bytes, edits }); + } + Ok((text, report)) +} + +/// Oldest explicit saves go first, bounded by both count and total retained payload. +fn insert(cache: &mut Vec, entry: Written) { + while cache.len() >= MAX_ENTRIES || cache.iter().fold(entry.bytes, |n, e| n.saturating_add(e.bytes)) > MAX_TOTAL { + cache.remove(0); + } + cache.push(entry); +} + +/// Runs before normalize/path resolution. Imports and peer ingress never call this function. +pub(crate) fn restore(value: &mut impl Durable, text: &str) { + let id = identity(text); + let cache = kept(); + if let Some(entry) = cache.iter().find(|e| (e.len, e.hash) == id) { + for e in &entry.edits { + if let Some(field) = value.field_mut(&e.at.owners, &e.at.leaf) + && field.equals(&e.written) + { + let mut original = e.original.clone(); + let _ = field.swap(&mut original); + } + } + } +} + +pub(crate) enum Kept { + Unchanged, + Changed(String), + Withheld(&'static str), +} + +/// Prefix recognition precedes parsing: malformed recognized wrappers never become opaque. +pub(crate) fn public_address(address: &str) -> Kept { + public_address_at(address, 0) +} + +/// Wrapper recursion is bounded independently of executable document nesting. +fn public_address_at(address: &str, depth: usize) -> Kept { + use rustgis_data::{ + address::{Public, public_web_address}, + s3_address, + }; + if let Some((mut public, _)) = rustgis_data::source::myconn::split_password(address) { + rustgis_data::address::redact_query(&mut public); + return if public == address { Kept::Unchanged } else { Kept::Changed(public) }; + } + if let Some((public, _)) = rustgis_data::source::msconn::split_password(address) { + return if public == address { Kept::Unchanged } else { Kept::Changed(public) }; + } + let text = address.trim_start(); + let lower = text.to_ascii_lowercase(); + let recognized = [ + "rustgis-login:", + "rustgis-anonymous:", + "rustgis-s3:", + "pmtiles:", + "cog:", + "postgresql:", + "postgres:", + "pg:", + "https:", + "http:", + "s3:", + ] + .into_iter() + .find(|p| lower.starts_with(p)); + let Some(prefix) = recognized else { return Kept::Unchanged }; + if depth >= 8 { + return Kept::Withheld(prefix); + } + if prefix == "rustgis-login:" { + let Ok(Some(bound)) = rustgis_data::login_address::parse(text) else { return Kept::Withheld(prefix) }; + return match public_address_at(bound.url, depth + 1) { + Kept::Unchanged => Kept::Changed(bound.url.to_string()), + other => other, + }; + } + if prefix == "rustgis-anonymous:" { + let Some(inner) = text.get(prefix.len()..) else { return Kept::Withheld(prefix) }; + return match public_address_at(inner, depth + 1) { + Kept::Unchanged => Kept::Unchanged, + Kept::Changed(s) => Kept::Changed(format!("{prefix}{s}")), + Kept::Withheld(_) => Kept::Withheld(prefix), + }; + } + if matches!(prefix, "pmtiles:" | "cog:") { + let Some((head, archive)) = crate::sources::template_archive(text) else { return Kept::Withheld(prefix) }; + let file = crate::render::rustgis_tiles::cog::split(archive).0; + let look = archive.get(file.len()..).unwrap_or(""); + return match public_address_at(file, depth + 1) { + Kept::Unchanged => Kept::Unchanged, + Kept::Changed(s) => Kept::Changed(format!("{head}{s}{look}")), + Kept::Withheld(_) => Kept::Withheld(prefix), + }; + } + if prefix == "rustgis-s3:" { + let file = crate::render::rustgis_tiles::cog::split(text).0; + let look = text.get(file.len()..).unwrap_or(""); + let Ok(Some(b)) = s3_address::parse(file) else { return Kept::Withheld(prefix) }; + let endpoint = public_address_at(b.endpoint, depth + 1); + let url = public_address_at(b.url, depth + 1); + if matches!(endpoint, Kept::Withheld(_)) || matches!(url, Kept::Withheld(_)) { + return Kept::Withheld(prefix); + } + if matches!((&endpoint, &url), (Kept::Unchanged, Kept::Unchanged)) { + return Kept::Unchanged; + } + let endpoint = match &endpoint { + Kept::Changed(s) => s.as_str(), + _ => b.endpoint, + }; + let url = match &url { + Kept::Changed(s) => s.as_str(), + _ => b.url, + }; + return match s3_address::bind(b.binding, endpoint, url) { + Ok(s) => Kept::Changed(format!("{s}{look}")), + Err(_) => Kept::Withheld(prefix), + }; + } + if matches!(prefix, "postgresql:" | "postgres:" | "pg:") { + return match rustgis_data::source::pgconn::public_label(text) { + Some(s) if s == address => Kept::Unchanged, + Some(s) => Kept::Changed(s), + None => Kept::Withheld(prefix), + }; + } + match public_web_address(address) { + Public::Unchanged => Kept::Unchanged, + Public::Changed { address, .. } => Kept::Changed(address), + Public::Withheld => Kept::Withheld(prefix), + } +} + +/// Emits preparation losses after the durable projection's journal has restored live fields. +/// This does not confirm delivery or commit the enclosing command's notices. +pub(crate) fn notify(s: &mut crate::Session, report: &Report) { + for text in report.notices() { + s.notify(format!("Prepared output: {text}"), false); + } +} + +/// Compares credential-bearing executable leaves through the same typed output inventory. +/// Peer input never restores these credentials; a changed or removed field is reported once. +pub(crate) fn changed_executables(old: &mut crate::Project, incoming: &mut crate::Project) -> Result { + let prepared = prepared(old, None, 0)?; + let mut count = 0u32; + for edit in prepared.edits { + if edit.kind != Kind::Credential + || !matches!( + edit.at.leaf, + Leaf::Task(_) + | Leaf::TaskCommand(_) + | Leaf::ModelStep(_) + | Leaf::ModelStepTool(_) + | Leaf::ModelNode(_) + | Leaf::ModelNodeTool(_) + | Leaf::HistoryParams + | Leaf::HistoryId + | Leaf::HistoryCommand + | Leaf::HistoryCommandId + | Leaf::CellSource(_) + ) + { + continue; + } + let same = match (old.field_mut(&edit.at.owners, &edit.at.leaf), incoming.field_mut(&edit.at.owners, &edit.at.leaf)) { + (Some(FieldMut::Text(a)), Some(FieldMut::Text(b))) => a == b, + (Some(FieldMut::Json(a)), Some(FieldMut::Json(b))) => a == b, + _ => false, + }; + count = count.saturating_add(u32::from(!same)); + } + Ok(count) +} + +impl Durable for Vec { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + for (i, v) in self.iter().enumerate() { + p.child(Owner::History(i), v); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + let (Owner::History(i), rest) = owners.split_first()? else { return None }; + self.get_mut(*i)?.field_mut(rest, leaf) + } +} + +impl Durable for rustgis_map::Map { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + for s in self.slots() { + p.string(Leaf::Map(s.at), s.value, s.address, s.local_path); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + if !owners.is_empty() { + return None; + } + let Leaf::Map(at) = leaf else { return None }; + self.slot_mut(at).map(FieldMut::Text) + } +} + +impl Durable for rustgis_map::Layer { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + for (role, value, address, path) in self.output_slots() { + p.string(Leaf::Layer(role), value, address, path); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + if !owners.is_empty() { + return None; + } + let Leaf::Layer(at) = leaf else { return None }; + self.output_slot_mut(*at).map(FieldMut::Text) + } +} + +impl Durable for rustgis_map::presentation::Presentation { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + use rustgis_map::presentation::{Page, PageKind}; + let Self { name: _, width: _, height: _, pages } = self; + for (i, Page { title: _, kind, notes: _, seconds: _ }) in pages.iter().enumerate() { + match kind { + PageKind::Video { source, poster: _, caption: _ } => p.string(Leaf::Video(i), source, true, false), + PageKind::Map { map: _, center: _, width_m: _ } + | PageKind::Blank { text: _, background: _ } + | PageKind::Image { image: _, caption: _ } => {} + } + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + if !owners.is_empty() { + return None; + } + let Leaf::Video(i) = leaf else { return None }; + match &mut self.pages.get_mut(*i)?.kind { + rustgis_map::presentation::PageKind::Video { source, .. } => Some(FieldMut::Text(source)), + _ => None, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn restore_memory_has_entry_total_and_count_bounds() { + let marker = "RGSYNTH_1"; + let mut p = crate::Project::default(); + p.tasks.push(crate::project::Task { + name: "large".into(), + description: String::new(), + steps: vec![crate::project::TaskStep { + title: String::new(), + instructions: String::new(), + command: "layer.addUrl".into(), + params: serde_json::json!({"token": marker.repeat(MAX_ENTRY / marker.len() + 1)}), + }], + }); + let (text, report) = p.saved_text(None).unwrap(); + assert!(report.not_remembered && report.notices().iter().any(|n| n.contains("memory limit"))); + assert!(crate::Project::from_json(&text).unwrap().tasks[0].steps[0].params["token"].as_str().unwrap().is_empty()); + let mut cache = Vec::new(); + for hash in 0..40 { + insert(&mut cache, Written { len: 0, hash, bytes: MAX_ENTRY, edits: Vec::new() }); + } + assert_eq!(cache.len(), MAX_TOTAL / MAX_ENTRY); + assert_eq!(cache.first().unwrap().hash, 32); + let mut cache = Vec::new(); + for hash in 0..40 { + insert(&mut cache, Written { len: 0, hash, bytes: 1, edits: Vec::new() }); + } + assert_eq!(cache.len(), MAX_ENTRIES); + assert_eq!(cache.first().unwrap().hash, 8); + } +} diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index b007d1ba1..7941d0f85 100644 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -7,12 +7,20 @@ pub mod cmd; pub mod connections; +pub mod durable; pub mod geometry_calc; pub mod model; +pub mod peer; #[cfg(not(target_arch = "wasm32"))] pub mod prefs; pub mod project; +mod roles; +pub use peer::Peer; +pub(crate) use roles::{Document, ParamRole, Role}; +mod parameters; pub mod project_log; +pub use parameters::{params_envelope, script_line}; +pub use project_log::{LogCopy, log_identifier, log_params}; pub mod s3_connections; pub mod s3_keys; #[cfg(test)] @@ -58,6 +66,8 @@ pub struct CommandSpec { /// `None` = enabled; `Some(reason)` = disabled. pub enabled: fn(&Session) -> Option<&'static str>, pub run: fn(&mut Session, &Value) -> CmdResult, + /// Reviewed builtin durable-output roles. Extension parameters remain opaque. + pub(crate) roles: Option<&'static [ParamRole]>, } fn always(_: &Session) -> Option<&'static str> { @@ -77,7 +87,12 @@ impl CommandSpec { } pub const fn new(id: &'static str, label: &'static str, location: &'static str, run: fn(&mut Session, &Value) -> CmdResult) -> Self { - CommandSpec { id, label, location, shortcut: "", params: "{}", mutates: true, enabled: always, run } + CommandSpec { id, label, location, shortcut: "", params: "{}", mutates: true, enabled: always, run, roles: None } + } + /// Declares builtin durable-output roles; an empty list uses reviewed common parameter rules. + pub(crate) const fn roles(mut self, roles: &'static [ParamRole]) -> Self { + self.roles = Some(roles); + self } pub const fn key(mut self, s: &'static str) -> Self { self.shortcut = s; @@ -444,17 +459,6 @@ pub struct GpRecord { pub outputs: Vec, } -impl GpRecord { - /// The record with its messages scrubbed of login bindings and secrets: a tool's error can - /// quote a bound address, and history rows are saved with the project and exported. - pub(crate) fn scrubbed(mut self) -> Self { - for message in &mut self.messages { - *message = render::rustgis_tiles::logins::scrub(message); - } - self - } -} - fn is_zero(v: &u64) -> bool { *v == 0 } @@ -478,6 +482,18 @@ pub struct Notice { pub(crate) severity: rustgis_data::diagnostics::Severity, } +/// One counted, already-redacted loss, independent of the changes a command can roll back. +struct LossNotice { + template: &'static str, + count: u32, + text: rustgis_data::diagnostics::Redacted, + source: Option<&'static str>, +} + +/// At most 256 KiB of redacted text per outer command (32 kinds, redaction's 8 KiB bound). +/// Further losses are counted in one overflow notice rather than silently discarded. +const MAX_LOSS_NOTICE_KINDS: usize = 32; + const UNDO_LIMIT: usize = 40; struct ProjectFiles { @@ -602,6 +618,8 @@ pub struct Session { /// the command came (the app, an agent, a script): the interface drops what it kept for the /// old one. pub opened: u64, + /// Session-local last bridge exchange; absent from project files and undo snapshots. + pub(crate) exchange: Option, pub tool: Tool, /// The layer selected in the Layers pane (the one the Style and Data tabs act on). pub active_layer: Option, @@ -624,6 +642,10 @@ pub struct Session { /// [`Session::read_in_place_again`]). pub in_place_wait: bool, pub notices: Vec, + /// Counted losses await the outermost guard's cleanup, including failed nested commands. + /// Runtime-only: never in Project, undo or PutAside, so replacement cannot erase them. + loss_notices: Vec, + loss_notice_overflow: u32, /// Troubleshooting history, independent of the active pane, project files and undo. pub diagnostics: rustgis_data::diagnostics::History, /// Notices given in this session, including ones no longer kept (see [`Session::notices_since`]), @@ -1003,6 +1025,7 @@ impl Session { registry: Registry { commands: cmd::all() }, revision: 1, opened: 0, + exchange: None, tool: Tool::Explore, active_layer: None, layer_selection: Vec::new(), @@ -1011,6 +1034,8 @@ impl Session { style_view: None, in_place_wait: true, notices: Vec::new(), + loss_notices: Vec::new(), + loss_notice_overflow: 0, diagnostics: rustgis_data::diagnostics::History::default(), notices_given: 0, table_layer: None, @@ -1441,7 +1466,12 @@ impl Session { } } + /// Cleans every returned error, including early refusal and plugin paths. fn run_command(&mut self, id: &str, params: &Value) -> CmdResult { + self.run_command_inner(id, params).map_err(|e| durable::generated_text(&e)) + } + + fn run_command_inner(&mut self, id: &str, params: &Value) -> CmdResult { // The merge key asked for ([`Session::merge_undo`]) goes to this command if it runs, and is // gone if it's refused first, so it can't merge a later command into the last step. let merge = if self.depth == 0 { self.merge.take() } else { None }; @@ -1453,7 +1483,7 @@ impl Session { self.merge = merge; return self.run_plugin(id, params); } - let Some(&spec) = self.registry.get(id) else { return Err(format!("unknown command '{id}'")) }; + let Some(&spec) = self.registry.get(id) else { return Err(format!("unknown command '{}'", project_log::log_identifier(id))) }; // Keys the command does not read: said, so a script or an agent can correct itself. if let (Value::Object(given), Some(known)) = (params, spec.param_keys()) { // Bounded: the keys come from the caller (the first few, each clipped). @@ -1520,8 +1550,7 @@ impl Session { { self.sources.note(src); } - // An error never quotes a web-service connection's password or key. - res.map_err(|e| render::rustgis_tiles::logins::scrub(&e)) + res } /// Retires only this consumed reservation, preserving unrelated pending imports in history. @@ -1673,7 +1702,7 @@ impl Session { .skip(history_from) .filter(|record| !record.ok && rustgis_analysis::tool(&record.id).is_some_and(cmd::gp::writes_files)) { - self.project.gp_history.push(record.scrubbed()); + self.push_gp_record(record); recorded_failure = true; } if recorded_failure { @@ -1702,6 +1731,11 @@ impl Session { } self.fix_active_layer(); self.revision = self.revision.wrapping_add(1); + // Nested guards leave their losses pending: only after the outermost rollback has + // withdrawn completion notices may these go to ordinary notice consumers. + if self.depth == 0 { + self.flush_loss_notices(); + } res } @@ -1996,7 +2030,7 @@ impl Session { /// redacted: what a notice or history record may show. Login values are known only here, so /// a failure redacted by [`Session::notify_redacted`]'s callers goes through this first. pub fn redacted(text: &str) -> rustgis_data::diagnostics::Redacted { - rustgis_data::diagnostics::Redacted::new(&render::rustgis_tiles::logins::scrub(text)) + rustgis_data::diagnostics::Redacted::new(&durable::generated_text(text)) } /// [`Session::notify`] for text already scrubbed and redacted by [`Session::redacted`] (a @@ -2007,11 +2041,64 @@ impl Session { self.notice_redacted(text, if error { Severity::Error } else { Severity::Info }); } + /// A counted loss that remains true after rollback, unlike a completion such as "Deleted + /// 3 features". Only static engine templates with `{n}` and a count enter this channel. + /// Counts saturate and repeated templates aggregate once per outer command. Text is redacted + /// before retention; at most 32 kinds are held, with excess counts reported as one overflow. + /// Outside a guard it is published immediately. A native panic caught by `guarded` follows + /// the same outermost drain as an error; nothing is saved in project or undo state. + pub(crate) fn notify_loss(&mut self, template: &'static str, count: u32) { + use rustgis_data::diagnostics::{Redacted, Severity}; + if count == 0 { + return; + } + if self.depth == 0 { + self.notice_redacted(Redacted::new(&template.replace("{n}", &count.to_string())), Severity::Info); + } else if let Some(loss) = self.loss_notices.iter_mut().find(|loss| loss.template == template) { + loss.count = loss.count.saturating_add(count); + loss.text = Redacted::new(&template.replace("{n}", &loss.count.to_string())); + } else if self.loss_notices.len() < MAX_LOSS_NOTICE_KINDS { + self.loss_notices.push(LossNotice { + template, + count, + text: Redacted::new(&template.replace("{n}", &count.to_string())), + source: self.diagnostic_source, + }); + } else { + self.loss_notice_overflow = self.loss_notice_overflow.saturating_add(count); + } + } + + /// Publishes and empties runtime losses only after outermost success/error/panic cleanup. + /// Stored attribution keeps nested command losses assigned to their original command. + fn flush_loss_notices(&mut self) { + use rustgis_data::diagnostics::Severity; + for loss in std::mem::take(&mut self.loss_notices) { + self.notice_redacted_from(loss.text, Severity::Info, loss.source); + } + let overflow = std::mem::take(&mut self.loss_notice_overflow); + if overflow > 0 { + self.notice(format!("{overflow} further loss(es) could not be described"), Severity::Info); + } + } + + /// Uses current attribution for ordinary notices; delayed losses retain their own source. fn notice_redacted(&mut self, text: rustgis_data::diagnostics::Redacted, severity: rustgis_data::diagnostics::Severity) { + self.notice_redacted_from(text, severity, self.diagnostic_source); + } + + /// The one publication path for the ordinary notice queue and diagnostics; called once per + /// loss at drain, so neither sink sees a duplicate from queuing and publication. + fn notice_redacted_from( + &mut self, + text: rustgis_data::diagnostics::Redacted, + severity: rustgis_data::diagnostics::Severity, + source: Option<&'static str>, + ) { if severity == rustgis_data::diagnostics::Severity::Error && self.rerun_said.iter().any(|said| said == text.as_str()) { return; } - self.diagnostics.push_redacted(severity, self.diagnostic_source.unwrap_or("Application"), &text); + self.diagnostics.push_redacted(severity, source.unwrap_or("Application"), &text); self.notices.push(Notice { text: text.into_string(), error: severity == rustgis_data::diagnostics::Severity::Error, severity }); self.notices_given += 1; if self.notices.len() > 200 { @@ -2432,3 +2519,42 @@ impl Session { self.project.name.clone() } } + +impl durable::Durable for GpRecord { + fn prepare(&self, p: &mut durable::Prepare<'_>) { + use durable::Leaf; + let Self { tool, id, params, command, at: _, ok: _, messages, millis: _, outputs: _ } = self; + let tool = if id.is_empty() { cmd::gp::tool_id(tool).unwrap_or("") } else { id }; + p.tool(Leaf::HistoryParams, Leaf::HistoryId, tool, params); + if let Some((id, params)) = command { + p.command(Leaf::HistoryCommand, Leaf::HistoryCommandId, id, params); + } + for (i, message) in messages.iter().enumerate() { + p.generated(Leaf::Message(i), message); + } + } + fn field_mut(&mut self, owners: &[durable::Owner], leaf: &durable::Leaf) -> Option> { + use durable::{FieldMut, Leaf}; + if !owners.is_empty() { + return None; + } + match leaf { + Leaf::HistoryParams => Some(FieldMut::Json(&mut self.params)), + Leaf::HistoryId => Some(FieldMut::Text(&mut self.id)), + Leaf::HistoryCommand => Some(FieldMut::Json(&mut self.command.as_mut()?.1)), + Leaf::HistoryCommandId => Some(FieldMut::Text(&mut self.command.as_mut()?.0)), + Leaf::Message(i) => self.messages.get_mut(*i).map(FieldMut::Text), + _ => None, + } + } +} + +impl Session { + /// The single history production seam: retained generated messages never keep address keys. + pub(crate) fn push_gp_record(&mut self, mut record: GpRecord) { + for message in &mut record.messages { + *message = durable::generated_text(message); + } + self.project.gp_history.push(record); + } +} diff --git a/crates/engine/src/parameters.rs b/crates/engine/src/parameters.rs new file mode 100644 index 000000000..410aff8c5 --- /dev/null +++ b/crates/engine/src/parameters.rs @@ -0,0 +1,141 @@ +//! Authoritative adapter envelope and script parsing. Redacted log/display copies never enter +//! this module: callers retain the returned private value for planning and execution. + +use serde_json::{Value, json}; + +/// The parameters an adapter executes: an object or JSON text of one. Missing, null and blank +/// mean `{}`; other input fails, so a malformed request cannot act with unintended defaults. +pub fn params_envelope(params: Option<&Value>) -> Result { + match params { + None | Some(Value::Null) => Ok(json!({})), + Some(v @ Value::Object(_)) => Ok(v.clone()), + Some(Value::String(t)) if t.trim().is_empty() => Ok(json!({})), + Some(Value::String(t)) => match serde_json::from_str::(t) { + Ok(v @ Value::Object(_)) => Ok(v), + Ok(_) => Err("params must be a JSON object".into()), + Err(e) => Err(format!("params is not valid JSON: {e}")), + }, + Some(_) => Err("params must be a JSON object".into()), + } +} + +/// One executable script line, preserving the executor's `id[=] {json}` grammar. Unlike an +/// adapter envelope, a script's JSON value is passed as written; no parameters means null. +/// Blank and comment lines have no command. Errors are annotated with line numbers by callers. +pub fn script_line(line: &str) -> Option> { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + return None; + } + let (id, rest) = line.split_once(char::is_whitespace).unwrap_or((line, "")); + let id = id.trim_end_matches('='); + let params = if rest.trim().is_empty() { + Value::Null + } else { + match serde_json::from_str(rest.trim()) { + Ok(value) => value, + Err(e) => return Some(Err(e.to_string())), + } + }; + if crate::cmd::workflows::refused_in_script(id) { + return Some(Err(format!("'{id}' can't run in a script"))); + } + Some(Ok((id.to_string(), params))) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn envelopes_and_script_lines_keep_the_executor_grammar() { + let object = json!({"layer": "private source", "token": "RGSYNTH_params_20261103"}); + assert_eq!(params_envelope(Some(&object)), params_envelope(Some(&json!(object.to_string())))); + for value in [json!(42), json!([]), json!("[1]"), json!("false"), json!("{bad")] { + assert!(params_envelope(Some(&value)).is_err(), "{value}"); + } + for value in [Value::Null, json!(" \t")] { + assert_eq!(params_envelope(Some(&value)).unwrap(), json!({})); + } + assert_eq!(params_envelope(None).unwrap(), json!({})); + assert!(script_line(" # comment").is_none()); + assert_eq!(script_line(" map.zoomIn= {} ").unwrap().unwrap(), ("map.zoomIn".into(), json!({}))); + assert_eq!(script_line("map.zoomIn").unwrap().unwrap().1, Value::Null); + assert_eq!(script_line("map.zoomIn []").unwrap().unwrap().1, json!([])); + assert!(script_line("map.zoomIn {bad").unwrap().is_err()); + assert!(script_line("script.run {}").unwrap().is_err()); + } + + /// Supplied documents and an explicitly supplied final template keep this command test + /// network-free while exercising the real Session notice boundary. + #[test] + fn service_discovery_reports_withheld_destinations_and_clean_input_is_silent() { + for (path, count) in [("tiles", 0), ("%252e/tiles", 1)] { + let href = format!("https://h.invalid/svc/{path}?token=RGSYNTH_notice_20261103"); + let document = json!({"title": "roads", "links": [{"rel": "tilesets-map", "href": href}]}).to_string(); + let mut params = json!({"url": "https://h.invalid/svc", "type": "ogcapi", "text": document}); + let mut session = crate::Session::new(crate::Project::default()); + session.run("service.layers", ¶ms).unwrap(); + let notices = std::mem::take(&mut session.notices); + assert_eq!(notices.len(), count); + assert!(notices.iter().all(|n| n.text.contains("1 address(es)") && !n.text.contains("RGSYNTH_"))); + params["layer"] = json!("map"); + params["tiles"] = json!("https://h.invalid/explicit/{z}/{y}/{x}"); + session.run("service.add", ¶ms).unwrap(); + let notices = std::mem::take(&mut session.notices); + assert_eq!(notices.len(), count); + assert!(notices.iter().all(|n| n.text.contains("1 address(es)") && !n.error)); + // All fail after discovery counted its withheld destinations. The forwarding notice + // is independent of the unchanged command error, and is emitted once on either path. + for (changes, expected) in [ + (json!({"time": {}}), "roads (map tiles): time is a date, \"slider\" or \"default\""), + (json!({"tiles": "not-a-template"}), "not-a-template: not a tile template"), + (json!({"layer": "missing"}), "https://h.invalid/svc: no layer 'missing' (service.layers lists them)"), + ] { + let mut bad = params.clone(); + for (key, value) in changes.as_object().unwrap() { + bad[key] = value.clone(); + } + assert_eq!(session.run("service.add", &bad).unwrap_err(), expected); + let notices = std::mem::take(&mut session.notices); + let forwarding = notices.iter().filter(|n| !n.error && n.text.contains("address(es) the service listed")).count(); + assert_eq!(forwarding, count); + assert_eq!(notices.iter().filter(|n| n.error).count(), 1, "the original error is still reported"); + assert_eq!(notices.len(), count + 1, "one aggregated forwarding notice, clean errors silent about forwarding"); + } + } + } + + /// Nested success and a later caught panic must retain one counted loss, while the + /// completion that was rolled back disappears. The next guard must publish nothing stale. + #[cfg(not(target_arch = "wasm32"))] + #[test] + fn nested_loss_notices_drain_once_after_panic_cleanup() { + let mut session = crate::Session::new(crate::Project::default()); + let name = session.project.name.clone(); + let mark = session.notice_mark(); + const LOSS: &str = "{n} resource(s) were omitted at https://h.invalid/x?token=RGSYNTH_2"; + let result = session.guarded("Loss rollback", true, |s| { + s.guarded("Nested loss", true, |s| { + s.notify_loss(LOSS, 1); + Ok(Value::Null) + })?; + assert!(s.notices_since(mark).is_empty(), "the outer guard has not finished"); + s.notify_loss(LOSS, 2); + s.project.name = "temporary".into(); + s.notify("Deleted 3 features", false); + panic!("synthetic loss rollback"); + }); + assert_eq!(result.unwrap_err(), "Loss rollback failed unexpectedly; the project was restored"); + assert_eq!(session.project.name, name); + let notices = session.notices_since(mark); + assert_eq!(notices.iter().filter(|n| !n.error).count(), 1); + assert!(notices.iter().any(|n| !n.error && n.text.starts_with("3 resource(s)"))); + assert!(notices.iter().all(|n| !n.text.contains("RGSYNTH_") && n.text != "Deleted 3 features")); + let diagnostic_losses = session.diagnostics.records().into_iter().filter(|r| r.message.starts_with("3 resource(s)")).count(); + assert_eq!(diagnostic_losses, 1); + let mark = session.notice_mark(); + session.guarded("Next command", false, |_| Ok(Value::Null)).unwrap(); + assert!(session.notices_since(mark).is_empty()); + } +} diff --git a/crates/engine/src/peer.rs b/crates/engine/src/peer.rs new file mode 100644 index 000000000..6cb89e350 --- /dev/null +++ b/crates/engine/src/peer.rs @@ -0,0 +1,530 @@ +//! Project exchange with a peer is an import, never an explicit saved-file reopen. A framing +//! page receives no credentials and cannot borrow the process's identical-text restore memory. + +use crate::{ + Project, Session, + durable::{self, Report}, +}; +use durable::{At, Durable, FieldMut, Kept, Leaf, Owner}; +use std::borrow::Cow; +use std::collections::{HashMap, HashSet}; + +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +enum Position<'a> { + Layer { id: u32, role: rustgis_map::LayerRole }, + Tile { id: Cow<'a, str>, role: rustgis_map::TileRole }, + Group { id: u32, child: Option }, + Terrain, +} + +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +enum SlotKey<'a> { + Map { index: usize, name: Cow<'a, str>, at: Position<'a> }, + Area { index: usize, at: Position<'a> }, + Table(u32), + Video { presentation: usize, page: usize }, +} + +impl SlotKey<'_> { + /// Only retained, budgeted identities own copies; raw censuses borrow hostile names. + fn owned(&self) -> SlotKey<'static> { + let position = |at: &Position<'_>| match at { + Position::Layer { id, role } => Position::Layer { id: *id, role: *role }, + Position::Tile { id, role } => Position::Tile { id: Cow::Owned(id.to_string()), role: *role }, + Position::Group { id, child } => Position::Group { id: *id, child: *child }, + Position::Terrain => Position::Terrain, + }; + match self { + Self::Map { index, name, at } => SlotKey::Map { index: *index, name: Cow::Owned(name.to_string()), at: position(at) }, + Self::Area { index, at } => SlotKey::Area { index: *index, at: position(at) }, + Self::Table(id) => SlotKey::Table(*id), + Self::Video { presentation, page } => SlotKey::Video { presentation: *presentation, page: *page }, + } + } +} + +struct Slot<'a> { + key: SlotKey<'a>, + at: At, + value: &'a str, + layer: Option<&'a rustgis_map::Layer>, +} +struct Entry { + key: SlotKey<'static>, + choice: Option, + public: String, + full: String, + ambiguous: bool, +} +pub(crate) struct Exchange { + generation: u64, + entries: Vec, +} +const MAX_ENTRIES: usize = 4096; +const MAX_BYTES: usize = 1 << 20; + +/// Enumerates slot identities before any normalization. Duplicate identities are retained in the +/// census: they must never be made unique by dropping or renumbering a hostile duplicate. +fn slots(p: &Project) -> Vec> { + let mut out = Vec::new(); + for (i, m) in p.maps.iter().enumerate() { + for slot in m.slots().into_iter().filter(|s| s.address) { + if let Some(at) = position(m, &slot.at) { + out.push(Slot { + key: SlotKey::Map { index: i, name: Cow::Borrowed(&m.name), at }, + at: At { owners: vec![Owner::Map(i)], leaf: Leaf::Map(slot.at.clone()) }, + value: slot.value, + layer: match slot.at { + rustgis_map::SlotPos::Layer { layer, .. } => m.layers.get(layer), + _ => None, + }, + }); + } + } + } + for (i, area) in p.offline_areas.iter().enumerate() { + for slot in area.data.slots().into_iter().filter(|s| s.address) { + if let Some(at) = position(&area.data, &slot.at) { + out.push(Slot { + key: SlotKey::Area { index: i, at }, + at: At { owners: vec![Owner::Area(i)], leaf: Leaf::Map(slot.at.clone()) }, + value: slot.value, + layer: match slot.at { + rustgis_map::SlotPos::Layer { layer, .. } => area.data.layers.get(layer), + _ => None, + }, + }); + } + } + } + for (i, table) in p.tables.iter().enumerate() { + if let Some(value) = &table.source { + out.push(Slot { + key: SlotKey::Table(table.id), + at: At { owners: vec![Owner::Table(i)], leaf: Leaf::Table }, + value, + layer: None, + }); + } + } + for (i, pres) in p.presentations.iter().enumerate() { + for (page, p) in pres.pages.iter().enumerate() { + if let rustgis_map::presentation::PageKind::Video { source, .. } = &p.kind { + out.push(Slot { + key: SlotKey::Video { presentation: i, page }, + at: At { owners: vec![Owner::Presentation(i)], leaf: Leaf::Video(page) }, + value: source, + layer: None, + }); + } + } + } + out +} + +/// Resolves raw owner identity without copying untrusted map or tile names. +fn position<'a>(map: &'a rustgis_map::Map, at: &rustgis_map::SlotPos) -> Option> { + use rustgis_map::SlotPos; + Some(match at { + SlotPos::Layer { layer, role } => Position::Layer { id: map.layers.get(*layer)?.id, role: *role }, + SlotPos::Tile { tile, role } => Position::Tile { id: Cow::Borrowed(&map.tiles.get(*tile)?.id), role: *role }, + SlotPos::Group { group, child } => { + let group = map.service_groups.get(*group)?; + Position::Group { + id: group.id, + child: match child { + Some(i) => Some(group.children.get(*i)?.layer), + None => None, + }, + } + } + SlotPos::Terrain => Position::Terrain, + }) +} + +/// Read selection, not drawing style or cached rows. The last exchange, live source and raw +/// incoming source must agree before normalization or a full address can be retained. +fn source_choice(layer: Option<&rustgis_map::Layer>) -> Result, ()> { + let Some(layer) = layer.filter(|l| l.source.is_some() || l.raster.is_some() || l.postgis.is_some()) else { return Ok(None) }; + if layer.output_slots().iter().any(|(_, value, _, _)| value.len() > MAX_BYTES) + || layer.source.as_ref().and_then(|s| s.embedded.as_ref()).is_some_and(|b| b.len() > MAX_BYTES) + || layer.raster.as_ref().and_then(|r| r.embedded.as_ref()).is_some_and(|b| b.len() > MAX_BYTES) + { + return Err(()); + } + let public = |s: &str| match durable::public_address(s) { + Kept::Unchanged | Kept::Withheld(_) => s.to_string(), + Kept::Changed(s) => s, + }; + let source = layer.source.as_ref().map(|s| { + ( + s.path.as_deref().map(public), + &s.file_name, + &s.table, + &s.shapefile_crs, + s.embedded.as_deref(), + s.gdal.as_ref().map(|g| (public(&g.path), &g.layer, g.family, &g.crs)), + s.duckdb.as_ref().map(|d| (&d.sql, d.database.as_deref().map(public), &d.table, &d.crs, d.family)), + s.postgis.as_ref().map(|d| (public(&d.database), &d.table, d.family, &d.crs)), + s.mysql.as_ref().map(|d| (public(&d.database), &d.table, d.family, &d.crs)), + s.sqlserver.as_ref().map(|d| (public(&d.database), &d.table, d.family, &d.crs)), + ) + }); + let raster = layer.raster.as_ref().map(|r| { + ( + r.path.as_deref().map(public), + r.in_place, + &r.file_name, + r.embedded.as_deref(), + &r.subdataset, + &r.crs, + r.placement, + r.time_zarr.as_ref().map(|z| (public(&z.store), &z.variable, &z.index, &z.dim, z.offset, &z.dates, &z.crs)), + ) + }); + let edit = layer.postgis.as_ref().map(|p| { + (public(&p.origin.database), &p.origin.table, p.origin.family, &p.origin.crs, &p.key, p.srid, p.writable, &p.condition, p.limit) + }); + struct Bounded(Vec); + impl std::io::Write for Bounded { + /// Stops at the retention budget before allocating another chunk. + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + if self.0.len().saturating_add(bytes.len()) > MAX_BYTES { + return Err(std::io::Error::other("peer source identity exceeds retention limit")); + } + self.0.extend_from_slice(bytes); + Ok(bytes.len()) + } + /// The bounded buffer has no pending I/O. + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + let mut out = Bounded(Vec::new()); + serde_json::to_writer(&mut out, &(source, raster, edit)).map_err(|_| ())?; + String::from_utf8(out.0).map(Some).map_err(|_| ()) +} + +/// A wrapper's binding belongs to S3 connection resolution, never exchange retention. +fn wrapped(value: &str) -> bool { + let value = crate::sources::template_archive(value).map_or(value, |(_, v)| v); + value.trim_start().get(..11).is_some_and(|v| v.eq_ignore_ascii_case("rustgis-s3:")) +} + +/// Charges both forms and the complete identity payload, bounded before copying either form. +fn key_bytes(key: &SlotKey<'_>) -> usize { + let extra = match key { + SlotKey::Map { name, at, .. } => name.len().saturating_add(match at { + Position::Tile { id, .. } => id.len(), + _ => 0, + }), + SlotKey::Area { at: Position::Tile { id, .. }, .. } => id.len(), + _ => 0, + }; + std::mem::size_of::>().saturating_add(extra) +} + +impl Exchange { + /// Recorded only after output rollback. Every entry is session-local and replaced per push. + fn record(p: &Project, generation: u64) -> (Self, u32) { + let mut entries: Vec = Vec::new(); + let mut forms: HashMap = HashMap::new(); + let mut recording = true; + let mut held = 0usize; + let mut omitted = 0u32; + for slot in slots(p) { + if wrapped(slot.value) || slot.layer.and_then(|l| l.source.as_ref()).is_some_and(|s| s.reconnect_required()) { + continue; + } + // A clean database address may still use a typed password held by its driver. + let database = slot + .layer + .and_then(|l| l.source.as_ref()) + .is_some_and(|s| s.postgis.is_some() || s.mysql.is_some() || s.sqlserver.is_some() || s.duckdb.is_some()); + let public = match durable::public_address(slot.value) { + Kept::Unchanged if database => slot.value.to_string(), + Kept::Unchanged => continue, + Kept::Changed(s) => s, + Kept::Withheld(prefix) => prefix.into(), + }; + // Even an account omitted by the cap cannot make an earlier recorded public form + // unambiguous. After the first overflow, no later slot is recorded. + if let Some((first, ambiguous)) = forms.get_mut(&public) { + *ambiguous |= entries.get(*first).is_none_or(|e| e.full != slot.value); + } + let Ok(choice) = source_choice(slot.layer) else { + omitted = omitted.saturating_add(1); + continue; + }; + let bytes = slot + .value + .len() + .saturating_add(public.len()) + .saturating_add(key_bytes(&slot.key)) + .saturating_add(choice.as_ref().map_or(0, String::len)); + if !recording || entries.len() >= MAX_ENTRIES || held.saturating_add(bytes) > MAX_BYTES { + recording = false; + omitted = omitted.saturating_add(1); + continue; + } + held += bytes; + forms.entry(public.clone()).or_insert((entries.len(), false)); + entries.push(Entry { key: slot.key.owned(), choice, public, full: slot.value.into(), ambiguous: false }); + } + for entry in &mut entries { + entry.ambiguous = forms.get(&entry.public).is_none_or(|(_, ambiguous)| *ambiguous); + } + (Self { generation, entries }, omitted) + } + + /// The pre-normalization census must contain exactly one byte-equal public slot. + fn incoming(&self, p: &Project) -> HashSet> { + let census = census(p); + self.entries + .iter() + .filter(|e| { + census.get(&e.key).is_some_and(|(n, value, layer)| { + *n == 1 && *value == e.public && source_choice(*layer).ok().as_ref() == Some(&e.choice) + }) + }) + .map(|e| e.key.clone()) + .collect() + } + + /// Checks last exchange, both incoming censuses, unique account and the exact current live + /// value in one synchronous replacement. Public URL or incoming id alone grants nothing. + fn retain(&self, live: &Project, incoming: &mut Project, raw: &HashSet>) { + let current = census(live); + let normalized = census(incoming); + let mut records: HashMap, (usize, &Entry)> = HashMap::new(); + let mut accounts: HashMap<&str, Option<&str>> = HashMap::new(); + for e in &self.entries { + records.entry(e.key.clone()).or_insert((0, e)).0 += 1; + let account = accounts.entry(&e.public).or_insert(Some(&e.full)); + if *account != Some(e.full.as_str()) { + *account = None; + } + } + let edits: Vec<_> = slots(incoming) + .into_iter() + .filter_map(|slot| { + let (count, e) = records.get(&slot.key)?; + let unique = |census: &Census<'_>, expected: &str| { + census.get(&slot.key).is_some_and(|(n, value, layer)| { + *n == 1 && *value == expected && source_choice(*layer).ok().as_ref() == Some(&e.choice) + }) + }; + (!e.ambiguous + && raw.contains(&e.key) + && !current + .get(&slot.key) + .is_some_and(|(_, _, layer)| layer.and_then(|l| l.source.as_ref()).is_some_and(|s| s.reconnect_required())) + && *count == 1 + && accounts.get(e.public.as_str()) == Some(&Some(e.full.as_str())) + && unique(¤t, &e.full) + && unique(&normalized, &e.public)) + .then(|| (slot.at, e.full.clone())) + }) + .collect(); + drop(normalized); + for (at, full) in edits { + if let Some(FieldMut::Text(slot)) = incoming.field_mut(&at.owners, &at.leaf) { + *slot = full; + } + reopen(incoming, &at); + } + } +} + +/// A retained address must reopen its own reader, which normalization previously opened with +/// the public form. Only the exact retained owner/role is touched; S3 wrappers are excluded. +fn reopen(p: &mut Project, at: &At) { + let Some(owner) = at.owners.first() else { return }; + let map = match owner { + Owner::Map(i) => p.maps.get_mut(*i), + Owner::Area(i) => p.offline_areas.get_mut(*i).map(|a| &mut a.data), + _ => None, + }; + let Some(map) = map else { return }; + let Leaf::Map(rustgis_map::SlotPos::Layer { layer, role }) = &at.leaf else { return }; + let Some(layer) = map.layers.get_mut(*layer) else { return }; + match role { + rustgis_map::LayerRole::Streamed + | rustgis_map::LayerRole::PostgisDatabase + | rustgis_map::LayerRole::MysqlDatabase + | rustgis_map::LayerRole::SqlServerDatabase + | rustgis_map::LayerRole::DuckDbDatabase + | rustgis_map::LayerRole::GdalPath => { + if let Some(source) = &mut layer.source { + source.allow_reconnect(); + if let Err(e) = source.reopen() { + source.error = Some(rustgis_data::diagnostics::Redacted::new(&e).into_string()); + } + } + } + rustgis_map::LayerRole::Raster => { + if let Some(raster) = &mut layer.raster + && let Err(e) = raster.reload() + { + raster.error = Some(rustgis_data::diagnostics::Redacted::new(&e).into_string()); + } + } + _ => {} + } +} + +/// Values borrow the project, but identities include all duplicates; no normalization runs here. +type Census<'a> = HashMap, (usize, &'a str, Option<&'a rustgis_map::Layer>)>; + +/// Counts raw duplicates while borrowing all untrusted identity text. +fn census(p: &Project) -> Census<'_> { + let mut out = HashMap::new(); + for slot in slots(p) { + let entry = out.entry(slot.key).or_insert((0usize, slot.value, slot.layer)); + entry.0 = entry.0.saturating_add(1); + } + out +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Peer { + Bridge, + Embedder, +} + +impl Session { + /// Writes a credential-free project for a peer, without evicting explicit-save restoration. + /// Pending downloads require a structural draft; settled projects use the live journal. + pub fn peer_text(&mut self, peer: Peer) -> Result<(String, Report), String> { + let runtime = self.project.bound_address_losses(true); + let (text, mut report) = if self.project.ensure_imports_complete().is_err() { + let mut draft = self.project.clone(); + draft.omit_pending_imports(); + durable::output_owned(draft, text) + } else { + durable::output(&mut self.project, None, text) + }?; + report.addresses = report.addresses.saturating_add(u32::try_from(runtime).unwrap_or(u32::MAX)); + self.exchange = if peer == Peer::Bridge { + let (record, omitted) = Exchange::record(&self.project, self.opened); + report.not_retained = omitted; + Some(record) + } else { + None + }; + Ok((text, report)) + } + + /// Replaces a project with peer/import semantics, bypassing explicit-save restoration. + /// Reset clears active connections, pending work and item trust before incoming resolution. + pub fn open_peer_text(&mut self, peer: Peer, text: &str) -> Result { + let mut p = Project::from_peer_raw(text)?; + let exchange = self.exchange.as_ref().filter(|e| peer == Peer::Bridge && e.generation == self.opened); + let raw = exchange.map(|e| e.incoming(&p)); + // Normalization may open a database with a typed process-local password. Keep every + // incoming database source closed until last/live/raw/normalized selection agrees. + for map in p.maps.iter_mut().chain(p.offline_areas.iter_mut().map(|a| &mut a.data)) { + for source in map.layers.iter_mut().filter_map(|l| l.source.as_mut()) { + if source.postgis.is_some() || source.mysql.is_some() || source.sqlserver.is_some() || source.duckdb.is_some() { + source.require_reconnect(); + } + } + } + crate::cmd::connections::clear_active(); + p.normalize_opened(); + let mut report = Report::default(); + + if let (Some(exchange), Some(raw)) = (exchange, raw) { + exchange.retain(&self.project, &mut p, &raw); + } + report.reconnect = p + .maps + .iter() + .chain(p.offline_areas.iter().map(|a| &a.data)) + .flat_map(|m| &m.layers) + .filter(|l| l.source.as_ref().is_some_and(|s| s.reconnect_required())) + .count() + .try_into() + .unwrap_or(u32::MAX); + let incoming = census(&p); + report.lost = slots(&self.project) + .into_iter() + .filter(|slot| { + !matches!(durable::public_address(slot.value), Kept::Unchanged) + && !incoming.get(&slot.key).is_some_and(|(count, value, _)| *count == 1 && *value == slot.value) + }) + .count() + .try_into() + .unwrap_or(u32::MAX); + drop(incoming); + report.executable_changes = durable::changed_executables(&mut self.project, &mut p)?; + crate::cmd::project::reset_to(self, p); + self.path = None; + crate::cmd::connections::resolve_project(self); + crate::cmd::connections::note_waiting(self); + self.revision = self.revision.wrapping_add(1); + if peer == Peer::Bridge { + let (exchange, omitted) = Exchange::record(&self.project, self.opened); + self.exchange = Some(exchange); + report.not_retained = omitted; + } + Ok(report) + } +} + +/// The same opening bound applies to peers; no destination is written on a formatting failure. +fn text(p: &Project) -> Result { + let text = serde_json::to_string(p).map_err(|e| e.to_string())?; + if text.len() as u64 > crate::project::MAX_BYTES { + return Err("the project is too large to exchange".into()); + } + Ok(text) +} + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod tests { + use super::*; + + /// Table/family/CRS are part of a retained reader, even before normalization opens it. + #[test] + fn bridge_census_binds_the_full_source_choice_before_normalization() { + let mut p = Project::default(); + let id = p.map_mut().add_dataset(rustgis_data::Dataset::new("Database", rustgis_data::GeometryType::None, vec![])); + let mut source = rustgis_map::source::SourceLayer::default(); + source.postgis = Some(rustgis_map::source::PostgisOrigin { + database: "postgresql://reader:RGSYNTH_peer_choice@db.invalid/gis".into(), + table: "approved".into(), + family: None, + crs: None, + }); + p.map_mut().layer_mut(id).unwrap().source = Some(source); + let mut s = Session::new(p); + let (public, _) = s.peer_text(Peer::Bridge).unwrap(); + let exchange = s.exchange.as_ref().unwrap(); + for change in ["style", "table", "family", "crs"] { + let mut p = Project::from_peer_raw(&public).unwrap(); + let layer = &mut p.maps[0].layers[0]; + let pg = layer.source.as_mut().unwrap().postgis.as_mut().unwrap(); + match change { + "table" => pg.table = "different".into(), + "family" => pg.family = Some(rustgis_data::source::Family::Points), + "crs" => pg.crs = Some("EPSG:4326".into()), + _ => layer.visible = false, + } + assert_eq!(exchange.incoming(&p).len(), usize::from(change == "style"), "{change}"); + } + // Reading a raster in place changes its source selection, not just its drawing style. + let mut p = Project::default(); + let mut raster = rustgis_map::raster::RasterLayer::unread_in_place("https://choice.invalid/scene.tif?token=RGSYNTH_raster_choice"); + raster.in_place = false; + p.map_mut().add_raster("Scene", raster); + let mut s = Session::new(p); + let (public, _) = s.peer_text(Peer::Bridge).unwrap(); + let exchange = s.exchange.as_ref().unwrap(); + let mut incoming = Project::from_peer_raw(&public).unwrap(); + assert_eq!(exchange.incoming(&incoming).len(), 1); + incoming.maps[0].layers[0].raster.as_mut().unwrap().in_place = true; + assert!(exchange.incoming(&incoming).is_empty(), "a changed raster read mode cannot retain credentials"); + } +} diff --git a/crates/engine/src/project.rs b/crates/engine/src/project.rs index b4e55eedc..8c7a4c5da 100644 --- a/crates/engine/src/project.rs +++ b/crates/engine/src/project.rs @@ -1,9 +1,6 @@ //! The project: maps, standalone tables and geoprocessing history, saved as JSON (`.rgis`). -use std::collections::hash_map::RandomState; -use std::hash::BuildHasher; use std::path::Path; -use std::sync::{Mutex, MutexGuard, OnceLock}; use rustgis_data::{Dataset, Feature, Field, GeometryType}; use rustgis_map::Map; @@ -62,74 +59,24 @@ pub const FORMAT: u32 = 1; /// Largest project file read or saved (512 MiB). pub const MAX_BYTES: u64 = 512 << 20; -/// The address a saved file keeps. Only web addresses can contain layer credentials. A tile -/// template (`pmtiles:…@archive`, `cog:…@archive`) is judged by its archive, and an address read -/// through a saved S3 connection by the address it wraps, the wrapper rebuilt around it (its -/// binding and endpoint are provenance, not secrets); a COG's look after either is kept. -fn saved_address(address: &str) -> Option { - if let Some(public) = without_login_binding(address) { - return Some(saved_address(&public).unwrap_or(public)); - } - use rustgis_data::s3_address; - if let Some((head, archive)) = crate::sources::template_archive(address) { - let (file, look) = split_look(archive); - return saved_address(file).map(|saved| format!("{head}{saved}{look}")); - } - if s3_address::is_wrapper(address) { - let (file, look) = split_look(address); - // One that doesn't parse stays as it is (and blocked): it names no connection. - let b = s3_address::parse(file).ok().flatten()?; - let inner = saved_address(b.url)?; - return s3_address::bind(b.binding, b.endpoint, &inner).ok().map(|saved| format!("{saved}{look}")); - } - let inner = address.strip_prefix(crate::render::rustgis_tiles::s3::ANONYMOUS).unwrap_or(address); - let (protocol, _) = inner.split_once("://")?; - if !["http", "https", "s3"].iter().any(|s| protocol.eq_ignore_ascii_case(s)) { - return None; - } - let (scheme, mut rest) = address.split_once("://")?; - let end = rest.find(['/', '?', '#']).unwrap_or(rest.len()); - if let Some(at) = rest.get(..end).and_then(|authority| authority.rfind('@')) { - rest = rest.get(at + 1..)?; - } - let rest = rest.split('#').next().unwrap_or(rest); - let mut out = format!("{scheme}://{rest}"); - rustgis_data::source::credentials::redact_query(&mut out); - (out != address).then_some(out) -} - -/// A COG address without its look (`…#bands=…`), and the look as written. -fn split_look(address: &str) -> (&str, &str) { - let file = crate::render::rustgis_tiles::cog::split(address).0; - (file, address.get(file.len()..).unwrap_or("")) -} - -/// Strips layer credentials, keeping originals in the visitor's order for the caller to restore. -/// Packaging calls this without registering the resulting text as a file this process wrote. -pub(crate) fn strip_addresses(addresses: Vec<&mut String>) -> Vec> { - addresses.into_iter().map(|a| saved_address(a).map(|redacted| std::mem::replace(a, redacted))).collect() -} - /// Runtime capabilities never survive a project boundary, including an in-process save restore. /// `None` for an address that holds none, and for a wrapper that doesn't parse (kept as it is, /// never emptied: its layer then fails to read, saying why). -fn without_login_binding(address: &str) -> Option { - if rustgis_data::login_address::is_wrapper(address) { - return rustgis_data::login_address::parse(address).ok().flatten().map(|b| b.url.to_string()); - } - if let Some((head, archive)) = crate::sources::template_archive(address) { - return without_login_binding(archive).map(|public| format!("{head}{public}")); - } - if let Ok(Some(bound)) = rustgis_data::s3_address::parse(address) { - return without_login_binding(bound.url) - .and_then(|public| rustgis_data::s3_address::bind(bound.binding, bound.endpoint, &public).ok()); - } - None +pub(crate) fn without_login_binding(address: &str) -> Option { + // A shallow recorded string can contain arbitrarily many template prefixes. Walk its + // suffixes without recursion, then rebuild the unchanged prefix only once. + let mut inner = address; + while let Some((_, archive)) = crate::sources::template_archive(inner) { + inner = archive; + } + let bound = rustgis_data::login_address::parse(inner).ok().flatten()?; + let prefix = address.get(..address.len() - inner.len())?; + Some(format!("{prefix}{}", bound.url)) } /// A recorded value may contain several addresses inside ordinary text. Remove just each /// capability header, not URL queries or words that happen to match an active credential. -fn without_recorded_login_bindings(text: &str) -> Option { +pub(crate) fn without_recorded_login_bindings(text: &str) -> Option { let whole = without_login_binding(text); let text = whole.as_deref().unwrap_or(text); let mut rest = text; @@ -205,13 +152,6 @@ fn recorded_login_url(text: &str) -> Option<&str> { Some(url) } -/// Whether stripping `original` from a saved file lost a secret the user must give again: not -/// when all it held was a login binding (the layer records its connection, and asks for its login -/// when it opens), unless the address under the binding held one too. -fn lost_secret(original: &str) -> bool { - without_login_binding(original).is_none_or(|public| saved_address(&public).is_some()) -} - pub(crate) fn clear_login_bindings<'a>(addresses: impl IntoIterator) { for address in addresses { if let Some(public) = without_recorded_login_bindings(address) { @@ -220,44 +160,42 @@ pub(crate) fn clear_login_bindings<'a>(addresses: impl IntoIterator) -> Vec> { - addresses.into_iter().map(|a| without_recorded_login_bindings(a).map(|public| std::mem::replace(a, public))).collect() -} - +/// Visits recorded strings without recursion; credential-named user data remains untouched. pub(crate) fn recorded_addresses<'a>(value: &'a mut serde_json::Value, addresses: &mut Vec<&'a mut String>) { - match value { - serde_json::Value::String(text) => addresses.push(text), - serde_json::Value::Array(values) => { - for value in values { - recorded_addresses(value, addresses); - } - } - serde_json::Value::Object(values) => { - for value in values.values_mut() { - recorded_addresses(value, addresses); - } + let mut pending = vec![value]; + while let Some(value) = pending.pop() { + match value { + serde_json::Value::String(text) => addresses.push(text), + serde_json::Value::Array(values) => pending.extend(values), + serde_json::Value::Object(values) => pending.extend(values.values_mut()), + _ => {} } - _ => {} } } -struct Written { - len: usize, - hash: u64, - was: Vec>, -} - -const MAX_WRITTEN: usize = 32; - -fn kept() -> MutexGuard<'static, Vec> { - static KEPT: OnceLock>> = OnceLock::new(); - KEPT.get_or_init(Default::default).lock().unwrap_or_else(|e| e.into_inner()) -} - -fn identity(text: &str) -> (usize, u64) { - static KEY: OnceLock = OnceLock::new(); - (text.len(), KEY.get_or_init(RandomState::new).hash_one(text.as_bytes())) +/// A role-independent runtime capability is never part of a recorded parameter's public form. +pub(crate) fn public_login_params(value: &serde_json::Value) -> Option { + let mut pending = vec![value]; + let mut found = false; + while let Some(value) = pending.pop() { + match value { + serde_json::Value::String(text) if text.contains(rustgis_data::login_address::PREFIX) => { + found = true; + break; + } + serde_json::Value::Array(values) => pending.extend(values), + serde_json::Value::Object(values) => pending.extend(values.values()), + _ => {} + } + } + if !found { + return None; + } + let mut public = value.clone(); + let mut addresses = Vec::new(); + recorded_addresses(&mut public, &mut addresses); + clear_login_bindings(addresses); + (public != *value).then_some(public) } #[derive(Clone, Debug, Serialize, Deserialize)] @@ -712,29 +650,42 @@ impl Project { if let Some(graph) = &mut model.graph { for node in &mut graph.nodes { recorded_addresses(&mut node.params, &mut addresses); - // A node's input may be an address rather than a layer's name. - if let Some(layer) = &mut node.layer { - addresses.push(layer); - } + // node.layer selects a name, including names that resemble runtime URLs. } } } addresses } - /// How many layers and tile layers read through a login binding but record no connection: - /// nothing would tell the opened project which login they need. One whose address also holds - /// a token isn't counted here: [`lost_secret`] counts it already. - fn unrecorded_bound(&mut self) -> usize { - let only_bound = |a: &str| without_login_binding(a).is_some_and(|public| saved_address(&public).is_none()); - let mut n = 0; - for map in self.maps.iter_mut().chain(self.offline_areas.iter_mut().map(|area| &mut area.data)) { - n += map.tiles.iter().filter(|t| t.connection.is_empty() && only_bound(&t.url)).count(); - for layer in map.layers.iter_mut().filter(|l| l.connection.is_empty()) { - n += usize::from(layer.addresses_mut().any(|a| only_bound(a))); + /// Runtime bindings that this output cannot restore: peer receivers never share a login; + /// explicit files can resolve layers and tile owners with a recorded saved connection. + pub(crate) fn bound_address_losses(&self, include_recorded: bool) -> usize { + let only_bound = |a: &str| { + without_login_binding(a) + .is_some_and(|public| matches!(crate::durable::public_address(&public), crate::durable::Kept::Unchanged)) + }; + let mut n = 0usize; + for map in self.maps.iter().chain(self.offline_areas.iter().map(|area| &area.data)) { + let mut layers = std::collections::BTreeSet::new(); + let mut tiles = std::collections::BTreeSet::new(); + for slot in map.slots().into_iter().filter(|slot| slot.address && only_bound(slot.value)) { + match slot.at { + rustgis_map::SlotPos::Layer { layer, .. } => { + if map.layers.get(layer).is_some_and(|l| include_recorded || l.connection.is_empty()) { + layers.insert(layer); + } + } + rustgis_map::SlotPos::Tile { tile, .. } => { + if map.tiles.get(tile).is_some_and(|t| include_recorded || t.connection.is_empty()) { + tiles.insert(tile); + } + } + rustgis_map::SlotPos::Group { .. } | rustgis_map::SlotPos::Terrain => n = n.saturating_add(1), + } } + n = n.saturating_add(layers.len()).saturating_add(tiles.len()); } - n + n.saturating_add(self.tables.iter().filter(|table| table.source.as_deref().is_some_and(only_bound)).count()) } /// The standalone table of an id. @@ -1022,77 +973,8 @@ impl Project { Ok(()) } - /// The file keeps no passwords, keys, tokens or signatures in layer addresses. The open project - /// keeps them. This process remembers the 32 most recent distinct saved snapshots with stripped - /// addresses: opening identical plain project text restores them until that snapshot is evicted - /// or RustGIS closes. Packages and imports never restore them. - /// Refuses text larger than the opening limit, restoring live paths and addresses on failure. - pub fn saved_text(&mut self, dir: Option<&Path>) -> Result<(String, usize), String> { - self.ensure_imports_complete()?; - let unrecorded = self.unrecorded_bound(); - let was = strip_addresses(self.addresses_mut()); - let recorded_was = take_login_bindings(self.recorded_mut()); - let n = was.iter().flatten().filter(|original| lost_secret(original)).count() + unrecorded; - let originals: Vec> = if let Some(d) = dir.and_then(Path::to_str) { - self.file_paths_mut() - .into_iter() - .map(|(p, _)| { - let relative = rustgis_map::paths::is_absolute_local(p) - .then(|| rustgis_map::paths::relative_to(d, p)) - .flatten() - // Keep the traversal stable during restoration, even for an unusually deep - // destination whose relative spelling exceeds the local-path size bound. - .filter(|r| rustgis_map::paths::is_local(r)); - relative.map(|r| std::mem::replace(p, r)) - }) - .collect() - } else { - Vec::new() - }; - let text = serde_json::to_string(self).map_err(|e| e.to_string()); - if !originals.is_empty() { - for ((p, _), original) in self.file_paths_mut().into_iter().zip(originals) { - if let Some(original) = original { - *p = original; - } - } - } - // Serialization can fail: put the live addresses back before handling its result. - for (address, original) in self.addresses_mut().into_iter().zip(&was) { - if let Some(original) = original { - *address = original.clone(); - } - } - for (address, original) in self.recorded_mut().into_iter().zip(&recorded_was) { - if let Some(original) = original { - *address = original.clone(); - } - } - let text = text?; - if text.len() as u64 > MAX_BYTES { - return Err(format!("the project is too large to save (maximum {} MiB)", MAX_BYTES >> 20)); - } - if n > 0 { - let (len, hash) = identity(&text); - let mut kept = kept(); - kept.retain(|entry| (entry.len, entry.hash) != (len, hash)); - if kept.len() >= MAX_WRITTEN { - kept.remove(0); - } - kept.push(Written { len, hash, was }); - } - Ok((text, n)) - } - - /// The project as an autosaved recovery copy keeps it: its text without passwords, keys, tokens - /// or signatures in layer addresses, from a copy of the project (autosave writes it off the - /// interface thread). Unlike [`Self::saved_text`], local file paths stay as the open project - /// holds them, absolute: the copy lives in the app's recovery folder and is opened from there, - /// not from the project's folder. Nor is it remembered as a file this process wrote, so - /// frequent copies can't push the user's own saves out of the snapshots that restore secrets. - /// Runtime import reservations are omitted while completed work remains recoverable. - /// Text too large to open again is refused. - pub fn recovery_text(mut self) -> Result { + /// Omits runtime-only reservations on an owned recovery or bridge draft. + pub(crate) fn omit_pending_imports(&mut self) { // A recovery copy preserves completed work without persisting runtime-only reservations. for map in self.maps.iter_mut().chain(self.offline_areas.iter_mut().map(|area| &mut area.data)) { let pending: std::collections::HashSet<_> = map @@ -1107,29 +989,45 @@ impl Project { } map.tiles.retain(|tile| !tile.id.starts_with("geolibre-pending:")); } - strip_addresses(self.addresses_mut()); - clear_login_bindings(self.recorded_mut()); - let text = serde_json::to_string(&self).map_err(|e| e.to_string())?; - if text.len() as u64 > MAX_BYTES { - return Err("the project is too large to keep a recovery copy".into()); - } - Ok(text) } - fn restore_addresses(&mut self, text: &str) { - let id = identity(text); - let kept = kept(); - let Some(entry) = kept.iter().find(|entry| (entry.len, entry.hash) == id) else { return }; - for (address, original) in self.addresses_mut().into_iter().zip(&entry.was) { - if let Some(full) = original - && saved_address(full).as_deref() == Some(address.as_str()) - { - *address = without_login_binding(full).unwrap_or_else(|| full.clone()); - } + /// Writes controlled durable fields and relative paths in one guarded transaction. Live + /// fields are restored on every return and native unwind; only explicit saves are remembered. + pub fn saved_text(&mut self, dir: Option<&Path>) -> Result<(String, crate::durable::Report), String> { + self.ensure_imports_complete()?; + let unrecorded = self.bound_address_losses(false); + let (text, mut report) = crate::durable::output_remembered(self, dir.and_then(Path::to_str), project_text)?; + report.addresses = report.addresses.saturating_add(u32::try_from(unrecorded).unwrap_or(u32::MAX)); + Ok((text, report)) + } + + /// The recovery text alone, without its counted loss report. + /// Recovery stores should use [`Self::recovery_text_with_report`] to retain its notices. + pub fn recovery_text(self) -> Result { + self.recovery_text_with_report().map(|(text, _)| text) + } + + /// The project as an autosaved recovery copy keeps it: its text without passwords, keys, tokens + /// or signatures in layer addresses, from a copy of the project (autosave writes it off the + /// interface thread). Unlike [`Self::saved_text`], local file paths stay as the open project + /// holds them, absolute: the copy lives in the app's recovery folder and is opened from there, + /// not from the project's folder. Nor is it remembered as a file this process wrote, so + /// frequent copies can't push the user's own saves out of the snapshots that restore secrets. + /// Runtime import reservations are omitted while completed work remains recoverable. + /// Text too large to open again is refused. Returns the text and its counted output notices. + pub fn recovery_text_with_report(mut self) -> Result<(String, crate::durable::Report), String> { + self.omit_pending_imports(); + let unrecorded = self.bound_address_losses(false); + let (text, mut report) = crate::durable::output_owned(self, project_text)?; + report.addresses = report.addresses.saturating_add(u32::try_from(unrecorded).unwrap_or(u32::MAX)); + if text.len() as u64 > MAX_BYTES { + return Err("the project is too large to keep a recovery copy".into()); } + Ok((text, report)) } - pub fn save(&mut self, path: &Path) -> Result { + /// Writes checked durable text atomically and returns counted losses from the guarded output. + pub fn save(&mut self, path: &Path) -> Result { let dir = std::path::absolute(path).ok().and_then(|p| p.parent().map(Path::to_path_buf)); let (text, stripped) = self.saved_text(dir.as_deref())?; rustgis_data::io::write_atomic(path, text.as_bytes()).map_err(|e| format!("could not save {}: {e}", path.display()))?; @@ -1138,17 +1036,23 @@ impl Project { /// Reads a project from its JSON text (a browser upload). pub fn from_json(text: &str) -> Result { - Self::read_json(text, true, None) + Self::read_json(text, true, None, true) } /// Reads imported JSON (including package contents) without restoring a saved file's secrets. pub(crate) fn from_import_json(text: &str, dir: Option<&Path>) -> Result { - Self::read_json(text, false, dir) + Self::read_json(text, false, dir, true) } - /// Validate compatibility, clear runtime bindings, resolve paths and normalize accepted - /// content; only ordinary project opens may restore this process's saved addresses. - fn read_json(text: &str, restore: bool, dir: Option<&Path>) -> Result { + /// Peer census must precede normalization: duplicate ids cannot become proof of a unique + /// matching slot after the normalizer renumbers or drops them. Saved-text restore is absent. + pub(crate) fn from_peer_raw(text: &str) -> Result { + Self::read_json(text, false, None, false) + } + + /// Validates compatibility before typed decoding. Saved files may restore exact fields; + /// peer input stays unnormalized until its slot census has rejected ambiguous identities. + fn read_json(text: &str, restore: bool, dir: Option<&Path>, normalize: bool) -> Result { check_format(text, "project")?; let mut p: Project = serde_json::from_str(text).map_err(|e| { let msg = e.to_string(); @@ -1163,7 +1067,7 @@ impl Project { return Err(format!("this project was saved by a newer RustGIS (format {}). Update RustGIS to open it.", p.format)); } if restore { - p.restore_addresses(text); + crate::durable::restore(&mut p, text); } // A binding is this computer's runtime token for a login: one in a file is no authority // to use any login here. @@ -1173,7 +1077,9 @@ impl Project { if let Some(d) = dir.and_then(Path::to_str) { p.resolve_paths(d); } - p.normalize_opened(); + if normalize { + p.normalize_opened(); + } p.label_read_notices = label_losses::notices(text, &p)?; Ok(p) } @@ -1184,7 +1090,7 @@ impl Project { let bytes = read_file(path, MAX_BYTES)?; let text = String::from_utf8(bytes).map_err(|_| format!("{} isn't a RustGIS project", path.display()))?; let dir = std::path::absolute(path).ok().and_then(|p| p.parent().map(Path::to_path_buf)); - Self::read_json(&text, true, dir.as_deref()) + Self::read_json(&text, true, dir.as_deref(), true) } } @@ -1269,6 +1175,191 @@ fn unique_names<'a>(names: impl Iterator) { } } +// Durable ownership is classified here so private fields cannot disappear from the inventory. +use crate::durable::{Durable, FieldMut, Leaf, Owner, Prepare}; + +impl Durable for Project { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + let Self { + label_read_notices: _, + format: _, + name: _, + maps, + active_map: _, + tables, + next_table_id: _, + gp_history, + layouts: _, + folders: _, + styles: _, + ramps: _, + tasks, + gp_env: _, + models, + workspaces: _, + link_views: _, + thumbnail: _, + jobs: _, + offline_areas, + reports: _, + presentations, + notebooks, + dl_models: _, + shared: _, + data_stores: _, + } = self; + for (i, v) in maps.iter().enumerate() { + p.child(Owner::Map(i), v); + } + for (i, v) in offline_areas.iter().enumerate() { + let OfflineArea { name: _, map: _, extent: _, created: _, synced: _, data } = v; + p.child(Owner::Area(i), data); + } + for (i, v) in tables.iter().enumerate() { + p.child(Owner::Table(i), v); + } + for (i, v) in tasks.iter().enumerate() { + p.child(Owner::Task(i), v); + } + for (i, v) in models.iter().enumerate() { + p.child(Owner::Model(i), v); + } + for (i, v) in gp_history.iter().enumerate() { + p.child(Owner::History(i), v); + } + for (i, v) in notebooks.iter().enumerate() { + p.child(Owner::Notebook(i), v); + } + for (i, v) in presentations.iter().enumerate() { + p.child(Owner::Presentation(i), v); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + let (owner, rest) = owners.split_first()?; + match owner { + Owner::Map(i) => self.maps.get_mut(*i)?.field_mut(rest, leaf), + Owner::Area(i) => self.offline_areas.get_mut(*i)?.data.field_mut(rest, leaf), + Owner::Table(i) => self.tables.get_mut(*i)?.field_mut(rest, leaf), + Owner::Task(i) => self.tasks.get_mut(*i)?.field_mut(rest, leaf), + Owner::Model(i) => self.models.get_mut(*i)?.field_mut(rest, leaf), + Owner::History(i) => self.gp_history.get_mut(*i)?.field_mut(rest, leaf), + Owner::Notebook(i) => self.notebooks.get_mut(*i)?.field_mut(rest, leaf), + Owner::Presentation(i) => self.presentations.get_mut(*i)?.field_mut(rest, leaf), + Owner::Graph => None, + } + } +} + +impl Durable for Table { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + let Self { id: _, dataset } = self; + let Dataset { + name: _, + geometry_type: _, + fields: _, + features: _, + source: provenance, + crs: _, + crs_incomplete: _, + zm_dropped: _, + windowed: _, + } = dataset; + if let Some(v) = provenance { + p.string(Leaf::Table, v, true, rustgis_map::paths::is_local(v)); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + if owners.is_empty() && *leaf == Leaf::Table { self.dataset.source.as_mut().map(FieldMut::Text) } else { None } + } +} + +impl Durable for Task { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + let Self { name: _, description: _, steps } = self; + for (i, TaskStep { title: _, instructions: _, command, params }) in steps.iter().enumerate() { + p.command(Leaf::Task(i), Leaf::TaskCommand(i), command, params); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + if !owners.is_empty() { + return None; + } + match leaf { + Leaf::Task(i) => Some(FieldMut::Json(&mut self.steps.get_mut(*i)?.params)), + Leaf::TaskCommand(i) => Some(FieldMut::Text(&mut self.steps.get_mut(*i)?.command)), + _ => None, + } + } +} + +impl Durable for Model { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + let Self { name: _, steps, graph } = self; + for (i, ModelStep { tool, params }) in steps.iter().enumerate() { + p.tool(Leaf::ModelStep(i), Leaf::ModelStepTool(i), tool, params); + } + if let Some(g) = graph { + p.child(Owner::Graph, g); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + if let Some((Owner::Graph, rest)) = owners.split_first() { + return self.graph.as_mut()?.field_mut(rest, leaf); + } + if !owners.is_empty() { + return None; + } + match leaf { + Leaf::ModelStep(i) => Some(FieldMut::Json(&mut self.steps.get_mut(*i)?.params)), + Leaf::ModelStepTool(i) => Some(FieldMut::Text(&mut self.steps.get_mut(*i)?.tool)), + _ => None, + } + } +} + +impl Durable for ModelGraph { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + let Self { nodes, edges: _ } = self; + for (i, ModelNode { id: _, kind: _, x: _, y: _, layer: _, tool, params, name: _ }) in nodes.iter().enumerate() { + p.tool(Leaf::ModelNode(i), Leaf::ModelNodeTool(i), tool.as_deref().unwrap_or(""), params); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + if !owners.is_empty() { + return None; + } + match leaf { + Leaf::ModelNode(i) => Some(FieldMut::Json(&mut self.nodes.get_mut(*i)?.params)), + Leaf::ModelNodeTool(i) => self.nodes.get_mut(*i)?.tool.as_mut().map(FieldMut::Text), + _ => None, + } + } +} + +/// Enforces the opening bound inside the formatter, before any destination is replaced. +fn project_text(p: &Project) -> Result { + let text = serde_json::to_string(p).map_err(|e| e.to_string())?; + if text.len() as u64 > MAX_BYTES { + return Err(format!("the project is too large to save (maximum {} MiB)", MAX_BYTES >> 20)); + } + Ok(text) +} + #[cfg(test)] #[allow(clippy::unwrap_used)] mod login_binding_tests { @@ -1334,10 +1425,33 @@ mod login_binding_tests { }); assert!(project.addresses_mut().is_empty()); let (saved, stripped) = project.saved_text(None).unwrap(); - assert_eq!(stripped, 0); + assert_eq!(stripped.layer_addresses(), 0); assert_eq!(Project::from_json(&saved).unwrap().tasks[0].steps[0].params["url"], note); } + /// String nesting is independent of JSON depth and must never consume the thread's stack. + #[test] + fn hostile_nested_templates_in_recorded_parameters_do_not_recurse() { + let prefix = "pmtiles:x@".repeat(16_384); + let plain = "https://project757.example.org/data.csv"; + let clean = format!("{prefix}{plain}"); + let mut project = Project::default(); + project.tasks.push(Task { + name: "Nested".into(), + description: String::new(), + steps: vec![TaskStep { + title: "Read".into(), + instructions: String::new(), + command: "layer.addUrl".into(), + params: serde_json::json!({"url":format!("{prefix}{}", bound(plain)),"clean":clean}), + }], + }); + let text = serde_json::to_string(&project).unwrap(); + let opened = Project::from_json(&text).unwrap(); + assert_eq!(opened.tasks[0].steps[0].params["url"], clean); + assert_eq!(opened.tasks[0].steps[0].params["clean"], clean); + } + #[test] fn embedded_recorded_bindings_are_removed_from_saves_recovery_and_exports() { let first = "https://project757.example.org/first.csv?token=ordinary#part"; @@ -1367,7 +1481,7 @@ mod login_binding_tests { }); let forged = serde_json::to_string(&session.project).unwrap(); let (saved, stripped) = session.project.saved_text(None).unwrap(); - assert_eq!(stripped, 0); + assert_eq!(stripped.layer_addresses(), 0); let recovery = session.project.clone().recovery_text().unwrap(); for text in [&saved, &recovery] { assert!(!text.contains(rustgis_data::login_address::PREFIX)); @@ -1462,7 +1576,7 @@ mod login_binding_tests { } #[test] - fn a_malformed_wrapper_is_never_emptied() { + fn a_malformed_wrapper_is_withheld_at_durable_output() { let bad = "rustgis-login:nothex:5:https://project757.example.org/{z}/{x}/{y}.png"; let mut project = Project::default(); project.tasks.push(Task { @@ -1475,9 +1589,14 @@ mod login_binding_tests { params: serde_json::json!({"url": bad}), }], }); - let (saved, _) = project.saved_text(None).unwrap(); - let opened = Project::from_json(&saved).unwrap(); - assert_eq!(opened.tasks[0].steps[0].params["url"], bad); + let (saved, report) = project.saved_text(None).unwrap(); + // Inspect the durable bytes: from_json may restore a credential edit from this + // process's explicit-save cache, including an invalid address that still cannot run. + let written: serde_json::Value = serde_json::from_str(&saved).unwrap(); + assert_eq!(written["tasks"][0]["steps"][0]["params"]["url"], rustgis_data::login_address::PREFIX); + assert_eq!((report.withheld, report.steps, report.layer_addresses()), (1, 1, 0)); + assert_eq!(report.notices().len(), 2, "the withheld slot and affected executable step are distinct report dimensions"); + assert_eq!(project.tasks[0].steps[0].params["url"], bad, "the live project is restored"); let mut address = bad.to_string(); clear_login_bindings([&mut address]); assert_eq!(address, bad); @@ -1488,21 +1607,44 @@ mod login_binding_tests { let plain = "https://project757.example.org/{z}/{x}/{y}.png"; let mut project = Project::default(); project.map_mut().tiles.push(tiles("Recorded", &bound(plain), "saved-id")); - assert_eq!(project.saved_text(None).unwrap().1, 0, "a layer that records its connection asks for the login again"); + assert_eq!( + project.saved_text(None).unwrap().1.layer_addresses(), + 0, + "a layer that records its connection asks for the login again" + ); project.map_mut().tiles.push(tiles("Unrecorded", &bound(plain), "")); - assert_eq!(project.saved_text(None).unwrap().1, 1, "a bound layer with no connection can't ask for one"); + assert_eq!(project.saved_text(None).unwrap().1.layer_addresses(), 1, "a bound layer with no connection can't ask for one"); project.map_mut().tiles.clear(); project.map_mut().tiles.push(tiles("Token", &format!("{plain}?token=abc123"), "")); - assert_eq!(project.saved_text(None).unwrap().1, 1, "a real token is still lost"); + assert_eq!(project.saved_text(None).unwrap().1.layer_addresses(), 1, "a real token is still lost"); project.map_mut().tiles.clear(); project.map_mut().tiles.push(tiles("Both", &bound(&format!("{plain}?token=abc123")), "saved-id")); - assert_eq!(project.saved_text(None).unwrap().1, 1, "the token under a binding is lost too"); + assert_eq!(project.saved_text(None).unwrap().1.layer_addresses(), 1, "the token under a binding is lost too"); project.map_mut().tiles.clear(); project.map_mut().tiles.push(tiles("Unrecorded token", &bound(&format!("{plain}?token=abc123")), "")); - assert_eq!(project.saved_text(None).unwrap().1, 1, "a bound layer with no connection and a token is one layer"); + assert_eq!(project.saved_text(None).unwrap().1.layer_addresses(), 1, "a bound layer with no connection and a token is one layer"); + + project.map_mut().tiles.clear(); + let legend = "https://project757.example.org/legend.png"; + let mut tile = tiles("Legend only", plain, ""); + tile.legends = vec![rustgis_data::ogc::Legend { url: bound(legend), ..Default::default() }; 2]; + project.map_mut().tiles.push(tile); + let (saved, report) = project.saved_text(None).unwrap(); + assert_eq!(report.layer_addresses(), 1, "several lost legend logins count their tile once"); + assert_eq!(project.map().tiles[0].legends[0].url, bound(legend), "the live login is restored"); + assert_eq!(Project::from_json(&saved).unwrap().map().tiles[0].legends[0].url, legend, "a saved binding is never restored"); + + project.map_mut().tiles[0].legends.clear(); + project.map_mut().tiles[0].time_url = Some(bound(plain)); + assert_eq!(project.saved_text(None).unwrap().1.layer_addresses(), 1, "an unrecorded time URL loses its login too"); + project.map_mut().tiles[0].time_url = None; + project.map_mut().tiles[0].time_urls = vec![(0.0, bound(plain)), (1.0, bound(plain))]; + assert_eq!(project.saved_text(None).unwrap().1.layer_addresses(), 1, "dated URLs count their tile once"); + project.map_mut().tiles[0].time_urls.clear(); + assert_eq!(project.saved_text(None).unwrap().1.layer_addresses(), 0, "a clean tile loses no login"); } } diff --git a/crates/engine/src/project_log.rs b/crates/engine/src/project_log.rs index 12ac74c07..e1114e025 100644 --- a/crates/engine/src/project_log.rs +++ b/crates/engine/src/project_log.rs @@ -6,6 +6,29 @@ use serde_json::Value; use crate::{CmdResult, Session}; +/// A redacted output projection. Never use it as executable parameters or approval identity. +pub struct LogCopy(Value); + +impl LogCopy { + /// Consumes the output-only wrapper for serialization. + pub fn into_log_value(self) -> Value { + self.0 + } +} + +/// The session log's bounded, document-aware parameter policy, shared with the agent audit. +/// The source value is never mutated and remains the only input to execution and planning. +pub fn log_params(params: &Value) -> LogCopy { + LogCopy(kept_params(params)) +} + +/// Cleans a structural command/tool identifier for display and audit output only. +/// The original identifier remains the sole input to lookup, execution and approval. +pub fn log_identifier(id: &str) -> String { + let public = crate::roles::identifier(id, &mut crate::durable::Report::default()); + crate::durable::generated_text(public.as_deref().unwrap_or(id)) +} + /// Now, in milliseconds since 1970 (UTC). pub(crate) fn now_ms() -> u64 { web_time::SystemTime::now().duration_since(web_time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0) @@ -119,6 +142,41 @@ fn redacted(v: &Value) -> Value { crate::cmd::history::redacted(&embedded_redacted(v, 0)) } +/// A bounded log-only context check for a short key beside nested connection settings. +/// User data/documents do not make an ordinary join or lookup key a credential. +pub(crate) fn address_context(key: &str, v: &Value, depth: usize) -> bool { + if depth >= 8 || matches!(key, "data" | "text" | "values" | "attributes" | "properties" | "features" | "geometry") { + return false; + } + match v { + Value::String(s) => { + let s = s.trim_start(); + [ + "http://", + "https://", + "s3://", + "rustgis-login:", + "rustgis-anonymous:", + "rustgis-s3:", + "postgres:", + "postgresql:", + "pg:", + "mysql:", + "mariadb:", + "mssql:", + "pmtiles:", + "cog:", + ] + .iter() + .any(|prefix| s.get(..prefix.len()).is_some_and(|head| head.eq_ignore_ascii_case(prefix))) + } + Value::Array(values) => values.iter().take(LOG_LIST_ITEMS).any(|v| address_context("", v, depth + 1)), + Value::Object(values) => values.iter().any(|(key, v)| address_context(key, v, depth + 1)), + _ => false, + } +} + +/// Scrubs bounded nested log parameters and JSON embedded in text before the log retains them. fn embedded_redacted(v: &Value, depth: usize) -> Value { match v { Value::String(t) => { @@ -126,7 +184,17 @@ fn embedded_redacted(v: &Value, depth: usize) -> Value { Value::String(sql_secrets_removed(&t)) } Value::Array(a) if depth < 8 => Value::Array(a.iter().map(|x| embedded_redacted(x, depth + 1)).collect()), - Value::Object(o) if depth < 8 => Value::Object(o.iter().map(|(k, x)| (k.clone(), embedded_redacted(x, depth + 1))).collect()), + Value::Object(o) if depth < 8 => Value::Object( + o.iter() + .map(|(k, x)| { + // Preserve History's plain-key refinement; the log-only fields share diagnostics' + // canonical list. Clipping already bounds recursion and text before this pass. + let sensitive = + rustgis_data::address::normalized_name(k).as_deref() != Some("key") && rustgis_data::address::log_sensitive(k); + (k.clone(), if sensitive { Value::String(String::new()) } else { embedded_redacted(x, depth + 1) }) + }) + .collect(), + ), other => other.clone(), } } diff --git a/crates/engine/src/roles.rs b/crates/engine/src/roles.rs new file mode 100644 index 000000000..cc3897628 --- /dev/null +++ b/crates/engine/src/roles.rs @@ -0,0 +1,477 @@ +//! Replayable parameters are cleaned by role, never by recursively deleting secret-named data +//! properties. Unknown nested content and foreign notebook cells remain opaque, with honest +//! notices. Logs deliberately have a separate, stricter policy. + +use std::collections::HashMap; +use std::sync::OnceLock; + +use serde::{Serialize, de::DeserializeOwned}; +use serde_json::Value; + +use crate::durable::{self, Durable, Kept, Report}; + +/// A replayable parameter's durable-output role. Permission policy does not consume roles. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct ParamRole { + pub key: &'static str, + pub role: Role, +} +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Role { + Secret, + Dataset, + Configuration, + Sql, + Opaque, + ToolParams, + Script, + TaskSteps, + ModelSteps, + ModelGraph, + Document(Document), +} +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Document { + Project, + Map, + Task, + Notebook, + Presentation, + Model, +} + +/// Compatibility metadata is read before any embedded document uses this version's types. +#[derive(serde::Deserialize)] +struct DocumentFormat { + format: u64, +} + +const WITHHELD: &str = "[parameters withheld]"; +const NESTED: &str = "[withheld: nested too deep]"; +const DATA: &[&str] = &[ + "text", + "data", + "values", + "value", + "attributes", + "properties", + "features", + "feature", + "geometry", + "rows", + "expression", + "where", + "html", + "markdown", + "caption", + "title", + "notes", + "description", + "label", + "name", + "field", + "fields", +]; +const ADDRESS: &[&str] = + &["url", "urls", "href", "source", "template", "tiles", "store", "endpoint", "address", "uri", "connection", "database"]; +const CONFIG: &[&str] = &["headers", "options", "auth", "connection", "source", "sources"]; + +/// Durable output reads its own field from reviewed builtin registrations. An explicit empty +/// declaration retains the reviewed common scalar/address/configuration rules below. +fn metadata(id: &str) -> Option<&'static [ParamRole]> { + static ROLES: OnceLock> = OnceLock::new(); + ROLES.get_or_init(|| crate::cmd::all().into_iter().filter_map(|c| c.roles.map(|r| (c.id, r))).collect()).get(id).copied() +} + +/// A conservative bounded read-only check, used solely to announce unchecked/withheld content. +fn suspicious(v: &Value) -> bool { + crate::sources::holds_secret(v) + || v.as_str().is_some_and(|s| { + s.split(|c: char| !(c.is_ascii_alphanumeric() || matches!(c, '_' | '-' | '%'))) + .take(65_536) + .any(|word| !word.is_empty() && rustgis_data::address::secret_key(word)) + }) +} + +/// Keeps invalid executable content invalid while withholding credential-bearing text. +fn invalid(value: &Value, report: &mut Report) -> Option { + if suspicious(value) { + report.withheld = report.withheld.saturating_add(1); + Some(Value::String(WITHHELD.into())) + } else { + None + } +} + +/// Cleans one command without treating parameter JSON strings as executor envelopes. +pub(crate) fn command(id: &str, params: &Value, depth: usize) -> (Option, Report) { + let mut report = Report::default(); + let changed = object(id, params, depth, 0, metadata(id).is_some(), &mut report); + (changed, report) +} + +/// Tools use the common parameter rules, including data-valued `input`/`values`; command-specific +/// metadata belongs to command registrations, not the separate tool registry. +pub(crate) fn tool(id: &str, params: &Value, depth: usize) -> (Option, Report) { + let mut report = Report::default(); + let builtin = rustgis_analysis::TOOLS.iter().any(|t| t.id == id); + let changed = object("", params, depth, 0, builtin, &mut report); + (changed, report) +} + +/// Identifiers are structural executable fields, not layer names or user data. Recognized +/// addresses are cleaned; credential-bearing malformed payloads are withheld as a whole. +pub(crate) fn identifier(id: &str, report: &mut Report) -> Option { + // A colon-separated credential key/value is payload; a qualified plugin/tool id alone is not. + let colon_value = id + .split(':') + .zip(id.split(':').skip(1)) + .take(65_536) + .any(|(key, value)| !value.is_empty() && rustgis_data::address::secret_key(key)); + match durable::public_address(id) { + Kept::Changed(public) => Some(public), + Kept::Withheld(prefix) if prefix != id => { + report.withheld = report.withheld.saturating_add(1); + Some(prefix.into()) + } + _ if (colon_value || id.contains(['=', '{', '[', '"']) || id.contains(char::is_whitespace)) + && suspicious(&Value::String(id.into())) => + { + report.withheld = report.withheld.saturating_add(1); + Some("[command withheld]".into()) + } + _ => None, + } +} + +/// Lazily copies only changed owning parameter objects. Configuration recursion is bounded. +fn object(id: &str, value: &Value, depth: usize, config_depth: usize, registered: bool, report: &mut Report) -> Option { + if value.is_null() { + return None; + } + let Some(obj) = value.as_object() else { return invalid(value, report) }; + if config_depth >= 8 { + return invalid(value, report); + } + let mut replacement = None; + let mut unchecked = false; + for (key, v) in obj { + unchecked |= !matches!(durable::public_address(key), Kept::Unchanged); + let k = rustgis_data::address::normalized_name(key).unwrap_or_else(|| key.to_ascii_lowercase()); + let role = if registered && config_depth == 0 { + metadata(id).and_then(|roles| roles.iter().find(|r| r.key == k).map(|r| r.role)) + } else { + None + }; + let clean = if let Some( + role @ (Role::ToolParams | Role::Script | Role::TaskSteps | Role::ModelSteps | Role::ModelGraph | Role::Document(_)), + ) = role + { + if depth >= 4 { + report.withheld = report.withheld.saturating_add(1); + Some(Value::String(NESTED.into())) + } else { + envelope(role, v, obj, depth + 1, report) + } + } else if role == Some(Role::Opaque) { + unchecked |= suspicious(v); + None + } else if (role == Some(Role::Secret) || rustgis_data::address::secret_key(key)) + && (k != "key" + || crate::sources::looks_like_a_key(v.as_str().unwrap_or("")) + || obj.iter().any(|(other, value)| other != key && !DATA.contains(&other.as_str()) && address_like(value))) + { + match v { + Value::String(s) if !s.is_empty() => Some(Value::String(String::new())), + Value::Number(_) => Some(Value::String(String::new())), + // Unknown structured credentials keep their shape and are reported as unchecked. + Value::Object(o) if !o.is_empty() => { + unchecked = true; + None + } + Value::Array(a) if !a.is_empty() => { + unchecked = true; + None + } + _ => None, + } + } else if role == Some(Role::Dataset) || DATA.contains(&k.as_str()) { + None + } else if role == Some(Role::Sql) { + v.as_str().and_then(|s| { + let clean = crate::project_log::sql_secrets_removed(s); + (clean != s).then_some(Value::String(clean)) + }) + } else if ADDRESS.contains(&k.as_str()) || address_like(v) { + if v.is_object() && registered && (role == Some(Role::Configuration) || CONFIG.contains(&k.as_str())) { + object("", v, depth, config_depth + 1, true, report) + } else { + // String leaves are cleaned below; opaque shapes keep their meaning. + unchecked |= match v { + Value::Object(_) => suspicious(v), + Value::Array(a) => a.iter().any(|v| !v.is_string() && suspicious(v)), + _ => false, + }; + address(v, report) + } + } else if registered && (role == Some(Role::Configuration) || CONFIG.contains(&k.as_str())) && v.is_object() { + object("", v, depth, config_depth + 1, true, report) + } else { + unchecked |= suspicious(v); + None + }; + if !registered && !DATA.contains(&k.as_str()) && (v.is_object() || v.is_array()) { + unchecked |= suspicious(v); + } + if let Some(clean) = clean.filter(|clean| clean != v) { + replacement.get_or_insert_with(|| obj.clone()).insert(key.clone(), clean); + } + } + if unchecked { + report.unchecked = report.unchecked.saturating_add(1); + } + replacement.map(Value::Object) +} + +/// Recognition precedes parsing, including malformed wrappers and differently cased prefixes. +fn address_like(v: &Value) -> bool { + match v { + Value::String(s) => { + let s = s.trim_start(); + [ + "http:", + "https:", + "s3:", + "rustgis-login:", + "rustgis-anonymous:", + "rustgis-s3:", + "pmtiles:", + "cog:", + "postgres:", + "postgresql:", + "pg:", + ] + .iter() + .any(|p| s.get(..p.len()).is_some_and(|s| s.eq_ignore_ascii_case(p))) + } + Value::Array(a) => a.iter().any(|v| v.is_string() && address_like(v)), + _ => false, + } +} + +/// Array addresses are string leaves only; data and nested opaque objects are never walked. +fn address(v: &Value, report: &mut Report) -> Option { + match v { + Value::String(s) => match durable::public_address(s) { + Kept::Unchanged => None, + Kept::Changed(s) => Some(Value::String(s)), + Kept::Withheld(prefix) if prefix != s => { + report.withheld = report.withheld.saturating_add(1); + Some(Value::String(prefix.into())) + } + Kept::Withheld(_) => None, + }, + Value::Array(a) => { + let mut changed = None; + for (i, v) in a.iter().enumerate() { + if v.is_string() + && let Some(clean) = address(v, report) + && let Some(slot) = changed.get_or_insert_with(|| a.clone()).get_mut(i) + { + *slot = clean; + } + } + changed.map(Value::Array) + } + _ => None, + } +} + +/// Explicit executable and document roles are the only route into nested executable content. +fn envelope(role: Role, v: &Value, siblings: &serde_json::Map, depth: usize, report: &mut Report) -> Option { + match role { + Role::ToolParams => { + let (clean, r) = tool(siblings.get("tool").and_then(Value::as_str).unwrap_or(""), v, depth); + report.merge(&r); + clean + } + Role::Script => match v.as_str() { + Some(s) => { + let (clean, r) = script(s, depth); + report.merge(&r); + clean.map(Value::String) + } + None => invalid(v, report), + }, + Role::TaskSteps => typed_value::(v, depth, report), + Role::ModelSteps => typed_value::(v, depth, report), + Role::ModelGraph => typed_value::(v, depth, report), + Role::Document(kind) => document(kind, v, depth, report), + _ => None, + } +} + +/// A typed envelope is prepared on its own copy, preserving the live outer params as a whole. +fn typed_value(v: &Value, depth: usize, report: &mut Report) -> Option { + let Ok(parsed) = serde_json::from_value::(v.clone()) else { return invalid(v, report) }; + let Ok((clean, r)) = durable::output_owned_changed(parsed, depth, |v, changed| { + if changed { serde_json::to_value(v).map(Some).map_err(|e| e.to_string()) } else { Ok(None) } + }) else { + return invalid(v, report); + }; + report.merge(&r); + clean +} + +#[derive(serde::Serialize, serde::Deserialize)] +#[serde(transparent)] +struct VecTask(Vec); + +impl Durable for VecTask { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut durable::Prepare<'_>) { + for (i, st) in self.0.iter().enumerate() { + p.command(durable::Leaf::Task(i), durable::Leaf::TaskCommand(i), &st.command, &st.params); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[durable::Owner], leaf: &durable::Leaf) -> Option> { + if !owners.is_empty() { + return None; + } + match leaf { + durable::Leaf::Task(i) => Some(durable::FieldMut::Json(&mut self.0.get_mut(*i)?.params)), + durable::Leaf::TaskCommand(i) => Some(durable::FieldMut::Text(&mut self.0.get_mut(*i)?.command)), + _ => None, + } + } +} + +#[derive(serde::Serialize, serde::Deserialize)] +#[serde(transparent)] +struct VecSteps(Vec); + +impl Durable for VecSteps { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut durable::Prepare<'_>) { + for (i, st) in self.0.iter().enumerate() { + p.tool(durable::Leaf::ModelStep(i), durable::Leaf::ModelStepTool(i), &st.tool, &st.params); + } + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[durable::Owner], leaf: &durable::Leaf) -> Option> { + if !owners.is_empty() { + return None; + } + match leaf { + durable::Leaf::ModelStep(i) => Some(durable::FieldMut::Json(&mut self.0.get_mut(*i)?.params)), + durable::Leaf::ModelStepTool(i) => Some(durable::FieldMut::Text(&mut self.0.get_mut(*i)?.tool)), + _ => None, + } + } +} + +/// Reads documents at the existing reader bound, without normalization, trust or restore-cache +/// access. Unchanged documents keep their exact text; only changed typed copies are serialized. +fn document(kind: Document, v: &Value, depth: usize, report: &mut Report) -> Option { + let Some(text) = v.as_str().filter(|s| s.len() as u64 <= crate::project::MAX_BYTES) else { return invalid(v, report) }; + if serde_json::from_str::(text).is_ok_and(|f| f.format > u64::from(crate::project::FORMAT)) { + report.unchecked = report.unchecked.saturating_add(1); + return None; + } + let result = match kind { + Document::Project => text_document::(text, depth), + Document::Map => { + if serde_json::from_str::(text).is_ok() { + text_document::(text, depth) + } else { + text_document::(text, depth) + } + } + Document::Task => text_document::(text, depth), + Document::Presentation => text_document::(text, depth), + Document::Notebook => crate::cmd::notebook::durable_document(text, depth), + Document::Model => model_document(text, depth), + }; + match result { + Ok((clean, r)) => { + let changed = clean != text; + report.merge(&r); + changed.then_some(Value::String(clean)) + } + Err(_) => invalid(v, report), + } +} + +/// Uses the owning type's ordinary deserializer, without losing unchanged text representation. +fn text_document(text: &str, depth: usize) -> Result<(String, Report), String> { + let parsed = serde_json::from_str::(text).map_err(|e| e.to_string())?; + durable::output_owned_changed(parsed, depth, |value, changed| { + if changed { serde_json::to_string(value).map_err(|e| e.to_string()) } else { Ok(text.into()) } + }) +} + +/// Model import has two envelopes; keeps the envelope's other fields and cleans only its model. +fn model_document(text: &str, depth: usize) -> Result<(String, Report), String> { + let mut v: Value = serde_json::from_str(text).map_err(|e| e.to_string())?; + let mut report = Report::default(); + let Some(obj) = v.as_object_mut() else { return Err("not a model".into()) }; + let (key, graph) = if obj.contains_key("graph") { ("graph", true) } else { ("steps", false) }; + let Some(field) = obj.get(key) else { return Err("not a model".into()) }; + let clean = if graph { + typed_value::(field, depth, &mut report) + } else { + typed_value::(field, depth, &mut report) + }; + if let Some(clean) = clean { + obj.insert(key.into(), clean); + Ok((v.to_string(), report)) + } else { + Ok((text.into(), report)) + } +} + +/// Uses the executor's line parser, preserving comments, blank lines and unchanged formatting. +pub(crate) fn script(source: &str, depth: usize) -> (Option, Report) { + let mut report = Report::default(); + let mut out = None; + let mut offset = 0usize; + for line in source.split_inclusive('\n') { + let ending = if line.ends_with("\r\n") { + "\r\n" + } else if line.ends_with('\n') { + "\n" + } else { + "" + }; + let replacement = match crate::script_line(line) { + None => None, + Some(Ok((id, params))) => { + let public_id = identifier(&id, &mut report); + let (clean, r) = command(&id, ¶ms, depth); + report.merge(&r); + match clean { + Some(params) => Some(format!("{} {params}{ending}", public_id.as_deref().unwrap_or(&id))), + None => public_id.map(|public| line.replacen(&id, &public, 1)), + } + } + Some(Err(_)) if suspicious(&Value::String(line.into())) => { + report.withheld = report.withheld.saturating_add(1); + let id = line.trim_start().split(char::is_whitespace).next().unwrap_or(""); + let public_id = identifier(id, &mut report); + Some(format!("{} {WITHHELD}{ending}", public_id.as_deref().unwrap_or(id))) + } + Some(Err(_)) => None, + }; + if let Some(clean) = replacement { + report.lines = report.lines.saturating_add(1); + out.get_or_insert_with(|| source.get(..offset).unwrap_or("").to_string()).push_str(&clean); + } else if let Some(out) = &mut out { + out.push_str(line); + } + offset += line.len(); + } + (out, report) +} diff --git a/crates/engine/src/s3_saved_tests.rs b/crates/engine/src/s3_saved_tests.rs index ff3df4c52..4a936b0a4 100644 --- a/crates/engine/src/s3_saved_tests.rs +++ b/crates/engine/src/s3_saved_tests.rs @@ -256,6 +256,10 @@ fn bound(s: &Session, id: &str, url: &str) -> String { #[test] fn s3_saved_keys_are_remembered_only_on_request_and_reused_by_a_fresh_session() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_keys_are_remembered_only_on_request_and_reused_by_a_fresh_session") { + return; + } let srv = serve_bucket("sv-remember", vec![("grid.fgb", grid_fgb())], vec![]); let store = Arc::new(Spy::default()); let mut a = session(Some(&store)); @@ -305,6 +309,10 @@ fn s3_saved_keys_are_remembered_only_on_request_and_reused_by_a_fresh_session() #[test] fn s3_saved_login_rules_sources_temporary_credentials_and_bad_values() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_login_rules_sources_temporary_credentials_and_bad_values") { + return; + } let srv = serve_bucket("sv-rules", vec![], vec![]); let store = Arc::new(Spy::default()); let mut s = session(Some(&store)); @@ -404,6 +412,10 @@ fn s3_saved_login_rules_sources_temporary_credentials_and_bad_values() { #[test] fn s3_saved_missing_or_refusing_stores_keep_the_session_login() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_missing_or_refusing_stores_keep_the_session_login") { + return; + } let srv = serve_bucket("sv-stores", vec![], vec![]); // No store at all (the browser, the command line). let mut s = session(None); @@ -440,6 +452,10 @@ fn s3_saved_missing_or_refusing_stores_keep_the_session_login() { #[test] fn s3_saved_lifecycle_keeps_deletes_and_forgets_keys() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_lifecycle_keeps_deletes_and_forgets_keys") { + return; + } let srv = serve_bucket("sv-life", vec![], vec![]); let spy = Arc::new(Spy::default()); let mut s = session(Some(&spy)); @@ -528,6 +544,11 @@ fn s3_saved_lifecycle_keeps_deletes_and_forgets_keys() { #[test] fn s3_saved_a_binding_signs_only_its_own_connection_and_errors_never_echo_the_key() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_a_binding_signs_only_its_own_connection_and_errors_never_echo_the_key") + { + return; + } let moved_to = serve_bucket("sv-bind", vec![("elsewhere.fgb", vec![1, 2, 3])], vec![]); let srv = serve_bucket("sv-bind", vec![("grid.fgb", grid_fgb())], vec![("moved.fgb", format!("{}/sv-bind/elsewhere.fgb", moved_to.base))]); @@ -665,6 +686,10 @@ fn reopened(p: &Saved, store: &Arc, keep: usize) -> Session { #[test] fn s3_saved_projects_hold_no_keys_and_reload_through_their_one_connection() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_projects_hold_no_keys_and_reload_through_their_one_connection") { + return; + } let p = saved_project("sv-proj-one"); for secret in ["SAVEDKEYA", "SAVEDKEYB", "saved-secret"] { assert!(!p.text.contains(secret) && !p.prefs.contains(secret), "{secret}"); @@ -736,6 +761,10 @@ fn s3_saved_projects_hold_no_keys_and_reload_through_their_one_connection() { #[test] fn s3_saved_several_candidates_wait_for_a_choice_and_nothing_is_fetched_before() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_several_candidates_wait_for_a_choice_and_nothing_is_fetched_before") { + return; + } let p = saved_project("sv-proj-two"); let store = p.store.clone(); let (from, saves) = (p.srv.requests(), store.count("save")); @@ -785,6 +814,12 @@ fn s3_saved_several_candidates_wait_for_a_choice_and_nothing_is_fetched_before() #[test] fn s3_saved_sources_without_keys_zero_matches_and_older_addresses_stay_blocked_or_as_they_were() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test( + "s3_saved_tests::s3_saved_sources_without_keys_zero_matches_and_older_addresses_stay_blocked_or_as_they_were", + ) { + return; + } let p = saved_project("sv-proj-edge"); let from = p.srv.requests(); // One candidate, whose keys are not in the store (another computer): blocked, pending for its @@ -868,6 +903,10 @@ fn s3_saved_sources_without_keys_zero_matches_and_older_addresses_stay_blocked_o #[test] fn s3_saved_imported_maps_layers_and_packages_never_bring_their_own_login() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_imported_maps_layers_and_packages_never_bring_their_own_login") { + return; + } let p = saved_project("sv-proj-import"); let token = rustgis_data::s3_address::parse(&p.fgb).unwrap().unwrap().binding.to_string(); let from = p.srv.requests(); @@ -910,6 +949,10 @@ fn s3_saved_imported_maps_layers_and_packages_never_bring_their_own_login() { #[test] fn s3_saved_typed_keys_are_in_no_script_result_notice_or_list_the_app_shows() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_typed_keys_are_in_no_script_result_notice_or_list_the_app_shows") { + return; + } let srv = serve_bucket("sv-seen", vec![], vec![]); let store = Arc::new(Spy::default()); let mut s = session(Some(&store)); @@ -973,6 +1016,12 @@ fn s3_saved_typed_keys_are_in_no_script_result_notice_or_list_the_app_shows() { #[test] fn s3_saved_database_and_s3_share_one_store_without_crossing_credentials_or_cleanup() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test( + "s3_saved_tests::s3_saved_database_and_s3_share_one_store_without_crossing_credentials_or_cleanup", + ) { + return; + } let store = Arc::new(Spy::default()); let mut s = session(Some(&store)); let db = s @@ -1037,6 +1086,10 @@ fn s3_saved_database_and_s3_share_one_store_without_crossing_credentials_or_clea #[test] fn s3_saved_raster_pixels_wait_for_selection_then_reload_with_style_intact() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_raster_pixels_wait_for_selection_then_reload_with_style_intact") { + return; + } let bytes = rustgis_data::raster::cog::example(256, 256); let pixels = rustgis_data::raster::read_bytes("scene.tif", &bytes).unwrap(); let srv = serve_bucket("sv-raster", vec![("scene.tif", bytes)], vec![]); @@ -1085,6 +1138,10 @@ fn s3_saved_raster_pixels_wait_for_selection_then_reload_with_style_intact() { #[test] fn s3_saved_resolving_one_bucket_does_not_retry_another_buckets_failed_source() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_resolving_one_bucket_does_not_retry_another_buckets_failed_source") { + return; + } let srv = serve_bucket("sv-group-one", Vec::new(), Vec::new()); let store = Arc::new(Spy::default()); let mut seed = session(Some(&store)); @@ -1121,6 +1178,12 @@ fn s3_saved_resolving_one_bucket_does_not_retry_another_buckets_failed_source() #[test] fn s3_saved_browse_reauthenticates_this_session_instead_of_reusing_a_foreign_namesake() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test( + "s3_saved_tests::s3_saved_browse_reauthenticates_this_session_instead_of_reusing_a_foreign_namesake", + ) { + return; + } let srv = serve_bucket("sv-session-owner", Vec::new(), Vec::new()); let store_a = Arc::new(Spy::default()); let store_b = Arc::new(Spy::default()); @@ -1155,6 +1218,12 @@ fn s3_saved_browse_reauthenticates_this_session_instead_of_reusing_a_foreign_nam #[test] fn s3_saved_favourites_keep_provenance_overhead_without_relaxing_destination_limits() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test( + "s3_saved_tests::s3_saved_favourites_keep_provenance_overhead_without_relaxing_destination_limits", + ) { + return; + } let endpoint = "https://s3.example.org"; let inner = format!("{endpoint}/bucket/{}.fgb?versionId={}", "k".repeat(990), "v".repeat(1000)); assert!(inner.len() <= 2048); @@ -1174,6 +1243,11 @@ fn s3_saved_favourites_keep_provenance_overhead_without_relaxing_destination_lim #[test] fn s3_saved_legacy_choice_never_rebinds_an_explicit_aws_source_for_the_same_object() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_legacy_choice_never_rebinds_an_explicit_aws_source_for_the_same_object") + { + return; + } for aws_first in [false, true] { let bucket = format!("sv-mixed-{aws_first}"); let srv = serve_bucket(&bucket, vec![("same.fgb", grid_fgb())], Vec::new()); @@ -1226,6 +1300,10 @@ fn s3_saved_legacy_choice_never_rebinds_an_explicit_aws_source_for_the_same_obje #[test] fn s3_saved_unscoped_connection_requires_a_choice_and_does_not_repeat_the_prompt() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_unscoped_connection_requires_a_choice_and_does_not_repeat_the_prompt") { + return; + } let p = saved_project("sv-unscoped"); let mut s = reopened(&p, &p.store, 1); s.connections.connections[0].options.remove("buckets"); @@ -1242,6 +1320,10 @@ fn s3_saved_unscoped_connection_requires_a_choice_and_does_not_repeat_the_prompt #[test] fn s3_saved_stale_marker_cannot_skip_credential_deletion() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_stale_marker_cannot_skip_credential_deletion") { + return; + } for (index, command) in ["connection.forgetSavedKeys", "connection.remove", "connection.edit"].into_iter().enumerate() { let spy = Arc::new(Spy::default()); let mut s = session(Some(&spy)); @@ -1278,6 +1360,10 @@ fn s3_saved_stale_marker_cannot_skip_credential_deletion() { #[test] fn s3_saved_changing_away_from_keys_forgets_the_pair() { + #[cfg(not(target_arch = "wasm32"))] + if crate::tests::isolate_process_test("s3_saved_tests::s3_saved_changing_away_from_keys_forgets_the_pair") { + return; + } let srv = serve_bucket("sv-mode-change", vec![], vec![]); let spy = Arc::new(Spy::default()); let mut s = session(Some(&spy)); diff --git a/crates/engine/src/sources.rs b/crates/engine/src/sources.rs index a00815378..e8315d6e6 100644 --- a/crates/engine/src/sources.rs +++ b/crates/engine/src/sources.rs @@ -160,7 +160,13 @@ pub fn clean(mut s: Source) -> Result { if !KINDS.contains(&s.kind.as_str()) { return Err(format!("unknown kind '{}' ({})", s.kind, KINDS.join(", "))); } - s.address = redact(s.address.trim()); + let address = s.address.trim(); + // A display-only prefix is safe to show, but cannot be used as a saved source. + s.address = match crate::durable::public_address(address) { + crate::durable::Kept::Withheld(_) => return Err("a source needs an address (a path, web address or connection)".into()), + crate::durable::Kept::Changed(public) => public, + crate::durable::Kept::Unchanged => redact(address), + }; // Provenance has its own structural bound; it must not consume the destination's limit. let (head, archive) = template_archive(&s.address).unwrap_or(("", s.address.as_str())); let bounded = match rustgis_data::s3_address::parse(archive) { @@ -344,31 +350,28 @@ pub(crate) fn template_archive(address: &str) -> Option<(&str, &str)> { /// saved S3 connection is redacted inside (its inner address), and its length-prefixed form /// rebuilt; a tile template, in its archive. Web-login runtime bindings are never published. pub fn redact(address: &str) -> String { - if let Some((head, archive)) = template_archive(address) { - return format!("{head}{}", redact(archive)); - } - let inner = crate::render::rustgis_tiles::logins::inner(address); - if inner != address { - return redact(inner); - } - if let Ok(Some(b)) = rustgis_data::s3_address::parse(address) - && let Ok(rebuilt) = rustgis_data::s3_address::bind(b.binding, b.endpoint, &redact(b.url)) + match crate::durable::public_address(address) { + crate::durable::Kept::Changed(s) => return s, + crate::durable::Kept::Withheld(prefix) => return prefix.into(), + crate::durable::Kept::Unchanged => {} + } + // A clean recognized wrapper or database label already passed the shared owner. Keep its + // layout, including a COG look; the generic scheme fallback is for older connection forms. + if template_archive(address).is_some() + || rustgis_data::s3_address::is_wrapper(address) + || ["rustgis-anonymous:", "postgres:", "postgresql:", "pg:"] + .iter() + .any(|p| address.trim_start().get(..p.len()).is_some_and(|s| s.eq_ignore_ascii_case(p))) { - return rebuilt; + return address.to_string(); } - if let Some((public, _)) = rustgis_data::source::myconn::split_password(address) { + if let Some((mut public, _)) = rustgis_data::source::myconn::split_password(address) { + rustgis_data::address::redact_query(&mut public); return public; } if let Some((public, _)) = rustgis_data::source::msconn::split_password(address) { return public; } - let lower = address.to_ascii_lowercase(); - // A PostgreSQL connection (`postgres:` or `postgresql:` keyword strings, GDAL's `PG:`, URIs): - // libpq's own rules (quoted values, escapes, `key = value`), without its secrets. - if ["postgres:", "postgresql:", "pg:"].iter().any(|p| lower.starts_with(p)) { - // One that can't be taken apart isn't kept (an empty address, refused by `clean`). - return rustgis_data::source::pgconn::public_label(address).unwrap_or_default(); - } let Some((scheme, rest)) = address.split_once("://") else { return address.to_string() }; // The user and password go first, before anything splits the address: a password may hold // an unencoded `/`, `?`, `#` or `@`. They end at the last `@` before the first `/`, or (a `/` @@ -748,9 +751,15 @@ MIIE for f in ["#Access_Token=t", "#sig=s", "#password=p", "#id_token=i", "#map=4/1/2"] { assert_eq!(redact(&format!("https://h.example/x.fgb{f}")), "https://h.example/x.fgb", "{f}"); } - // A connection with nothing to take apart isn't kept. - for bare in ["postgres:dbname", "PG:", "postgres:password='x"] { - assert!(clean(Source { kind: "database".into(), address: bare.into(), ..Source::default() }).is_err(), "{bare}"); + // Unreadable connections, web addresses and wrappers cannot become saved sources. + for (kind, bare) in [ + ("database", "postgres:dbname"), + ("database", "PG:"), + ("database", "postgres:password='x"), + ("url", "https:///path?token=RGSYNTH_unusable_source_20261104"), + ("tiles", "pmtiles:unreadable"), + ] { + assert!(clean(Source { kind: kind.into(), address: bare.into(), ..Source::default() }).is_err(), "{bare}"); } assert_eq!( redact("https://acct.blob.core.windows.net/c/a.tif?se=2026&sp=r&sv=1&sig=zzz"), diff --git a/crates/engine/src/tests.rs b/crates/engine/src/tests.rs index 1b6fd7bc8..66b780a66 100644 --- a/crates/engine/src/tests.rs +++ b/crates/engine/src/tests.rs @@ -5,6 +5,9 @@ use crate::{Project, Session, sample}; #[path = "tests/task_reentry.rs"] mod task_reentry; +#[path = "tests/credential_output.rs"] +mod credential_output; + #[cfg(not(target_arch = "wasm32"))] #[path = "tests/remembered_database.rs"] mod remembered_database; @@ -10130,7 +10133,7 @@ fn jupyter_python_notebooks_survive_open_and_save() { let nb = s.project.notebooks.last().unwrap(); let _ = (nb.language_label(), nb.cells.iter().map(crate::cmd::notebook::Cell::images).count()); let i = s.project.notebooks.len() - 1; - assert!(crate::cmd::notebook::notebook_file(&s, i, true).is_ok(), "{junk}"); + assert!(crate::cmd::notebook::notebook_file(&mut s, i, true).is_ok(), "{junk}"); s.run("notebook.remove", &json!({"notebook": i})).unwrap(); } } @@ -10141,7 +10144,7 @@ fn jupyter_python_notebooks_survive_open_and_save() { s.run("notebook.open", &json!({"text": cmds.to_string()})).unwrap(); s.run("notebook.run", &json!({"cell": 0})).unwrap(); let i = s.project.notebooks.len() - 1; - let saved: Value = serde_json::from_slice(&crate::cmd::notebook::notebook_file(&s, i, true).unwrap()).unwrap(); + let saved: Value = serde_json::from_slice(&crate::cmd::notebook::notebook_file(&mut s, i, true).unwrap()).unwrap(); let cell = &saved["cells"][0]; assert_eq!(cell["id"], "go"); assert_eq!(cell["execution_count"], Value::Null); @@ -12954,35 +12957,31 @@ fn adding_a_collection_of_3d_tiles_says_why_it_is_left_out() { fn an_arcgis_token_in_a_service_address_reaches_its_layers() { use crate::render::rustgis_tiles::arcgis_auth; let mut s = Session::new(Project::default()); - let base = "https://secure914.example.org/server/rest/services/Roads/MapServer"; + let base = "https://secure914.invalid/server/rest/services/Roads/MapServer"; let doc = r#"{"currentVersion": 11, "mapName": "Roads", "capabilities": "Map,Query,Data", "singleFusedMapCache": true, "tileInfo": {"rows": 256, "spatialReference": {"wkid": 102100}, "lods": [{"level": 0, "resolution": 156543.033928041}]}, "layers": [{"id": 0, "name": "Highways"}]}"#; - let v = s.run("service.layers", &json!({"url": format!("{base}?f=json&token=secret.Token-914"), "text": doc})).unwrap(); - assert!(!v.to_string().contains("secret.Token-914"), "{v}"); + let v = s.run("service.layers", &json!({"url": format!("{base}?f=json&%2574oken=RGSYNTH_3"), "text": doc})).unwrap(); + assert!(!v.to_string().contains("RGSYNTH_3"), "{v}"); let tiles = format!("{base}/tile/0/0/0"); let query = format!("{base}/0/query?where=1%3D1&f=geojson"); - assert_eq!(arcgis_auth::authorization(&tiles).as_deref(), Some("Bearer secret.Token-914")); - assert_eq!(arcgis_auth::authorization(&query).as_deref(), Some("Bearer secret.Token-914")); + assert_eq!(arcgis_auth::authorization(&tiles).as_deref(), Some("Bearer RGSYNTH_3")); + assert_eq!(arcgis_auth::authorization(&query).as_deref(), Some("Bearer RGSYNTH_3")); // Another server never gets it. - assert_eq!(arcgis_auth::authorization("https://other914.example.org/server/rest/services/Roads/MapServer/tile/0/0/0"), None); + assert_eq!(arcgis_auth::authorization("https://other914.invalid/server/rest/services/Roads/MapServer/tile/0/0/0"), None); // Added, the tile layer keeps the address without the token. let layer = v["layers"].as_array().unwrap().iter().find(|l| l["offer"]["type"] == "tiles").unwrap()["name"].as_str().unwrap().to_string(); - s.run("service.add", &json!({"url": format!("{base}?token=secret.Token-914"), "layer": layer, "text": doc})).unwrap(); + s.run("service.add", &json!({"url": format!("{base}?token=RGSYNTH_3"), "layer": layer, "text": doc})).unwrap(); let project = format!("{:?}", s.project); - assert!(project.contains("secure914.example.org") && !project.contains("secret.Token-914")); + assert!(project.contains("secure914.invalid") && !project.contains("RGSYNTH_3")); // Listed and forgotten by arcgis.status and arcgis.signOut. let status = s.run("arcgis.status", &json!({})).unwrap(); assert!( - status["sessions"] - .as_array() - .unwrap() - .iter() - .any(|x| x["portal"] == "https://secure914.example.org/server" && x["kind"] == "token") + status["sessions"].as_array().unwrap().iter().any(|x| x["portal"] == "https://secure914.invalid/server" && x["kind"] == "token") ); - assert!(!status.to_string().contains("secret.Token-914")); - s.run("arcgis.signOut", &json!({"portal": "https://secure914.example.org/server/"})).unwrap(); + assert!(!status.to_string().contains("RGSYNTH_3")); + s.run("arcgis.signOut", &json!({"portal": "https://secure914.invalid/server/"})).unwrap(); assert_eq!(arcgis_auth::authorization(&tiles), None); // Junk is refused rather than taken for "every session". for junk in [json!({"portal": 7}), json!({"portal": [true]}), json!({"portal": {"a": 1}})] { @@ -13213,6 +13212,12 @@ fn wms_layers_with_time() { let mut s = Session::new(Project::default()); let added = s.run("service.add", &json!({"url": url, "layer": "population", "text": WMS, "time": "slider"})).unwrap(); assert!(added["url"].as_str().is_some_and(|u| u.ends_with("&TIME=2025")), "{added}"); + let mut keyed = Session::new(Project::default()); + let capabilities = WMS.replace("https://maps.example.com/wms?", "https://maps.example.com/wms?token=RGSYNTH_service&"); + let public = keyed.run("service.add", &json!({"url":url,"layer":"population","text":capabilities,"time":"slider"})).unwrap(); + let live = &keyed.map().tiles[public["index"].as_u64().unwrap() as usize]; + assert!(live.url.contains("RGSYNTH_service") && live.url.ends_with("TIME=2025")); + assert!(!public["url"].as_str().unwrap().contains("RGSYNTH_service"), "moving to the default time cannot expose the live address"); let year = |y: i32| rustgis_data::time::parse_text(&format!("{y}-01-01")).unwrap(); let slider = s.map().time.clone().unwrap(); assert_eq!((slider.steps(), slider.current), (vec![year(1990), year(2000), year(2025)], year(2025))); @@ -13290,6 +13295,8 @@ fn wms_layers_with_time() { // A layer without times has none to follow it with. let e = s.run("service.add", &json!({"url": url, "layer": "roads", "text": WMS, "time": "slider"})).unwrap_err(); assert!(e.contains("no time to follow"), "{e}"); + let foreign = WMS.replace("https://maps.example.com/wms?", "https://fallback-b.invalid/wms?"); + service_logins::foreign_service_fallback_stays_anonymous("wms", url, "population", WMS, &foreign); } /// Identify asks a WMS layer what is at the point (GetFeatureInfo): the request from the @@ -15966,11 +15973,22 @@ fn a_plugins_dated_cog_is_added_without_waiting() { let base = format!("http://{}", server.local_addr().unwrap()); let mut s = Session::new(Project::default()); let start = std::time::Instant::now(); - let r = - s.run_leaving_downloads("layer.addUrl", &json!({"url": format!("{base}/{{date:YYYY}}.tif"), "start": 2020, "end": 2021})).unwrap(); + let r = s + .run_leaving_downloads( + "layer.addUrl", + &json!({"url": format!("{base}/{{date:YYYY}}.tif?token=RGSYNTH_dated_cog"), "start": 2020, "end": 2021}), + ) + .unwrap(); assert!(start.elapsed() < std::time::Duration::from_secs(10), "waited {:?}", start.elapsed()); assert_eq!(r["inPlace"], true, "{r}"); assert!(r["warning"].as_str().is_some_and(|w| w.contains("wasn't moved")), "{r}"); + let tile = &s.map().tiles[r["index"].as_u64().unwrap() as usize]; + assert!( + tile.url.contains(".tif?token=RGSYNTH_dated_cog") && !tile.url.contains("{date"), + "the live tile keeps the effective private address" + ); + let clean = ["warning", "url"].map(|key| r[key].as_str().is_some_and(|text| !text.contains("RGSYNTH_dated_cog"))); + assert_eq!(clean, [true, true], "both public warning and URL must omit credentials"); drop(server); } @@ -16481,6 +16499,10 @@ fn a_look_naming_bands_the_file_hasnt_is_said() { /// the queue dropped is asked for again, so the layer lands rather than failing with "dropped". #[test] fn an_overview_of_more_tiles_than_the_fetch_queue_holds_lands() { + // This fixture fills the shared native queue; other tests must keep their own requests. + if isolate_process_test("tests::an_overview_of_more_tiles_than_the_fetch_queue_holds_lands") { + return; + } use rustgis_data::raster::cog::{Example, example_with}; let (base, _) = serve_ranges(example_with(&Example { size: 1024, tile: 32, no_overviews: true, ..Example::default() })); let mut s = Session::new(Project::default()); @@ -16497,10 +16519,14 @@ fn an_overview_of_more_tiles_than_the_fetch_queue_holds_lands() { /// A window over more tiles than the fetch queue holds (small tiles) asks for them a batch at a /// time, the next as the map draws again, rather than all at once (which would push its own first /// requests out of the queue, to ask for them all again at the next draw), and is pending until -/// the last lands. (The tile cache is one for the process: other tests' reads may evict some of -/// these tiles meanwhile, so how many are fetched in all isn't checked.) +/// the last lands. The batch bound and final pixels are checked; the total fetch count may vary +/// because dropped requests can be tried again. #[test] fn a_window_of_more_tiles_than_the_fetch_queue_holds_asks_for_them_in_batches() { + // This fixture fills the shared native queue; other tests must keep their own requests. + if isolate_process_test("tests::a_window_of_more_tiles_than_the_fetch_queue_holds_asks_for_them_in_batches") { + return; + } use crate::render::rustgis_tiles::{cache, cog}; use rustgis_data::raster::ColorHint; use rustgis_data::raster::cog::{Example, example_with}; @@ -19126,14 +19152,20 @@ fn layer_address_redaction_preserves_templates_and_local_paths() { ("upload:map?token=local", "upload:map?token=local"), ("postgres:host=localhost dbname=gis", "postgres:host=localhost dbname=gis"), ]; - let mut addresses: Vec = cases.iter().map(|(a, _)| a.to_string()).collect(); - crate::project::strip_addresses(addresses.iter_mut().collect()); - for (address, (_, expected)) in addresses.iter().zip(cases) { + for (original, expected) in cases { + let address = match crate::durable::public_address(original) { + crate::durable::Kept::Unchanged => original.to_owned(), + crate::durable::Kept::Changed(address) => address, + crate::durable::Kept::Withheld(_) => panic!("valid address was withheld: {original}"), + }; assert_eq!(address, expected); } let prefix = crate::render::rustgis_tiles::s3::ANONYMOUS; - let mut anonymous = format!("{prefix}https://bucket.example/a?token=secret"); - crate::project::strip_addresses(vec![&mut anonymous]); + let crate::durable::Kept::Changed(anonymous) = + crate::durable::public_address(&format!("{prefix}https://bucket.example/a?token=secret")) + else { + panic!("anonymous address was not redacted") + }; assert_eq!(anonymous, format!("{prefix}https://bucket.example/a")); } @@ -19309,41 +19341,75 @@ fn recovery_copies_keep_no_layer_tokens_and_restore_none() { #[test] fn saved_projects_keep_no_layer_tokens() { let mut s = world(); - let url = "https://gis.example/arcgis/rest/services/a/MapServer/tile/{z}/{y}/{x}?token=s3cret"; + let url = "https://gis.example/arcgis/rest/services/a/MapServer/tile/{z}/{y}/{x}?token=s3cret-tile"; // Its legend, built from the service's address, with the key in it too. - let legend = "https://gis.example/wms?key=l3g&SERVICE=WMS&REQUEST=GetLegendGraphic&LAYER=a"; + let legend = "https://gis.example/wms?key=l3g-secret&SERVICE=WMS&REQUEST=GetLegendGraphic&LAYER=a"; s.run("map.addTiles", &json!({"url":url,"name":"Secured","legend":legend})).unwrap(); let layer = s.map().layers[0].id; - s.map_mut().layers[0].dataset.source = Some("https://u:pw@gis.example/f?f=json&access_token=tok2".into()); + s.map_mut().layers[0].dataset.source = Some("https://u:pw-secret@gis.example/f?f=json&access_token=tok2-secret".into()); s.map_mut().service_groups.push(rustgis_map::ServiceGroup { id: 123, name: "Protected service".into(), - url: "https://gis.example/wfs?key=k3y&map=m".into(), + url: "https://gis.example/wfs?key=k3y-secret&map=m".into(), kind: "wfs".into(), anonymous: false, expanded: true, children: vec![rustgis_map::ServiceChild { layer, collection: "roads".into(), - source_url: "https://gis.example/roads?sig=x&sv=1".into(), + source_url: "https://gis.example/roads?sig=x-secret&sv=1".into(), }], }); let path = std::env::temp_dir().join(format!("rustgis-token-audit-{}.rgis", std::process::id())); let result = s.run("project.saveAs", &json!({"path":path})).unwrap(); let text = std::fs::read_to_string(&path).unwrap(); - for secret in ["s3cret", "pw@", "tok2", "k3y", "sig=x", "l3g"] { + for secret in ["s3cret-tile", "pw-secret@", "tok2-secret", "k3y-secret", "sig=x-secret", "l3g-secret"] { assert!(!text.contains(secret), "{secret} leaked into the file"); } assert!(text.contains("MapServer/tile/{z}/{y}/{x}") && text.contains("map=m") && text.contains("REQUEST=GetLegendGraphic")); assert!(result["stripped"].as_u64().unwrap() >= 4); assert!(s.notices.iter().any(|n| n.text.contains("saved without their keys"))); assert!(s.map().tiles.iter().any(|t| t.url == url)); - for kind in ["map", "projectPackage", "mapPackage", "mobilePackage"] { - let (_, bytes) = crate::cmd::items::item_bytes(&mut s, kind, &Value::Null).unwrap(); - let packaged = - if kind == "map" { String::from_utf8(bytes.clone()).unwrap() } else { crate::cmd::items::package_text(&bytes).unwrap() }; - for secret in ["s3cret", "pw@", "tok2", "k3y", "sig=x", "l3g"] { - assert!(!packaged.contains(secret), "{kind} leaked {secret}"); + // Every kind of item file, each with something in the project to write. + s.run("style.new", &json!({"layer": "Cities", "name": "Cities"})).unwrap(); + s.run("gp.run", &json!({"tool": "buffer", "params": {"input": "Cities", "distance": 100}})).unwrap(); + s.run("task.new", &Value::Null).unwrap(); + s.run("model.new", &json!({"fromHistory": true})).unwrap(); + s.run("report.new", &json!({"layer": "Cities"})).unwrap(); + s.run("presentation.new", &Value::Null).unwrap(); + s.run("notebook.new", &Value::Null).unwrap(); + // Kinds that hold no layer addresses, or need more than the sample gives. + let exempt = [ + ("ramps", "colour ramps hold no addresses"), + ("dlPackage", "a trained model holds no addresses"), + ("hostedImagery", "tiles rendered from a raster layer: pictures, not addresses"), + ("linkChart", "needs relationships the sample lacks"), + ]; + let mut report = Vec::new(); + for &(kind, _, _) in crate::cmd::items::ITEMS { + if exempt.iter().any(|(k, _)| *k == kind) { + continue; + } + let bytes = match crate::cmd::items::item_bytes(&mut s, kind, &Value::Null) { + Ok((_, bytes)) => bytes, + Err(e) => { + report.push(format!("{kind}: {e}")); + continue; + } + }; + let packaged = if bytes.starts_with(b"PK") { + rustgis_data::zip::read_matching(&bytes, |_| true) + .unwrap() + .into_iter() + .map(|(_, b)| String::from_utf8_lossy(&b).into_owned()) + .collect::() + } else { + String::from_utf8_lossy(&bytes).into_owned() + }; + for secret in ["s3cret-tile", "pw-secret@", "tok2-secret", "k3y-secret", "sig=x-secret", "l3g-secret"] { + if packaged.contains(secret) { + report.push(format!("{kind}: leaked {secret}")); + } } if kind == "projectPackage" { // Even a hostile archive carrying the exact registered JSON must not restore it. @@ -19352,21 +19418,22 @@ fn saved_projects_keep_no_layer_tokens() { std::fs::write(&package_path, hostile).unwrap(); let mut opened = Session::new(Project::default()); opened.run("project.open", &json!({"path":package_path})).unwrap(); - assert!(!serde_json::to_string(&opened.project).unwrap().contains("s3cret")); + assert!(!serde_json::to_string(&opened.project).unwrap().contains("s3cret-tile")); opened.run("project.openText", &json!({"text":text,"package":true})).unwrap(); - assert!(!serde_json::to_string(&opened.project).unwrap().contains("s3cret")); + assert!(!serde_json::to_string(&opened.project).unwrap().contains("s3cret-tile")); std::fs::remove_file(package_path).unwrap(); } } + assert!(report.is_empty(), "{report:#?}"); let mut t = Session::new(Project::default()); t.run("project.open", &json!({"path":path})).unwrap(); assert!(t.map().tiles.iter().any(|t| t.url == url)); - assert!(t.map().layers[0].dataset.source.as_deref().unwrap().contains("tok2")); + assert!(t.map().layers[0].dataset.source.as_deref().unwrap().contains("tok2-secret")); let modified = text.replace("\"name\":\"Secured\"", "\"name\":\"Secures\""); assert_ne!(text, modified); t.run("project.openText", &json!({"text":modified})).unwrap(); let live = serde_json::to_string(&t.project).unwrap(); - for secret in ["s3cret", "pw@", "tok2", "k3y", "sig=x"] { + for secret in ["s3cret-tile", "pw-secret@", "tok2-secret", "k3y-secret", "sig=x-secret"] { assert!(!live.contains(secret), "{secret} restored into a modified hostile file"); } let public = url.split('?').next().unwrap(); @@ -19374,7 +19441,7 @@ fn saved_projects_keep_no_layer_tokens() { hand.run("map.addTiles", &json!({"url":public,"name":"Hand made"})).unwrap(); t.run("project.openText", &json!({"text":serde_json::to_string(&hand.project).unwrap()})).unwrap(); assert!(t.map().tiles.iter().any(|t| t.url == public)); - assert!(!serde_json::to_string(&t.project).unwrap().contains("s3cret")); + assert!(!serde_json::to_string(&t.project).unwrap().contains("s3cret-tile")); std::fs::remove_file(path).unwrap(); } @@ -22043,6 +22110,11 @@ fn a_run_is_copied_from_the_history_as_code() { assert_eq!((tool_of(&runs[0]).as_deref(), tool_of(&runs[1]), tool_of(&runs[2]).as_deref()), (Some("buffer"), None, Some("buffer"))); let py = code(&mut s, 2, "python").unwrap(); assert!(py.contains("tool=\"buffer\"") && py.contains("https://h.example/a") && !py.contains("s3cr3t"), "{py}"); + // Replay keeps opaque text and unknown nested content, with an honest warning. The log's + // separate, stricter projection still exercises the legacy connection/text cases below. + assert!(py.contains("odbcpw") && py.contains("libpqpw") && py.contains("shortk"), "{py}"); + assert!(s.notices.iter().any(|n| n.text.contains("opaque parameters"))); + let py = serde_json::to_string_pretty(&crate::cmd::history::redacted(&old.params)).unwrap(); // A libpq connection keeps its words but its password, though spaces part them. assert!(py.contains("host=db") && !py.contains("hunter2"), "{py}"); // A connection with no prefix `redact` knows keeps its words but its secret-named values. @@ -22143,6 +22215,9 @@ fn secrets_in_a_query_or_a_run_s_error_stay_out_of_copied_code() { outputs: Vec::new(), }); let py = s.run("gp.historyCode", &json!({"run": 0})).unwrap()["code"].as_str().unwrap().to_string(); + assert!(py.contains("password=qp"), "opaque query text stays replayable: {py}"); + assert!(s.notices.iter().any(|n| n.text.contains("opaque parameters"))); + let py = serde_json::to_string_pretty(&crate::cmd::history::redacted(&s.project.gp_history[0].params)).unwrap(); assert!(py.contains("a=1&password=&b=2") && !py.contains("qp"), "{py}"); // An `&` in a value, or in quotes, isn't another pair: the whole value goes. So does an ODBC // braced value with an escaped `}}` in it. @@ -25097,6 +25172,10 @@ fn time_slider_fills_dated_tile_addresses_and_exports_its_steps() { ) .unwrap(); assert_eq!(r["url"], "https://example.com/2023-08-01/{z}/{x}/{y}.png"); + let mut keyed_session = timed_events(); + let keyed = keyed_session.run("map.addTiles", &json!({"url":"https://example.com/{date:YYYY-MM-DD}/{z}/{x}/{y}.png?token=RGSYNTH_dated_tiles","start":"2023-08-01","end":"2023-08-31"})).unwrap(); + assert!(!keyed["url"].as_str().unwrap().contains("RGSYNTH_dated_tiles")); + assert!(keyed_session.map().tiles[keyed["index"].as_u64().unwrap() as usize].url.contains("RGSYNTH_dated_tiles")); let i = s.run("time.info", &Value::Null).unwrap(); assert_eq!((i["unit"].as_str(), i["steps"].as_u64()), (Some("day"), Some(31)), "{i}"); // Only tile kinds that fill dates take a dated address. @@ -26241,6 +26320,41 @@ fn geopackage_trigger_rebuild_rejects_a_valid_envelope_with_unreadable_wkb() { assert_eq!(s.run("gpkg.pendingEdits", &json!({"layer":"R"})).unwrap()["updates"], 1); } +/// A template keeps no copies of the data: offline snapshots, archived edits and a streamed +/// layer's file would each bring features back when it opens. +#[test] +fn project_template_carries_no_data() { + let path = grid_fgb("template-streamed", 20); + let mut s = with_streamed(&path); + s.run("offline.download", &json!({"name": "Paris", "extent": [1.0, 48.0, 4.0, 50.0]})).unwrap(); + s.run("data.archive", &json!({"layer": "Cities"})).unwrap(); + s.run("edit.setValue", &json!({"layer": "Cities", "row": 0, "field": "name", "value": "Renamed"})).unwrap(); + s.run("graphics.add", &json!({"shape": "rectangle", "points": [[0, 45], [10, 55]]})).unwrap(); + let mut table = rustgis_data::Dataset::new( + "Visits", + rustgis_data::GeometryType::None, + vec![rustgis_data::Field::new("n", rustgis_data::FieldType::Integer)], + ); + table.push(None, vec![rustgis_data::Value::Int(1)]); + table.source = Some(path.display().to_string()); + s.project.add_table(table); + assert!(!s.project.offline_areas.is_empty()); + let (_, bytes) = crate::cmd::items::item_bytes(&mut s, "template", &Value::Null).unwrap(); + let text = String::from_utf8(bytes).unwrap(); + assert!(!text.contains(path.to_str().unwrap()), "the streamed layer's file is in the template"); + let t: Project = serde_json::from_str(&text).unwrap(); + assert!(t.offline_areas.is_empty()); + assert!(t.tables.iter().all(|t| t.dataset.features.is_empty() && t.dataset.source.is_none())); + assert!(t.maps.iter().flat_map(|m| &m.layers).all(|l| l.dataset.features.is_empty()), "graphics and features stay out"); + let cities = t.maps[0].layer_by_name("Cities").unwrap(); + assert_eq!(cities.design.archive.as_ref().map(Vec::len), Some(0), "archiving stays on, without its edits"); + let streamed = t.maps[0].layer_by_name("Streamed").unwrap(); + assert!(streamed.source.is_none() && streamed.dataset.features.is_empty()); + let mut opened = Session::new(Project::default()); + opened.run("project.fromTemplate", &json!({"text": text, "name": "Setup"})).unwrap(); + assert!(opened.project.maps.iter().flat_map(|m| &m.layers).all(|l| l.dataset.features.is_empty())); +} + #[cfg(not(target_arch = "wasm32"))] #[test] fn geopackage_template_drops_edit_binding() { @@ -30380,6 +30494,28 @@ fn a_scripts_tools_run_ahead_are_taken_by_its_run() { } } +/// Runs a test that owns process-wide state in a fresh instance of the test executable. +pub(crate) fn isolate_process_test(name: &str) -> bool { + if std::env::var("RUSTGIS_ISOLATED_SCENARIO").as_deref() == Ok(name) { + return false; + } + let output = std::process::Command::new(std::env::current_exe().unwrap()) + .args(["--exact", name]) + .env("RUSTGIS_ISOLATED_SCENARIO", name) + .output() + .unwrap(); + let stdout = String::from_utf8_lossy(&output.stdout); + let mut summaries = stdout.lines().filter(|line| line.starts_with("test result:")); + let ran_one = summaries.next().is_some_and(|line| line.starts_with("test result: ok. 1 passed; 0 failed; 0 ignored;")) + && summaries.next().is_none(); + assert!( + output.status.success() && ran_one, + "isolated process scenario failed or ran no test:\n{stdout}\n{}", + String::from_utf8_lossy(&output.stderr) + ); + true +} + #[test] fn topology_validate_says_when_a_polar_check_is_incomplete() { // A line winding round the North Pole about 19,000 times: its dangle check measures it coarsely, diff --git a/crates/engine/src/tests/credential_output.rs b/crates/engine/src/tests/credential_output.rs new file mode 100644 index 000000000..da66e7789 --- /dev/null +++ b/crates/engine/src/tests/credential_output.rs @@ -0,0 +1,972 @@ +use super::*; +use crate::cmd::notebook::{Cell, CellKind, Notebook}; +use crate::{ + GpRecord, Peer, durable, + project::{Model, ModelGraph, ModelNode, ModelStep, Task, TaskStep}, +}; +use rustgis_map::presentation::{Page, PageKind, Presentation}; + +const SECRET: &str = "RGSYNTH_output_20261103"; +const DATA: &str = "RGSYNTH_data_20261103"; + +/// Database labels and durable addresses share query-key classification, without changing +/// the executable connection string or dropping clean connection options. +#[test] +fn durable_database_query_aliases_match_public_labels() { + for scheme in ["mysql", "mariadb", "mssql", "postgresql"] { + for key in ["access_token", "subscription-key", "api_key"] { + let public = format!("{scheme}://reader@db.invalid/gis?ssl=true"); + let private = format!("{public}&{key}={SECRET}"); + let kept = match durable::public_address(&private) { + durable::Kept::Changed(s) => s, + _ => private.clone(), + }; + assert_eq!(kept, public, "durable {scheme}/{key}"); + assert_eq!(crate::sources::redact(&private), public, "public label {scheme}/{key}"); + assert!(matches!(durable::public_address(&public), durable::Kept::Unchanged)); + assert_eq!(crate::sources::redact(&public), public); + } + } +} + +/// Graph input fields select existing names; source-address cleaning must not rename those +/// references when saved text is opened without this process's credential restore memory. +#[test] +fn durable_model_input_names_survive_a_cold_round_trip() { + let bound = rustgis_data::login_address::bind("0123456789abcdef", "https://example.invalid", "https://example.invalid/roads").unwrap(); + for name in ["https://example.invalid/roads#north", bound.as_str()] { + let mut s = world(); + s.map_mut().layers[0].name = name.into(); + let graph = ModelGraph { + nodes: vec![ModelNode { + id: "input1".into(), + kind: crate::project::ModelNodeKind::Input, + layer: Some(name.into()), + ..Default::default() + }], + edges: Vec::new(), + }; + s.project.models.push(Model { name: "Names".into(), steps: Vec::new(), graph: Some(graph) }); + let (text, _) = s.project.saved_text(None).unwrap(); + for cold in [Project::from_json(&text).unwrap(), Project::from_import_json(&text, None).unwrap()] { + let model = cold.models.last().unwrap().graph.as_ref().unwrap(); + let selector = model.nodes[0].layer.as_deref().unwrap(); + assert_eq!(selector, name, "a graph input selects a name, even when it resembles an address or runtime wrapper"); + assert!(cold.map().layer_by_name(selector).is_some(), "the opened graph still resolves its real input"); + } + } +} + +/// Address parsing must precede active-pattern masking at every generated history/error +/// boundary, even when the active key is part of the URL scheme itself. +#[cfg(not(target_arch = "wasm32"))] +#[test] +fn generated_errors_and_history_clean_addresses_before_active_patterns() { + if isolate_process_test("tests::credential_output::generated_errors_and_history_clean_addresses_before_active_patterns") { + return; + } + use crate::render::rustgis_tiles::logins; + let binding = logins::activate( + "https://history-collision.invalid", + "synthetic-history-collision", + logins::Method::Header("X-Key".into()), + "https", + false, + ) + .unwrap(); + let raw = format!("failed at https://h.invalid/a?token={SECRET}&layer=roads"); + let mut s = world(); + s.push_gp_record(GpRecord { + tool: "Unknown".into(), + id: "unknown".into(), + params: Value::Null, + command: None, + at: 0, + ok: false, + messages: vec![raw.clone()], + millis: 0, + outputs: Vec::new(), + }); + s.notify(&raw, true); + let error = s.run("layer.rename", &json!({"layer":format!("https://h.invalid/a?token={SECRET}"),"name":"unused"})).unwrap_err(); + let history = s.project.gp_history.last().unwrap().messages[0].clone(); + let notice = s.notices.last().unwrap().text.clone(); + assert!(logins::deactivate(&binding)); + assert!(!history.contains(SECRET), "history: {history}"); + assert!(!notice.contains(SECRET), "notice: {notice}"); + assert!(!error.contains(SECRET), "command error: {error}"); + for id in [ + format!("https://h.invalid/a?token={SECRET}"), + format!("sql.connect={{\"password\":\"{SECRET}\"}}"), + format!("sql.connect {{\"password\":\"{SECRET}\"}}"), + ] { + let error = s.run(&id, &Value::Null).unwrap_err(); + assert!(!error.contains(SECRET), "unknown command error: {error}"); + } + assert!(!s.project.saved_text(None).unwrap().0.contains(SECRET)); +} + +/// Invalid command/tool identifiers are executable slots too: an address there must not +/// bypass cleaning through a null parameter or a malformed script line. +#[test] +fn durable_identifier_addresses_are_cleaned_and_counted() { + let url = format!("https://identifier.invalid/a?token={SECRET}&layer=roads"); + let source = format!( + "# clean comment\r\n{url}\r\n{url} {{broken\nsql.connect={{\"password\":\"{SECRET}\"}}\nsql.connect={{\"password\":\"{SECRET}\"}} junk\nmap.zoomIn {{}}\n" + ); + let mut project = Project::default(); + project.notebooks.push(Notebook { + name: "Identifiers".into(), + jupyter: None, + cells: vec![Cell { kind: CellKind::Code, source, output: None, error: false, jupyter: None }], + }); + project.tasks.push(Task { + name: "Identifiers".into(), + description: String::new(), + steps: [ + url.clone(), + format!("sql.connect {{\"password\":\"{SECRET}\"}}"), + format!("sql.connect:{{\"password\":\"{SECRET}\"}}"), + format!("sql.connect:password:{SECRET}"), + format!("sql.connect:token:{SECRET}"), + ] + .into_iter() + .map(|command| TaskStep { title: String::new(), instructions: String::new(), command, params: Value::Null }) + .collect(), + }); + project.models.push(Model { + name: "Identifiers".into(), + steps: vec![ModelStep { tool: url.clone(), params: Value::Null }], + graph: Some(ModelGraph { + nodes: vec![ModelNode { id: "tool1".into(), tool: Some(url.clone()), ..Default::default() }], + edges: Vec::new(), + }), + }); + let mut s = Session::new(project); + s.project.gp_history.push(GpRecord { + tool: "Unknown".into(), + id: url.clone(), + params: Value::Null, + command: Some((url, Value::Null)), + at: 0, + ok: false, + messages: Vec::new(), + millis: 0, + outputs: Vec::new(), + }); + let original = serde_json::to_value(&s.project).unwrap(); + let (text, report) = s.project.saved_text(None).unwrap(); + assert!(!text.contains(SECRET), "an identifier bypassed durable projection: {text}"); + assert!(report.lines >= 2, "{report:?}"); + assert_eq!(report.steps, 9, "each changed executable is counted once"); + assert_eq!(serde_json::to_value(&s.project).unwrap(), original, "live identifiers are restored exactly"); + let (clean, report) = crate::roles::script( + "# clean comment\r\nmap.zoomIn {}\r\nmap.zoomOut\nplugin:token-tools/read\nplugin:demo/token\ngeolibre:token\n", + 0, + ); + assert!(clean.is_none() && report.notices().is_empty(), "ordinary ids and formatting stay exact"); +} + +/// Cleaning both fields of one operation still affects one executable step. +#[test] +fn durable_executable_count_is_per_step() { + let mut project = Project::default(); + project.tasks.push(Task { + name: "One step".into(), + description: String::new(), + steps: vec![TaskStep { + title: String::new(), + instructions: String::new(), + command: format!("https://identifier.invalid/?token={SECRET}"), + params: json!({"password":SECRET}), + }], + }); + let (text, report) = project.saved_text(None).unwrap(); + assert!(!text.contains(SECRET)); + assert_eq!(report.steps, 1, "identifier and params belong to the same step: {report:?}"); +} + +/// Inline synthetic state exercises owning fields without contacting any service. +fn marked() -> Session { + let mut s = world(); + let url = format!("https://h.invalid/a?token={SECRET}&layer=a%20b"); + s.project.maps[0].layers[0].dataset.source = Some(url.clone()); + s.project.tasks.push(Task { name: "Credentials".into(), description: DATA.into(), steps: vec![ + TaskStep { title: DATA.into(), instructions: DATA.into(), command: "layer.addUrl".into(), params: json!({"url": url}) }, + TaskStep { title: "Malformed wrapper".into(), instructions: String::new(), command: "layer.addUrl".into(), params: json!({"url": format!("rustgis-s3:broken:{SECRET}")}) }, + TaskStep { title: "Data".into(), instructions: String::new(), command: "layer.addDataText".into(), params: json!({"text": format!(r#"{{"type":"Feature","geometry":null,"properties":{{"token":"{DATA}","key":"{DATA}","password":"{DATA}","url":"https://h.invalid/?token={DATA}"}}}}"#)}) }, + ] }); + let params = json!({"input": "Cities", "token": SECRET, "values": {"token": DATA, "key": DATA, "password": DATA}}); + s.project.models.push(Model { + name: "Credentials".into(), + steps: vec![ModelStep { tool: "buffer".into(), params: params.clone() }], + graph: Some(ModelGraph { + nodes: vec![ModelNode { id: "tool1".into(), tool: Some("buffer".into()), params: params.clone(), ..Default::default() }], + edges: Vec::new(), + }), + }); + s.project.gp_history.push(GpRecord { + tool: "Buffer".into(), + id: "buffer".into(), + params, + command: Some(("gp.bufferPoint".into(), json!({"token": SECRET}))), + at: 0, + ok: true, + messages: vec![format!("source {url}")], + millis: 0, + outputs: vec!["Cities".into()], + }); + s.project.notebooks.push(Notebook { + name: "Credentials".into(), + jupyter: None, + cells: vec![Cell { + kind: CellKind::Code, + source: format!("# {DATA}\nlayer.addUrl {{\"url\":\"{url}\"}}\nlayer.addUrl {{\"url\":\"{url}\"\n"), + output: Some(format!("source {url}")), + error: true, + jupyter: None, + }], + }); + s.project.presentations.push(Presentation { + name: "Credentials".into(), + pages: vec![Page { + title: DATA.into(), + kind: PageKind::Video { source: url, poster: None, caption: DATA.into() }, + notes: DATA.into(), + seconds: 5.0, + }], + ..Default::default() + }); + s +} + +#[test] +fn durable_writers_clean_all_owned_roles_and_preserve_the_live_project() { + let mut s = marked(); + let before = serde_json::to_value(&s.project).unwrap(); + let (text, report) = s.project.saved_text(None).unwrap(); + assert!(!text.contains(SECRET)); + assert!(report.layer_addresses() > 0 && report.steps > 0 && report.lines > 0 && report.videos > 0 && report.withheld > 0); + assert!(text.contains(DATA)); + assert_eq!(before, serde_json::to_value(&s.project).unwrap()); + let opened = Project::from_json(&text).unwrap(); + assert_eq!(opened.tasks[0].steps, s.project.tasks[0].steps); + assert_eq!(opened.models[0], s.project.models[0]); + assert_eq!(opened.notebooks[0].cells[0].source, s.project.notebooks[0].cells[0].source); + assert!(!opened.notebooks[0].cells[0].output.as_ref().unwrap().contains(SECRET), "generated text is never restored"); + assert!(!s.project.clone().recovery_text().unwrap().contains(SECRET)); + for kind in [ + "map", + "layer", + "template", + "projectPackage", + "mapPackage", + "mobilePackage", + "gpPackage", + "task", + "notebook", + "jupyter", + "presentation", + "webTool", + ] { + let (_, bytes) = crate::cmd::items::item_bytes(&mut s, kind, &Value::Null).unwrap(); + let text = if bytes.starts_with(b"PK") { + rustgis_data::zip::read_matching(&bytes, |_| true) + .unwrap() + .into_iter() + .map(|(_, b)| String::from_utf8_lossy(&b).into_owned()) + .collect::() + } else { + String::from_utf8_lossy(&bytes).into_owned() + }; + assert!(!text.contains(SECRET), "{kind}"); + assert_eq!(before, serde_json::to_value(&s.project).unwrap(), "{kind} mutated live fields"); + } + let code = s.run("gp.historyCode", &json!({"run": 0, "as": "command"})).unwrap(); + assert!(!code.to_string().contains(SECRET)); + s.project.gp_history[0].command = None; + let code = s.run("gp.historyScript", &Value::Null).unwrap(); + assert!(!code.to_string().contains(SECRET)); + assert!(code.to_string().contains(DATA), "replayable data properties must remain"); +} + +/// Empty query pairs and clean padding survive saves; an actual token gives one counted loss. +#[test] +fn durable_empty_query_pairs_preserve_clean_urls_and_report_real_loss() { + for (query, kept, losses) in [ + ("", "", 0), + ("&", "&", 0), + ("layer=1&&f=json", "layer=1&&f=json", 0), + ("layer=1&", "layer=1&", 0), + ("layer=1 \t", "layer=1 \t", 0), + ("layer=1&&token=RGSYNTH_empty_query&", "layer=1", 1), + ("layer=1&appid=RGSYNTH_empty_query", "layer=1", 1), + ("layer=1&%61ppid=RGSYNTH_empty_query", "layer=1", 1), + ("layer=1&subscription-key=RGSYNTH_empty_query", "layer=1", 1), + ("layer=1&client_key=RGSYNTH_empty_query", "layer=1", 1), + ("layer=1&sas-key=RGSYNTH_empty_query", "layer=1", 1), + ("client_id=public&application_id=public&map=1", "client_id=public&application_id=public&map=1", 0), + ] { + let url = format!("https://h.invalid/tiles/{{z}}/{{x}}/{{y}}.png?{query}"); + let mut project = Project::default(); + project.map_mut().tiles.push(rustgis_map::TileLayer::new("Clean", rustgis_map::TileKind::Raster, &url)); + let before = serde_json::to_value(&project).unwrap(); + let (saved, report) = project.saved_text(None).unwrap(); + assert_eq!(report.layer_addresses(), losses, "{url}"); + assert_eq!(report.notices().len(), usize::from(losses > 0), "{url}: {:?}", report.notices()); + let saved: Value = serde_json::from_str(&saved).unwrap(); + let expected = format!("https://h.invalid/tiles/{{z}}/{{x}}/{{y}}.png?{kept}"); + assert_eq!(saved["maps"][0]["tiles"][0]["url"], expected); + assert_eq!(serde_json::to_value(&project).unwrap(), before); + } +} + +/// Fragments are deliberately omitted, but their notice must not call them credentials. +#[test] +fn durable_fragment_removal_is_reported_accurately_and_clean_addresses_stay_quiet() { + for fragment in ["", "#section", "#token=RGSYNTH_fragment"] { + let url = format!("https://h.invalid/tiles/{{z}}/{{x}}/{{y}}.png{fragment}"); + let mut project = Project::default(); + project.map_mut().tiles.push(rustgis_map::TileLayer::new("Fragment", rustgis_map::TileKind::Raster, &url)); + let (saved, report) = project.saved_text(None).unwrap(); + let saved: Value = serde_json::from_str(&saved).unwrap(); + assert_eq!(saved["maps"][0]["tiles"][0]["url"], "https://h.invalid/tiles/{z}/{x}/{y}.png"); + assert_eq!(project.map().tiles[0].url, url); + assert_eq!(report.layer_addresses(), u32::from(!fragment.is_empty())); + if fragment.is_empty() { + assert!(report.notices().is_empty()); + } else { + let notices = report.notices(); + assert_eq!(notices.len(), 1); + assert!(notices[0].contains("URL fragments"), "{notices:?}"); + } + } +} + +#[test] +fn durable_clean_and_opaque_content_keep_their_meaning() { + let mut s = marked(); + let original = s.project.tasks[0].steps[2].params.clone(); + let (clean, report) = crate::roles::command("layer.addDataText", &original, 0); + assert!(clean.is_none() && report.notices().is_empty()); + let source = format!("print('https://h.invalid/?token={SECRET}')"); + let notebook = Notebook { + name: "Python".into(), + jupyter: Some(std::sync::Arc::new(json!({"metadata":{"kernelspec":{"language":"python"}}}))), + cells: vec![Cell { kind: CellKind::Code, source: source.clone(), output: None, error: false, jupyter: None }], + }; + let (text, report) = durable::output_owned(notebook, |n| serde_json::to_string(n).map_err(|e| e.to_string())).unwrap(); + assert!(text.contains(SECRET) && report.notices().is_empty()); + let params = json!({"nested": {"token": SECRET}}); + let (clean, report) = crate::roles::command("plugin.unknown", ¶ms, 0); + assert!(clean.is_none() && report.unchecked == 1); + for command in ["layer.addUrl", "plugin.unknown"] { + for (token, unchecked) in [ + (json!({"value": SECRET}), true), + (json!([SECRET]), true), + (json!({}), false), + (json!([]), false), + (json!(false), false), + (Value::Null, false), + ] { + let params = json!({"token": token}); + let mut project = Project::default(); + project.tasks.push(Task { + name: "Opaque credential value".into(), + description: String::new(), + steps: vec![TaskStep { + title: String::new(), + instructions: String::new(), + command: command.into(), + params: params.clone(), + }], + }); + let before = serde_json::to_value(&project).unwrap(); + let (text, report) = project.saved_text(None).unwrap(); + let written: Value = serde_json::from_str(&text).unwrap(); + assert_eq!(written["tasks"][0]["steps"][0]["params"], params, "{command}: opaque values keep their meaning"); + assert_eq!(serde_json::to_value(&project).unwrap(), before, "{command}: output preserves the live project"); + assert_eq!(report.unchecked, u32::from(unchecked), "{command}: {params}"); + assert_eq!(report.steps, 0); + assert_eq!(report.withheld, 0); + let expected = if unchecked { + vec!["1 step(s) contain opaque parameters that may hold credentials and were kept unchecked".to_string()] + } else { + Vec::new() + }; + assert_eq!(report.notices(), expected, "{command}: {params}"); + if unchecked { + assert_eq!(crate::log_params(¶ms).into_log_value(), json!({"token": ""}), "logs withhold the whole value"); + } + } + } + let clean = Project::default(); + let expected = serde_json::to_string(&clean).unwrap(); + let (text, report) = durable::output_owned(clean, |p| serde_json::to_string(p).map_err(|e| e.to_string())).unwrap(); + assert_eq!(text, expected); + assert!(report.notices().is_empty()); + s.project.tasks[0].steps[0].params = json!({"url": "rustgis-s3:broken:RGSYNTH_output_20261103"}); + let (text, report) = s.project.saved_text(None).unwrap(); + assert!(!text.contains(SECRET) && report.withheld > 0); +} + +/// Address lists clean string leaves, while opaque values keep their data and say so. +#[test] +fn durable_address_roles_report_opaque_shapes_without_rewriting_data() { + let public = "https://h.invalid/a?layer=1"; + let secret = format!("{public}&token={SECRET}"); + let opaque = json!({"url":{"href":secret}}); + let nested = json!({"urls":[[secret]]}); + let clean_object = json!({"url":{"href":public}}); + for (params, expected, steps, unchecked) in [ + (json!({"url":public}), json!({"url":public}), 0, 0), + (json!({"urls":[secret]}), json!({"urls":[public]}), 1, 0), + (clean_object.clone(), clean_object, 0, 0), + (opaque.clone(), opaque, 0, 1), + (nested.clone(), nested, 0, 1), + ] { + let mut project = Project::default(); + project.tasks.push(Task { + name: "Address shapes".into(), + description: String::new(), + steps: vec![TaskStep { + title: String::new(), + instructions: String::new(), + command: "layer.addUrl".into(), + params: params.clone(), + }], + }); + let before = serde_json::to_value(&project).unwrap(); + let (text, report) = project.saved_text(None).unwrap(); + let written: Value = serde_json::from_str(&text).unwrap(); + assert_eq!(written["tasks"][0]["steps"][0]["params"], expected); + assert_eq!(serde_json::to_value(&project).unwrap(), before); + assert_eq!((report.steps, report.unchecked, report.withheld), (steps, unchecked, 0), "{params}"); + assert_eq!(report.notices().len(), usize::from(steps > 0) + usize::from(unchecked > 0)); + if unchecked > 0 { + assert_eq!(report.notices(), ["1 step(s) contain opaque parameters that may hold credentials and were kept unchecked"]); + } + assert!(!crate::log_params(¶ms).into_log_value().to_string().contains(SECRET), "logs have a separate stricter policy"); + } +} + +/// A newer embedded project must not be rewritten with this writer's schema or version. +#[test] +fn durable_future_documents_keep_their_exact_text_unchecked() { + let mut future = Project::default(); + future.map_mut().tiles.push(rustgis_map::TileLayer::new( + "Future", + rustgis_map::TileKind::Raster, + format!("https://h.invalid/{{z}}/{{x}}/{{y}}.png?token={SECRET}"), + )); + let mut document = serde_json::to_value(&future).unwrap(); + let format = crate::project::FORMAT + 1; + document["format"] = json!(format); + document["futureRequired"] = json!({"meaning":"preserve this unknown field"}); + let source = serde_json::to_string_pretty(&document).unwrap(); + for command in ["project.openText", "map.import"] { + let mut project = Project::default(); + project.tasks.push(Task { + name: "Future document".into(), + description: String::new(), + steps: vec![TaskStep { + title: String::new(), + instructions: String::new(), + command: command.into(), + params: json!({"text":source}), + }], + }); + let before = serde_json::to_value(&project).unwrap(); + let (text, report) = project.saved_text(None).unwrap(); + let written: Value = serde_json::from_str(&text).unwrap(); + assert_eq!(written["tasks"][0]["steps"][0]["params"]["text"], source, "{command}: future document bytes stay exact"); + let embedded: Value = serde_json::from_str(written["tasks"][0]["steps"][0]["params"]["text"].as_str().unwrap()).unwrap(); + assert_eq!(embedded["format"], format); + assert_eq!(embedded["futureRequired"], document["futureRequired"]); + assert_eq!(serde_json::to_value(&project).unwrap(), before); + assert_eq!((report.steps, report.unchecked, report.withheld), (0, 1, 0)); + assert_eq!(report.notices(), ["1 step(s) contain opaque parameters that may hold credentials and were kept unchecked"]); + } + let malformed = source.strip_suffix('}').unwrap(); + let (clean, report) = crate::roles::command("project.openText", &json!({"text":malformed}), 0); + assert_eq!(clean.unwrap()["text"], "[parameters withheld]", "malformed credential-bearing text keeps its existing handling"); + assert_eq!((report.withheld, report.unchecked), (1, 0)); +} + +#[test] +fn durable_formatter_errors_unwinds_and_apply_failures_restore_whole_fields() { + use durable::{Durable, FieldMut, Leaf, Owner, Prepare}; + let mut s = marked(); + s.project.maps[0].layers[0].dataset.source = Some("/tmp/rustgis-output/data.geojson".into()); + let mut document = Project::default(); + document.tasks = s.project.tasks.clone(); + let text = serde_json::to_string_pretty(&document).unwrap(); + let mut params = json!({"text": text}); + let mut id = "project.openText"; + for _ in 0..6 { + params = json!({"steps": [{"title":"nested", "command":id, "params":params}]}); + id = "task.new"; + } + s.project.tasks[0].steps[0].command = id.into(); + s.project.tasks[0].steps[0].params = params; + // A separate document is actually parsed and rewritten before the deeper envelope withholds. + s.project.tasks[0].steps.push(TaskStep { + title: "document".into(), + instructions: String::new(), + command: "project.openText".into(), + params: json!({"text":text}), + }); + let before = serde_json::to_value(&s.project).unwrap(); + let result = durable::output(&mut s.project, Some("/tmp/rustgis-output"), |p| { + assert_eq!(p.maps[0].layers[0].dataset.source.as_deref(), Some("./data.geojson")); + assert!(!serde_json::to_string(p).unwrap().contains(SECRET)); + Err::<(), _>("formatter failed".into()) + }); + assert!(result.is_err()); + assert_eq!(before, serde_json::to_value(&s.project).unwrap()); + #[cfg(not(target_arch = "wasm32"))] + { + let unwound = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let _ = durable::output(&mut s.project, Some("/tmp/rustgis-output"), |_| -> Result<(), String> { + std::panic::resume_unwind(Box::new("formatter failed")) + }); + })); + assert!(unwound.is_err()); + assert_eq!(before, serde_json::to_value(&s.project).unwrap()); + } + struct Bad(Project); + impl Durable for Bad { + /// Offers classified whole fields once, before any live mutation. + fn prepare(&self, p: &mut Prepare<'_>) { + self.0.prepare(p); + p.string(Leaf::Video(usize::MAX), "https://h.invalid/?token=RGSYNTH_output_20261103", true, false); + } + + /// Resolves a prepared location through owned fields, without allocation or policy. + fn field_mut(&mut self, owners: &[Owner], leaf: &Leaf) -> Option> { + self.0.field_mut(owners, leaf) + } + } + let mut bad = Bad(s.project); + assert!(durable::output(&mut bad, Some("/tmp/rustgis-output"), |_| Ok(())).is_err()); + assert_eq!(before, serde_json::to_value(&bad.0).unwrap()); +} + +#[test] +fn model_and_presentation_non_item_writers_use_public_forms() { + let mut s = marked(); + let graph = s.project.models[0].graph.clone().unwrap(); + let out = s.run("model.export", &json!({"model":"Credentials"})).unwrap(); + assert!(!out["text"].as_str().unwrap().contains(SECRET)); + let file = std::env::temp_dir().join(format!("rustgis-output-model-{}.json", std::process::id())); + s.run("model.export", &json!({"model":"Credentials", "path":file})).unwrap(); + assert!(!std::fs::read_to_string(&file).unwrap().contains(SECRET)); + std::fs::remove_file(file).unwrap(); + assert_eq!(s.project.models[0].graph.as_ref(), Some(&graph)); + let userinfo = format!("https://u:{SECRET}@h.invalid"); + assert_eq!(crate::cmd::presentation::video_label(&userinfo), "h.invalid"); + assert_eq!(crate::cmd::presentation::video_label("/tmp/movie?cut#1.mp4"), "movie?cut#1.mp4"); + s.run("presentation.videoPage", &json!({"source":userinfo})).unwrap(); + assert_eq!(s.project.presentations[0].pages.last().unwrap().title, "h.invalid"); + let (bytes, _) = crate::cmd::presentation::export_bytes(&mut s, 0, "html", 48.0).unwrap(); + let html = String::from_utf8(bytes).unwrap(); + assert!(!html.contains(SECRET)); + assert!(html.contains(DATA)); + assert!(s.notices.iter().any(|n| n.text.contains("video address(es)"))); +} + +#[test] +fn generated_notebook_errors_are_cleaned_before_the_output_is_retained() { + let mut s = marked(); + let source = format!("layer.addUrl {{\"url\":\"https://h.invalid/?token={SECRET}\",\"cellSize\":1}}"); + s.project.notebooks[0].cells[0].source = source.clone(); + s.trusted_notebooks.insert("credentials".into()); + let _ = s.run("notebook.run", &json!({"notebook":0})); + assert_eq!(s.project.notebooks[0].cells[0].source, source); + assert!(!s.project.notebooks[0].cells[0].output.as_ref().unwrap().contains(SECRET)); + for kind in ["notebook", "jupyter"] { + assert!(!String::from_utf8_lossy(&crate::cmd::items::item_bytes(&mut s, kind, &Value::Null).unwrap().1).contains(SECRET)); + } + // Failed file tools retain their failure history through the command rollback. The missing + // input is a layer name, so this reaches the production error path without any network I/O. + let history = s.project.gp_history.len(); + assert!( + s.run( + "gp.run", + &json!({"tool":"vectorToPmtiles", "params":{"input":format!("https://h.invalid/?token={SECRET}"), "minZoom":0, "maxZoom":0}}) + ) + .is_err() + ); + assert_eq!(s.project.gp_history.len(), history + 1); + let record = s.project.gp_history.last().unwrap(); + assert_eq!(record.id, "vectorToPmtiles"); + assert!(!record.ok && !record.messages.join("\n").contains(SECRET)); +} + +#[test] +fn peers_bypass_saved_restore_and_retention_requires_the_current_exact_slot() { + let mut s = marked(); + let saved = s.project.saved_text(None).unwrap().0; + s.trusted_tasks.insert("credentials".into()); + let generation = s.opened; + s.open_peer_text(Peer::Embedder, &saved).unwrap(); + assert!(s.opened != generation && s.trusted_tasks.is_empty()); + assert!(!serde_json::to_string(&s.project).unwrap().contains(SECRET)); + for scenario in ["unchanged", "rotation", "removed", "inserted", "moved", "accounts", "generation", "duplicate_table"] { + let mut s = marked(); + s.project.tasks.clear(); + s.project.models.clear(); + s.project.gp_history.clear(); + s.project.notebooks.clear(); + s.project.presentations.clear(); + if scenario == "accounts" { + let mut other = s.project.maps[0].layers[0].clone(); + other.id = 9000; + other.dataset.source = Some("https://h.invalid/a?token=RGSYNTH_other_20261103&layer=a%20b".to_string()); + s.project.maps[0].layers.push(other); + } + if scenario == "duplicate_table" { + let mut table = rustgis_data::Dataset::new("Table", rustgis_data::GeometryType::None, vec![]); + table.source = Some(format!("https://h.invalid/table?token={SECRET}")); + s.project.add_table(table); + } + let (public, _) = s.peer_text(Peer::Bridge).unwrap(); + assert!(!public.contains(SECRET)); + let mut incoming: Value = serde_json::from_str(&public).unwrap(); + match scenario { + "rotation" => { + s.project.maps[0].layers[0].dataset.source = Some("https://h.invalid/a?token=RGSYNTH_rotated_20261103&layer=a%20b".into()) + } + "removed" => { + s.project.maps[0].layers.remove(0); + } + "inserted" => incoming["maps"][0]["layers"][0]["id"] = json!(9001), + "moved" => { + incoming["maps"][0]["name"] = json!("Another map"); + } + "generation" => { + s.run("project.new", &Value::Null).unwrap(); + } + "duplicate_table" => { + let table = incoming["tables"][0].clone(); + incoming["tables"].as_array_mut().unwrap().push(table); + } + _ => {} + } + s.open_peer_text(Peer::Bridge, &incoming.to_string()).unwrap(); + let full = serde_json::to_string(&s.project).unwrap(); + if scenario == "unchanged" || scenario == "duplicate_table" { + assert!(full.contains(SECRET), "{scenario}: ordinary layer stays"); + } else { + assert!(!full.contains(SECRET), "{scenario}"); + } + if scenario == "duplicate_table" { + assert!(s.project.tables.iter().all(|t| !t.source.as_deref().unwrap_or("").contains(SECRET))); + } + assert!(s.trusted_tasks.is_empty() && s.trusted_models.is_empty() && s.trusted_notebooks.is_empty()); + } +} + +#[test] +fn peer_video_credential_losses_are_counted() { + let public = "https://h.invalid/video.mp4"; + let private = format!("{public}?token={SECRET}"); + for (peer, source, replacement, lost) in [ + (Peer::Embedder, public, public, 0), + (Peer::Embedder, private.as_str(), public, 1), + (Peer::Bridge, private.as_str(), public, 0), + (Peer::Bridge, private.as_str(), "https://h.invalid/other.mp4", 1), + ] { + let mut project = Project::default(); + project.presentations.push(Presentation { + name: "Video".into(), + pages: vec![Page { + title: "Video".into(), + kind: PageKind::Video { source: source.into(), poster: None, caption: String::new() }, + notes: String::new(), + seconds: 5.0, + }], + ..Default::default() + }); + let mut s = Session::new(project); + let (text, _) = s.peer_text(peer).unwrap(); + assert!(!text.contains(SECRET)); + let report = s.open_peer_text(peer, &text.replace(public, replacement)).unwrap(); + assert_eq!(report.lost(), lost, "{peer:?}: {source} -> {replacement}"); + if lost == 0 { + assert!(report.notices().is_empty(), "clean or retained video sources lose no credentials"); + } else { + let notices = report.notices(); + assert_eq!(notices.len(), 1); + assert!(notices[0].starts_with("1 ") && notices[0].contains("lost their keys")); + assert!(!serde_json::to_string(&s.project).unwrap().contains(SECRET)); + } + } +} + +#[cfg(not(target_arch = "wasm32"))] +#[test] +fn short_basic_passwords_are_scrubbed_only_from_output_while_active() { + if isolate_process_test("tests::credential_output::short_basic_passwords_are_scrubbed_only_from_output_while_active") { + return; + } + use crate::render::rustgis_tiles::logins; + + let mut other = Session::new(Project::default()); + let value = logins::basic("synthetic-user", "a"); + let origin = "https://short-basic.invalid"; + let binding = + logins::activate(origin, "synthetic-short-basic-output-owner", logins::Method::Header("Authorization".into()), &value, false) + .unwrap(); + let payload = value.strip_prefix("Basic ").unwrap(); + assert_eq!(logins::scrub(&format!("echo {value} {payload} a")), "echo [redacted] [redacted] [redacted]"); + other.notify("alpha", false); + assert_eq!(other.notices.last().unwrap().text, "[redacted]lph[redacted]", "active-secret matching covers other sessions' output"); + other.notify("loop done", false); + assert_eq!(other.notices.last().unwrap().text, "loop done", "unmatched output stays intact"); + let url = format!("{origin}/data"); + let bound = logins::bind(&binding, &url).unwrap(); + let request = logins::apply(&bound).unwrap().unwrap(); + assert_eq!(request.url, url, "output redaction never changes the request URL"); + assert_eq!(request.header.unwrap().1, value, "the raw credential remains the request input"); + assert!(logins::deactivate(&binding)); + other.notify("alpha", false); + assert_eq!(other.notices.last().unwrap().text, "alpha", "revocation does not retain the short-password pattern"); +} + +/// Imported database origins are output fields even when no database has been opened. +#[test] +fn durable_database_origins_clean_each_owned_field_without_opening_a_database() { + for (role, scheme, suffix) in [ + ("edit", "postgresql", "/postgis/origin/database"), + ("postgis", "postgresql", "/source/postgis/database"), + ("mysql", "mysql", "/source/mysql/database"), + ("sqlserver", "mssql", "/source/sqlserver/database"), + ("duckdb", "postgresql", "/source/duckdb/database"), + ("gdal+duckdb", "postgresql", "/source/duckdb/database"), + ] { + for private in [false, true] { + let public = format!("{scheme}://reader@db-origin.invalid/gis"); + let address = + if private { format!("{scheme}://reader:RGSYNTH_db_origin_20261106@db-origin.invalid/gis") } else { public.clone() }; + let dataset = rustgis_data::Dataset::new("Origin", rustgis_data::GeometryType::None, Vec::new()); + let mut layer = serde_json::to_value(rustgis_map::Layer::new(1, dataset)).unwrap(); + let mut origin = json!({"database":address, "table":"rows"}); + if matches!(role, "duckdb" | "gdal+duckdb") { + origin["family"] = json!("Points"); + } + if role == "edit" { + layer["postgis"] = json!({"origin":origin, "key":"id", "srid":4326, "writable":false}); + } else { + layer["source"] = json!({}); + layer["source"][if role == "gdal+duckdb" { "duckdb" } else { role }] = origin; + if role == "gdal+duckdb" { + layer["source"]["gdal"] = json!({"path":"unused.geojson", "layer":"rows"}); + } + } + let mut project = Project::default(); + project.map_mut().layers.push(serde_json::from_value(layer).unwrap()); + // This is the real import decoder before normalization opens any source. Session::new + // preserves the decoded project; neither output API opens a database or runs a command. + let imported = Project::from_peer_raw(&serde_json::to_string(&project).unwrap()).unwrap(); + let mut s = Session::new(imported); + let before = serde_json::to_value(&s.project).unwrap(); + let slot = format!("/maps/0/layers/0{suffix}"); + assert_eq!(before.pointer(&slot), Some(&json!(address)), "{role}: Deserialize keeps the imported field"); + // Output serialization uses each field's JSON representation (not f32 -> Value widening). + let mut expected: Value = serde_json::from_str(&serde_json::to_string(&s.project).unwrap()).unwrap(); + *expected.pointer_mut(&slot).unwrap() = json!(public); + let (saved, report) = s.project.saved_text(None).unwrap(); + let outputs = [ + ("save", (saved, report)), + ("embedder", s.peer_text(Peer::Embedder).unwrap()), + ("bridge", s.peer_text(Peer::Bridge).unwrap()), + ]; + for (kind, (text, report)) in outputs { + assert!(!text.contains("RGSYNTH_db_origin_20261106"), "{kind}: {role}"); + assert_eq!(serde_json::from_str::(&text).unwrap(), expected, "{kind}: {role}, private={private}"); + assert_eq!(report.layer_addresses(), u32::from(private), "{kind}: {role}"); + let notices = if private { + vec!["1 layer address(es) were saved without their keys, tokens, passwords or URL fragments; after RustGIS closes, add those layers again to restore missing values".to_string()] + } else { + Vec::new() + }; + assert_eq!(report.notices(), notices, "{kind}: {role}"); + } + if role == "gdal+duckdb" { + let mark = s.notice_mark(); + let (_, bytes) = crate::cmd::items::item_bytes(&mut s, "layer", &json!({"layer":1})).unwrap(); + let written: Value = serde_json::from_slice(&bytes).unwrap(); + assert_eq!(written, expected["maps"][0]["layers"][0]); + assert_eq!(s.notices_since(mark).len(), usize::from(private), "layer export has one aggregate notice or none"); + } + assert_eq!(serde_json::to_value(&s.project).unwrap(), before, "{role}: all output rolls back live fields"); + } + } +} + +/// Runtime-only login omissions and executable replacement losses use different notice kinds. +#[test] +fn peer_output_counts_runtime_logins_and_changed_executable_fields() { + let public = "https://peer-runtime.invalid/data"; + let bound = rustgis_data::login_address::bind("0123456789abcdef", "https://peer-runtime.invalid", public).unwrap(); + let mut s = Session::new(Project::default()); + let mut dataset = rustgis_data::Dataset::new("Bound", rustgis_data::GeometryType::None, Vec::new()); + dataset.source = Some(bound.clone()); + let id = s.map_mut().add_dataset(dataset); + s.map_mut().layer_mut(id).unwrap().connection = "saved-connection".into(); + s.project.tasks.push(Task { + name: "Login".into(), + description: String::new(), + steps: vec![TaskStep { + title: String::new(), + instructions: String::new(), + command: "connection.login".into(), + params: json!({"apiKey":SECRET}), + }], + }); + let (text, report) = s.peer_text(Peer::Embedder).unwrap(); + assert_eq!(report.layer_addresses(), 1, "a foreign receiver cannot use the recorded connection's live binding"); + assert!(!text.contains("rustgis-login:") && !text.contains(SECRET)); + assert_eq!(s.map().layer(id).unwrap().dataset.source.as_deref(), Some(bound.as_str())); + let report = s.open_peer_text(Peer::Embedder, &text).unwrap(); + assert_eq!(report.executable_changes, 1); + assert!(report.notices().iter().any(|n| n.contains("executable field(s) changed"))); + assert_eq!(s.project.tasks[0].steps[0].params["apiKey"], ""); +} + +/// A valid S3 wrapper's endpoint is also an address, not an opaque place to store a password. +#[test] +fn durable_s3_endpoints_clean_recognized_credentials_and_keep_provenance() { + for (endpoint, expected, count) in [ + ("https://reader:RGSYNTH_endpoint@h.invalid", "https://h.invalid", 1), + ("https://h.invalid?token=RGSYNTH_endpoint", "https://h.invalid", 1), + ("https://h.invalid", "https://h.invalid", 0), + ("", "", 0), + ] { + let address = rustgis_data::s3_address::bind("0123456789abcdef", endpoint, "s3://bucket/data").unwrap(); + let mut p = Project::default(); + let mut table = rustgis_data::Dataset::new("S3", rustgis_data::GeometryType::None, Vec::new()); + table.source = Some(address.clone()); + p.add_table(table); + let (text, report) = p.saved_text(None).unwrap(); + assert!(!text.contains("RGSYNTH_endpoint")); + assert_eq!(report.layer_addresses(), count); + let value: Value = serde_json::from_str(&text).unwrap(); + let parsed = rustgis_data::s3_address::parse(value["tables"][0]["source"].as_str().unwrap()).unwrap().unwrap(); + assert_eq!((parsed.binding, parsed.endpoint, parsed.url), ("0123456789abcdef", expected, "s3://bucket/data")); + assert_eq!(p.tables[0].source.as_deref(), Some(address.as_str())); + } +} + +#[test] +fn nested_notebook_source_stays_opaque_but_is_reported_unchecked() { + for command in ["notebook.addCell", "notebook.editCell"] { + for source in ["layer.addUrl {\"url\":\"https://h.invalid/?token=RGSYNTH_nested\"}", "print('RGSYNTH_nested token')", "clean prose"] + { + let mut p = Project::default(); + p.tasks.push(Task { + name: "Nested".into(), + description: String::new(), + steps: vec![TaskStep { + title: String::new(), + instructions: String::new(), + command: command.into(), + params: json!({"source":source}), + }], + }); + let (text, report) = p.saved_text(None).unwrap(); + let written: Value = serde_json::from_str(&text).unwrap(); + assert_eq!(written["tasks"][0]["steps"][0]["params"]["source"], source); + assert_eq!(report.unchecked, u32::from(source != "clean prose")); + assert_eq!(report.steps, 0); + } + } + for url in ["https://h.invalid/a", "pg:host=h dbname=gis", "mariadb://h/gis", "pmtiles:https://h.invalid/a", "cog:https://h.invalid/a"] + { + assert_eq!(crate::log_params(&json!({"key":"abc", "source":{"url":url}})).into_log_value()["key"], "", "{url}"); + } + assert_eq!(crate::log_params(&json!({"key":"abc", "data":{"url":"https://h.invalid/a"}})).into_log_value()["key"], "abc"); +} + +/// A retained address must actually reopen its protected reader, not merely survive as text. +#[cfg(not(target_arch = "wasm32"))] +#[test] +fn bridge_round_trip_reopens_its_protected_reader() { + if super::isolate_process_test("tests::credential_output::bridge_round_trip_reopens_its_protected_reader") { + return; + } + use rustgis_data::source::{fgb, remote}; + let (dataset, _) = rustgis_data::io::read_geojson(r#"{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[1,2]},"properties":{"value":7}}]}"#, "Protected").unwrap(); + let bytes = fgb::write(&dataset).unwrap(); + let original = fgb::FgbSource::open_bytes(bytes.clone(), "Protected").unwrap(); + let full = "https://protected-reader.invalid/data.fgb?token=RGSYNTH_reader"; + // Session construction installs the native fetcher; replace it only after that setup. + let mut s = Session::new(Project::default()); + remote::set_fetcher(move |url, want, done| { + if url != full { + done(Err(remote::Failed::Other("protected reader needs its token".into()))); + } else if matches!(want, remote::Want::Whole) { + done(Ok(bytes.clone())); + } else { + done(Err(remote::Failed::NoRanges)); + } + }); + let id = s.map_mut().add_dataset(dataset); + s.map_mut().layer_mut(id).unwrap().source = Some(rustgis_map::source::SourceLayer::from_path(full, None, Box::new(original))); + let (text, _) = s.peer_text(Peer::Bridge).unwrap(); + assert!(!text.contains("RGSYNTH_reader")); + let report = s.open_peer_text(Peer::Bridge, &text).unwrap(); + assert_eq!(report.lost, 0, "the unchanged streamed source retains its address"); + let source = s.map_mut().layer_mut(id).unwrap().source.as_mut().unwrap(); + assert_eq!(source.path.as_deref(), Some(full), "the exact source keeps its protected address"); + source.wait_open(std::time::Duration::from_secs(5)); + assert!(source.is_open(), "retained reader failed to open: {:?}", source.error); + let (read, _) = source.read_all().unwrap(); + assert_eq!(read.features.len(), 1); + assert_eq!(read.features[0].attrs[0], rustgis_data::Value::Int(7)); +} + +/// Discovery must announce omitted settings even when the following supplied document is invalid. +#[test] +fn failed_collections_discovery_retains_its_counted_query_notice() { + for (query, count) in [("", 0), ("?token=RGSYNTH_discovery", 1)] { + let mut s = Session::new(Project::default()); + let p = json!({"url":format!("https://discovery.invalid/root{query}"), "type":"ogcapi", + "text":json!({"links":[{"rel":"data", "type":"application/json", "href":"/catalog"}]}).to_string(), + "documents":[{"url":"https://discovery.invalid/catalog", "text":"invalid capabilities"}]}); + assert!(s.run("service.layers", &p).is_err()); + let messages: Vec<_> = s.notices.iter().filter(|n| n.text.contains("address(es) the service listed")).collect(); + assert_eq!(messages.len(), count); + if count > 0 { + assert!(messages[0].text.starts_with("1 address(es)")); + } + } +} + +/// Coverage reports discovery omissions even when the selected coverage is missing. +#[test] +fn coverage_discovery_retains_its_counted_query_notice() { + for (query, count) in [("", 0), ("?token=RGSYNTH_coverage", 1)] { + let mut s = Session::new(Project::default()); + let p = json!({"url":format!("https://coverage.invalid/root{query}"), "type":"ogcapi", "layer":"missing", + "text":json!({"links":[{"rel":"data", "type":"application/json", "href":"/catalog"}]}).to_string(), + "documents":[{"url":"https://coverage.invalid/catalog", "text":"{\"collections\":[]}"}]}); + assert!(s.run("service.coverage", &p).unwrap_err().contains("no coverage")); + assert_eq!(s.notices.iter().filter(|n| n.text.contains("address(es) the service listed")).count(), count); + } +} + +/// Capabilities are opaque protocol documents, distinct from user feature data and expressions. +#[test] +fn supplied_service_documents_are_preserved_with_counted_unchecked_notices() { + for command in ["service.layers", "service.add", "service.coverage", "service.import", "service.groupPlan"] { + for private in [false, true] { + let text = if private { "https://h.invalid/a?token=RGSYNTH_document" } else { "https://h.invalid/a" }; + let (clean, report) = crate::roles::command(command, &json!({"text":text}), 0); + assert!(clean.is_none(), "{command}: protocol documents keep exact bytes"); + assert_eq!(report.unchecked, u32::from(private), "{command}: {report:?}"); + assert_eq!(report.notices().len(), usize::from(private)); + } + } +} diff --git a/crates/engine/src/tests/remembered_database_packets.rs b/crates/engine/src/tests/remembered_database_packets.rs index 954d439e4..6e94260fd 100644 --- a/crates/engine/src/tests/remembered_database_packets.rs +++ b/crates/engine/src/tests/remembered_database_packets.rs @@ -925,3 +925,54 @@ fn remembered_database_session_refusal_suppresses_only_matching_saved_value() { } } } + +/// A peer cannot use a typed or saved password to select a different database table. +#[test] +fn peer_database_replacement_requires_the_exact_previously_shared_source() { + if isolated("peer_database_replacement_requires_the_exact_previously_shared_source") { + return; + } + for saved in [false, true] { + for changed in [true, false] { + let server = Server::start(true); + let mut s = app(Arc::new(MemorySecrets::new()), "{}"); + let id = if saved { + remember(&mut s, "Peer", &server.address, "RGSYNTH_peer_password") + } else { + let id = s.run("connection.new", &json!({"kind":"postgres", "address":server.address})).unwrap()["id"] + .as_str() + .unwrap() + .to_string(); + let (_, key) = crate::connections::database_key(&server.address).unwrap(); + rustgis_data::source::duckdb::remember_password(&key, "RGSYNTH_peer_password"); + id + }; + // Install the approved source without starting its reader before the exchange. + s.project = serde_json::from_str(&project_layers(&server.address, 1)).unwrap(); + let (text, _) = s.peer_text(crate::Peer::Bridge).unwrap(); + let mut incoming: Value = serde_json::from_str(&text).unwrap(); + if changed { + incoming["maps"][0]["layers"][0]["source"]["postgis"]["table"] = json!("unapproved"); + } else { + incoming["maps"][0]["layers"][0]["visible"] = json!(false); + } + let report = s.open_peer_text(crate::Peer::Bridge, &incoming.to_string()).unwrap(); + if changed { + server.empty(); + assert!(report.notices().iter().any(|n| n.contains("reconnect"))); + let source = s.map_mut().layers[0].source.as_mut().unwrap(); + assert!(source.open().is_err(), "a later automatic open must not adopt a password either"); + server.empty(); + let (again, _) = s.peer_text(crate::Peer::Bridge).unwrap(); + let report = s.open_peer_text(crate::Peer::Bridge, &again).unwrap(); + assert!(report.notices().iter().any(|n| n.contains("reconnect")), "another peer round trip cannot approve the source"); + server.empty(); + s.run("connection.login", &json!({"id":id})).unwrap(); + assert_eq!(server.take(), b"RGSYNTH_peer_password\0", "explicit reconnect remains available"); + } else { + assert_eq!(server.take(), b"RGSYNTH_peer_password\0"); + assert!(report.notices().is_empty(), "unchanged readers keep their login: {report:?}"); + } + } + } +} diff --git a/crates/engine/src/tests/service_logins.rs b/crates/engine/src/tests/service_logins.rs index 2fb107c60..3d9657255 100644 --- a/crates/engine/src/tests/service_logins.rs +++ b/crates/engine/src/tests/service_logins.rs @@ -73,6 +73,7 @@ impl LoginFile { } Err(_) => break, }; + stream.set_nonblocking(false).unwrap(); stream.set_read_timeout(Some(std::time::Duration::from_secs(5))).unwrap(); let mut request = Vec::new(); let mut chunk = [0u8; 4096]; @@ -242,6 +243,7 @@ fn an_oversized_ogc_items_binding_is_refused_before_native_fetch() { use crate::render::rustgis_tiles::cache; let server = LoginFile::new(br#"{"type":"FeatureCollection","features":[]}"#.to_vec()); let public = LoginFile::new(b"public".to_vec()); + assert_ne!(server.base, public.base, "the fixtures need different origins"); let mut s = session(None); let id = new(&mut s, json!({"kind":"ogcapi","address":format!("{}/collections",server.base),"auth":"header","keyName":"X-Key"})); s.run("connection.login", &json!({"id":id,"apiKey":"fixture-login-757","allowInsecure":true})).unwrap(); @@ -265,8 +267,38 @@ fn an_oversized_ogc_items_binding_is_refused_before_native_fetch() { assert!(server.requests.lock().unwrap().is_empty(), "an oversized binding must never fetch anonymously"); assert!(cache::download_blocking(items("valid")).is_ok()); assert!(server.requests.lock().unwrap()[0].to_ascii_lowercase().contains("\r\nx-key: fixture-login-757\r\n")); - assert!(!rustgis_data::login_address::is_wrapper(items("foreign"))); - assert_eq!(cache::download_blocking(items("foreign")).unwrap(), b"public"); + let foreign = items("foreign"); + assert!(!rustgis_data::login_address::is_wrapper(foreign)); + assert_eq!(foreign, format!("{}/public?limit=10", public.base)); + let downloaded = cache::download_blocking(foreign).unwrap_or_else(|_| { + // Only request structure is diagnostic: never print headers, targets or error text. + let summarize = |request: &String| { + let mut words = request.split_whitespace(); + let method = match words.next() { + Some("GET") => "GET", + Some("CONNECT") => "CONNECT", + Some(_) => "other", + None => "missing", + }; + let target = words.next().unwrap_or(""); + let form = match target { + "" => "missing", + t if t.starts_with('/') => "origin", + t if t.starts_with("http://") || t.starts_with("https://") => "absolute", + _ if method == "CONNECT" => "authority", + _ => "other", + }; + json!({ + "bytes": request.len(), "fullHeader": request.contains("\r\n\r\n"), + "method": method, "targetForm": form, "publicPrefix": target.starts_with("/public"), + "xKeyPresent": request.to_ascii_lowercase().contains("\r\nx-key:"), + }) + }; + let protected = server.requests.lock().unwrap().iter().map(&summarize).collect::>(); + let anonymous = public.requests.lock().unwrap().iter().map(&summarize).collect::>(); + panic!("foreign fixture download failed; protected={protected:?}; public={anonymous:?}"); + }); + assert_eq!(downloaded, b"public"); assert!(!public.requests.lock().unwrap()[0].to_ascii_lowercase().contains("\r\nx-key:")); } @@ -398,7 +430,7 @@ fn service_login_edit_and_remove_do_not_depend_on_a_credential_store() { assert_eq!(later.run("connection.login", &json!({"id":id})).unwrap_err(), web::REQUIRED); s.run("connection.edit", &json!({"id":id,"address":"https://changed-memory757.example.org/ogc"})).unwrap(); assert!(logins::apply(&address).is_err()); - s.run("connection.login", &json!({"id":id,"apiKey":"replacement-sentinel-757"})).unwrap(); + s.run("connection.login", &json!({"id":id,"apiKey":"fake004"})).unwrap(); let replacement = bound(&mut s, &id, "https://changed-memory757.example.org/features"); s.run("connection.remove", &json!({"id":id})).unwrap(); assert!(logins::apply(&replacement).is_err()); @@ -689,7 +721,7 @@ fn logins_drop_the_tile_cache_only_when_one_ends_or_is_replaced() { assert_eq!(bound(&mut s, &private, "https://cache-private757.example.org/wms"), first); assert_eq!(web::cache_clears(), start); // Replacing or ending a login leaves authenticated responses behind: those go. - s.run("connection.login", &json!({"id":private,"apiKey":"replacement-sentinel-757"})).unwrap(); + s.run("connection.login", &json!({"id":private,"apiKey":"fake004"})).unwrap(); assert_eq!(web::cache_clears(), start + 1); s.run("connection.logout", &json!({"id":private})).unwrap(); assert_eq!(web::cache_clears(), start + 2); @@ -742,11 +774,11 @@ fn an_unsaved_anonymous_test_never_carries_a_saved_login() { let mut s = session(None); let address = "https://probe757.example.org/wms"; let id = new(&mut s, json!({"kind":"wms","address":address,"auth":"header","keyName":"X-Key"})); - s.run("connection.login", &json!({"id":id,"apiKey":"probe-sentinel-757"})).unwrap(); + s.run("connection.login", &json!({"id":id,"apiKey":"fake005"})).unwrap(); let unsaved = s.run("connection.test", &json!({"kind":"wms","address":address,"probe":true})).unwrap(); assert!(header(unsaved["fetch"].as_str().unwrap()).is_none(), "{unsaved}"); let saved = s.run("connection.test", &json!({"id":id,"probe":true})).unwrap(); - assert_eq!(header(saved["fetch"].as_str().unwrap()).unwrap().1, "probe-sentinel-757"); + assert_eq!(header(saved["fetch"].as_str().unwrap()).unwrap().1, "fake005"); // The anonymous read ended with the test. assert!(header(&web::request(&mut s, None, address).unwrap()).is_some()); } @@ -1105,18 +1137,18 @@ fn plugin_downloads_stay_anonymous_after_the_front_end_adds_them() { s.run_leaving_downloads( "plugin:eng-deferred-login-757/f", &json!({ - "op":"layer.addUrl","params":{"url":format!("{origin}/roads.geojson"),"name":"Deferred","anonymous":false} + "op":"layer.addUrl","params":{"url":format!("{origin}/roads.geojson"),"name":"Deferred","anonymous":false,"service":true} }), ) .unwrap(); - let mut pending = s.take_url_requests().pop().unwrap(); - // ui.addUrl binds/probes after the plugin has returned. It must still fetch anonymously. - let prepared = web::request_params(&mut s, &pending).unwrap().into_owned(); - assert!(header(prepared["url"].as_str().unwrap()).is_none()); + let request = s.take_url_requests().pop().unwrap(); + // The claimed UI path preserves the engine's bound parameters and anonymous provenance. + let mut pending = request.clone(); + assert_eq!(pending["anonymous"], true); + assert!(header(pending["url"].as_str().unwrap()).is_none()); let token = s.download_started(); assert!(s.download_landed(token, br#"{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[1,2]},"properties":{}}]}"#.to_vec())); pending["id"] = json!(token); - pending["service"] = json!(true); s.run("layer.addDownload", &pending).unwrap(); s.run("layer.addUrl", &json!({"url":format!("{origin}/deferred.pmtiles"),"anonymous":true,"name":"Archive"})).unwrap(); s.run("connection.login", &json!({"id":id,"apiKey":"deferred-sentinel-757"})).unwrap(); @@ -1247,16 +1279,21 @@ fn templates_tasks_and_models_written_out_keep_no_login_binding() { let tile = bound(&mut s, &id, "https://export757.example.org/tiles/{z}/{x}/{y}.png"); s.run("map.addTiles", &json!({"url": tile, "name": "Bound"})).unwrap(); s.run("task.new", &json!({"name":"Fetch","steps":[{"title":"Add","command":"map.addTiles","params":{"url": tile}}]})).unwrap(); - s.run("model.new", &json!({"name":"Tiles","steps":[{"tool":"buffer","params":{"input": tile, "distance": 1}}]})).unwrap(); - // A model drawn as a graph keeps its input's address in the node itself. - let graph = json!({"nodes":[{"id":"in","kind":"input","layer": tile}], "edges": []}); + // Keep the synthetic address in a tool's parameters, not a layer-name selector. + s.run("model.new", &json!({"name":"Tiles","graph":{"nodes":[{"id":"tool1","kind":"tool","tool":"buffer","params":{"input": tile,"distance":1}}],"edges":[]}})).unwrap(); + // A graph input selects an existing layer by name; its actual source fields are cleaned. + let graph = json!({"nodes":[{"id":"in","kind":"input","layer": "Cities"}], "edges": []}); s.run("model.new", &json!({"name":"Drawn","graph": graph})).unwrap(); let dir = std::env::temp_dir().join(format!("export757-{}", std::process::id())); std::fs::create_dir_all(&dir).unwrap(); let (template, task) = (dir.join("bound.rgistemplate"), dir.join("bound.rgistask")); s.run("project.saveTemplate", &json!({"path": template.to_string_lossy()})).unwrap(); s.run("task.save", &json!({"task":"Fetch","path": task.to_string_lossy()})).unwrap(); - let model = s.run("model.export", &json!({"model":"Tiles"})).unwrap()["text"].as_str().unwrap().to_string(); + let model = s.run("model.export", &json!({"name":"Tiles"})).unwrap()["text"].as_str().unwrap().to_string(); + let drawn = s.run("model.export", &json!({"name":"Drawn"})).unwrap()["text"].as_str().unwrap().to_string(); + let graph: Value = serde_json::from_str(&drawn).unwrap(); + assert_eq!(graph["graph"]["nodes"][0]["layer"], "Cities"); + assert!(!drawn.contains(rustgis_data::login_address::PREFIX) && !drawn.contains("export-sentinel-757")); let written = [std::fs::read_to_string(&template).unwrap(), std::fs::read_to_string(&task).unwrap(), model]; let _ = std::fs::remove_dir_all(&dir); for text in &written { @@ -1282,7 +1319,7 @@ fn service_requests_take_a_connection_without_saying_they_ignore_it() { let url = "https://browse757.example.org/collections"; let mut s = session(None); let id = new(&mut s, json!({"kind":"ogcapi","address":url,"auth":"header","keyName":"X-Key"})); - s.run("connection.login", &json!({"id":id,"apiKey":"browse-sentinel-757"})).unwrap(); + s.run("connection.login", &json!({"id":id,"apiKey":"fake006"})).unwrap(); let document = json!({"collections":[{"id":"roads","title":"roads", "links":[{"rel":"items","type":"application/geo+json","href":format!("{url}/roads/items")}]}]}); let mark = s.notice_mark(); @@ -1292,8 +1329,43 @@ fn service_requests_take_a_connection_without_saying_they_ignore_it() { assert!(said.iter().all(|n| !n.contains("ignores")), "{said:?}"); } +#[test] +fn resolved_same_origin_tiles_keep_authenticated_and_public_connection_provenance() { + use crate::cmd::service::add_resolved_tiles; + for auth in ["header", "none"] { + let origin = format!("https://resolved-{auth}-r16.example.invalid"); + let url = format!("{origin}/collections"); + let list = format!("{origin}/tiles"); + let template = format!("{origin}/paint/{{z}}/{{x}}/{{y}}.png"); + let document = json!({"collections":[{"id":"relief","links":[{"rel":"tilesets-map","href":list}]}]}).to_string(); + let mut s = session(None); + let id = new(&mut s, json!({"kind":"ogcapi","address":url,"auth":auth,"keyName":"X-Key"})); + if auth == "header" { + s.run("connection.login", &json!({"id":id,"apiKey":"fake007"})).unwrap(); + } + let template = bound(&mut s, &id, &template); + add_resolved_tiles(&mut s, &json!({"url":url,"text":document,"layer":"relief (map tiles)","connection":id}), &template, false) + .unwrap(); + let tile = &s.map().tiles[0]; + assert!(!tile.anonymous); + assert_eq!(tile.connection, id); + assert_eq!(header(&tile.url).is_some(), auth == "header"); + if auth == "none" { + // No binding was present, but no foreign hop occurred: the saved connection is + // still the user's provenance when they later add a login to it. + s.run("connection.edit", &json!({"id":id,"auth":"header","keyName":"X-Key"})).unwrap(); + s.run("connection.login", &json!({"id":id,"apiKey":"fake008"})).unwrap(); + assert_eq!(header(&s.map().tiles[0].url).unwrap().1, "fake008"); + } + } +} + #[test] fn notebook_output_keeps_no_login_binding() { + #[cfg(not(target_arch = "wasm32"))] + if super::isolate_process_test("tests::service_logins::notebook_output_keeps_no_login_binding") { + return; + } let origin = "https://notebook757.example.org"; let mut s = session(None); let id = new(&mut s, json!({"kind":"wmts","address":format!("{origin}/tiles"),"auth":"header","keyName":"X-Key"})); @@ -1307,6 +1379,33 @@ fn notebook_output_keeps_no_login_binding() { assert!(output.contains(&format!("{origin}/tiles/")), "the cell still shows the address: {output}"); let (saved, _) = s.project.clone().saved_text(None).unwrap(); assert!(!saved.contains(rustgis_data::login_address::PREFIX), "the saved project keeps a binding: {saved}"); + + // An active secret can also be ordinary address syntax. It must not hide the scheme + // from the separate URL-credential sanitizer before that sanitizer has run. + s.run("connection.login", &json!({"id":id,"apiKey":"https"})).unwrap(); + let token = "RGSYNTH_notebook_collision_r18"; + s.run( + "map.addTiles", + &json!({"url":format!("https://tiles-collision.invalid/{{z}}/{{x}}/{{y}}.png?token={token}"),"name":"Collision"}), + ) + .unwrap(); + s.run("notebook.run", &Value::Null).unwrap(); + let output = s.project.notebooks[0].cells.last().unwrap().output.as_deref().unwrap(); + assert!(!output.contains(token), "URL query secret survives a colliding active key"); + assert!(output.contains("tiles-collision.invalid/"), "the useful address survives: {output}"); + let (saved, _) = s.project.clone().saved_text(None).unwrap(); + assert!(!saved.contains(token)); + assert!(!saved.contains(rustgis_data::login_address::PREFIX)); + + // The durable generated-output boundary also receives raw command results, such as + // an imported notebook's display, independently of this notebook's run producer. + let raw = s.run("map.tiles", &Value::Null).unwrap().to_string(); + assert!(raw.contains(token)); + let mut project = s.project.clone(); + project.notebooks[0].cells.last_mut().unwrap().output = Some(raw); + let (saved, _) = project.saved_text(None).unwrap(); + assert!(!saved.contains(token), "durable generated text keeps a URL query secret"); + assert!(!saved.contains(rustgis_data::login_address::PREFIX)); } /// The server has the credential before the lifecycle action, but its refusal arrives after it. @@ -1432,6 +1531,166 @@ fn delayed_head_refusal_keeps_secrets_out_of_history_across_login_lifecycle() { } } +/// A foreign capability fallback cannot select a destination login now or after reopening. +pub(super) fn foreign_service_fallback_stays_anonymous(kind: &str, source: &str, layer: &str, clean: &str, foreign: &str) { + let mut results = Vec::new(); + for (crossed, text) in [(false, clean), (true, foreign)] { + let mut s = session(None); + let a = new(&mut s, json!({"kind":kind,"address":source,"auth":"header","keyName":"X-Key"})); + let b = new(&mut s, json!({"kind":kind,"address":"https://fallback-b.invalid","auth":"header","keyName":"X-Key"})); + s.run("connection.login", &json!({"id":a,"apiKey":"RGSYNTH_fallback_a"})).unwrap(); + s.run("connection.login", &json!({"id":b,"apiKey":"RGSYNTH_fallback_b"})).unwrap(); + s.run("service.add", &json!({"url":source,"layer":layer,"text":text,"connection":a,"time":"default"})).unwrap(); + for reopened in [false, true] { + if reopened { + let saved = s.project.saved_text(None).unwrap().0; + s.run("project.openText", &json!({"text":saved})).unwrap(); + } + s.run("connection.login", &json!({"id":b,"apiKey":"RGSYNTH_fallback_rotated"})).unwrap(); + let tile = &s.map().tiles[0]; + results.push(( + crossed, + reopened, + if crossed { + tile.anonymous && tile.connection.is_empty() && header(&tile.url).is_none() + } else { + !tile.anonymous && tile.connection == a && header(&tile.url).is_some() + }, + )); + } + } + assert!(results.iter().all(|(_, _, kept)| *kept), "foreign fallback adopted a login or same-origin binding changed: {results:?}"); +} + +/// A foreign capabilities document remains foreign even when it links back to the source origin. +#[test] +fn capabilities_round_trip_keeps_foreign_lineage() { + let mut results = Vec::new(); + for crossed in [false, true] { + let mut s = session(None); + let a = "https://lineage-a.invalid"; + let bridge = if crossed { "https://lineage-b.invalid/bridge" } else { "https://lineage-a.invalid/bridge" }; + let id = new(&mut s, json!({"kind":"ogcapi","address":a,"auth":"header","keyName":"X-Key"})); + s.run("connection.login", &json!({"id":id,"apiKey":"RGSYNTH_capabilities_lineage"})).unwrap(); + let link = |url: &str| json!({"links":[{"rel":"data","type":"application/json","href":url}]}).to_string(); + s.run("service.add", &json!({ + "url":format!("{a}/root"),"type":"ogcapi","connection":id,"layer":"relief (map tiles)", + "text":link(bridge),"documents":[ + {"url":bridge,"text":link(&format!("{a}/returned"))}, + {"url":format!("{a}/returned"),"text":json!({"collections":[{"id":"relief","links":[{"rel":"tilesets-map","href":format!("{a}/list")}]}]}).to_string()} + ],"tiles":format!("{a}/paint/{{z}}/{{x}}/{{y}}.png") + })).unwrap(); + let tile = &s.map().tiles[0]; + results.push(( + crossed, + if crossed { + tile.anonymous && tile.connection.is_empty() && header(&tile.url).is_none() + } else { + !tile.anonymous && tile.connection == id && header(&tile.url).is_some() + }, + )); + } + assert!(results.iter().all(|(_, kept)| *kept), "capabilities discovery forgot its foreign hop: {results:?}"); +} + +/// Styles reached through capabilities retain the same origin history as raster/vector tiles. +#[test] +fn capabilities_styles_keep_foreign_lineage() { + let mut results = Vec::new(); + for crossed in [false, true] { + let mut s = session(None); + let a = "https://style-lineage-a.invalid"; + let b = "https://style-lineage-b.invalid"; + let source = if crossed { b } else { a }; + let doc = format!("{source}/rest/services/X/VectorTileServer"); + let login = new(&mut s, json!({"kind":"ogcapi","address":a,"auth":"header","keyName":"X-Key"})); + let other = new(&mut s, json!({"kind":"ogcapi","address":b,"auth":"header","keyName":"X-Key"})); + s.run("connection.login", &json!({"id":login,"apiKey":"RGSYNTH_style_a"})).unwrap(); + s.run("connection.login", &json!({"id":other,"apiKey":"RGSYNTH_style_b"})).unwrap(); + s.run( + "service.add", + &json!({ + "url":format!("{a}/root"),"type":"ogcapi","connection":login,"layer":"style", + "text":json!({"links":[{"rel":"data","type":"application/json","href":doc}]}).to_string(), + "documents":[{"url":doc,"text":json!({"currentVersion":11,"name":"Private style"}).to_string()}] + }), + ) + .unwrap(); + for reopened in [false, true] { + if reopened { + let saved = s.project.saved_text(None).unwrap().0; + s.run("project.openText", &json!({"text":saved})).unwrap(); + } + s.run("connection.login", &json!({"id":other,"apiKey":"RGSYNTH_style_rotated"})).unwrap(); + let tile = &s.map().tiles[0]; + results.push(( + crossed, + reopened, + if crossed { + tile.anonymous && tile.connection.is_empty() && header(&tile.url).is_none() + } else { + !tile.anonymous && tile.connection == login && header(&tile.url).is_some() + }, + )); + } + } + assert!(results.iter().all(|(_, _, kept)| *kept), "style discovery forgot its foreign hop: {results:?}"); +} + +/// Features reached through foreign capabilities keep the same opt-out as tile discovery. +#[test] +fn feature_group_plans_and_imports_keep_foreign_capabilities_anonymous() { + #[cfg(not(target_arch = "wasm32"))] + if super::isolate_process_test("tests::service_logins::feature_group_plans_and_imports_keep_foreign_capabilities_anonymous") { + return; + } + for (document_crossed, items_crossed, authenticated) in + [(false, false, false), (false, false, true), (true, false, true), (false, true, true)] + { + let crossed = document_crossed || items_crossed; + let mut s = session(None); + let a = "https://feature-lineage-a.invalid"; + let b = "https://feature-lineage-b.invalid"; + let bridge = if document_crossed { format!("{b}/bridge") } else { format!("{a}/bridge") }; + let items = if items_crossed { format!("{b}/items") } else { format!("{a}/items") }; + let login = new(&mut s, json!({"kind":"ogcapi", "address":a, "auth":if authenticated {"header"} else {"none"}, "keyName":"X-Key"})); + if authenticated { + s.run("connection.login", &json!({"id":login, "apiKey":"RGSYNTH_feature_lineage"})).unwrap(); + } + let other = new(&mut s, json!({"kind":"ogcapi", "address":b, "auth":"header", "keyName":"X-Key"})); + s.run("connection.login", &json!({"id":other, "apiKey":"RGSYNTH_feature_destination"})).unwrap(); + let link = |url: &str| json!({"links":[{"rel":"data", "type":"application/json", "href":url}]}).to_string(); + let mut params = json!({"url":format!("{a}/root"), "type":"ogcapi", "connection":login, + "text":link(&bridge), "documents":[ + {"url":bridge,"text":link(&format!("{a}/returned"))}, + {"url":format!("{a}/returned"),"text":json!({"collections":[{"id":"roads", "links":[{"rel":"items", "type":"application/geo+json", "href":items}]}]}).to_string()} + ]}); + let plan = s.run("service.groupPlan", ¶ms).unwrap(); + let feature = &plan["layers"][0]; + assert_eq!(feature.get("anonymous").and_then(Value::as_bool).unwrap_or(false), crossed); + let listed = s.run("service.layers", ¶ms).unwrap(); + assert_eq!(listed["layers"][0].get("anonymous").and_then(Value::as_bool).unwrap_or(false), crossed); + params["responses"] = json!([{"url":feature["offer"]["url"], "text":r#"{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[1,2]},"properties":{}}]}"#}]); + s.run("service.import", ¶ms).unwrap(); + let layer = &s.map().layers[0]; + assert_eq!(layer.anonymous, crossed); + assert_eq!(layer.connection.is_empty(), crossed); + assert_eq!(header(layer.dataset.source.as_deref().unwrap()).is_none(), crossed || !authenticated); + assert_eq!(layer.dataset.features.len(), 1); + params["group"] = json!(s.map().service_groups[0].id); + s.run("service.refresh", ¶ms).unwrap(); + let layer = &s.map().layers[0]; + assert_eq!(layer.anonymous, crossed); + assert_eq!(layer.connection.is_empty(), crossed); + assert_eq!(header(layer.dataset.source.as_deref().unwrap()).is_none(), crossed || !authenticated); + let saved = s.project.saved_text(None).unwrap().0; + s.run("project.openText", &json!({"text":saved})).unwrap(); + let layer = &s.map().layers[0]; + assert_eq!(layer.anonymous, crossed); + assert_eq!(layer.connection.is_empty(), crossed); + } +} + /// A cloud-optimised GeoTIFF added by address from a saved web-service connection is a raster /// layer read in place that records the connection, as the tile layer it was did, its address /// bound to the connection's login (never carrying the key) and read with it; the login made diff --git a/crates/map/src/lib.rs b/crates/map/src/lib.rs index 4c7cb87bd..7ed3c1151 100644 --- a/crates/map/src/lib.rs +++ b/crates/map/src/lib.rs @@ -23,7 +23,9 @@ pub mod projection; pub mod qgis; pub mod raster; pub mod report; +pub mod slots; pub mod source; +pub use slots::{LayerRole, Slot, SlotAt, SlotPos, TileRole}; pub mod symbol_by; pub mod symbology; pub mod tiles; @@ -2754,6 +2756,222 @@ fn de_reference_scale<'de, D: serde::Deserializer<'de>>(d: D) -> Result Vec> { + let Map { + name: _, + layers, + service_groups, + groups: _, + basemap: _, + tiles, + camera: _, + bookmarks: _, + graticule: _, + projection: _, + popup_mode: _, + label_display: _, + label_lock: _, + label_lock_line_anchors: _, + editing: _, + swipe: _, + view, + thumbnail: _, + animation: _, + time: _, + body: _, + next_layer_id: _, + instance_id: _, + } = self; + let mut out = Vec::new(); + for (group, g) in service_groups.iter().enumerate() { + let ServiceGroup { id: _, name: _, url, kind: _, anonymous: _, expanded: _, children } = g; + out.push(Slot { at: SlotPos::Group { group, child: None }, value: url, address: true, local_path: false }); + for (child, c) in children.iter().enumerate() { + let ServiceChild { layer: _, collection: _, source_url } = c; + out.push(Slot { at: SlotPos::Group { group, child: Some(child) }, value: source_url, address: true, local_path: false }); + } + } + for (tile, t) in tiles.iter().enumerate() { + out.extend(t.output_slots().into_iter().map(|(role, value, local_path)| Slot { + at: SlotPos::Tile { tile, role }, + value, + address: true, + local_path, + })); + } + let view::ViewSettings { + effects: _, + color_vision: _, + wireframe: _, + clip_distance: _, + terrain, + fast: _, + navigator: _, + first_person: _, + } = view; + if let Some(view::Terrain { source, exaggeration: _ }) = terrain { + match source { + view::TerrainSource::Tiles { url, encoding: _ } => { + out.push(Slot { at: SlotPos::Terrain, value: url, address: true, local_path: false }) + } + view::TerrainSource::Layer(_) => {} + } + } + for (layer, l) in layers.iter().enumerate() { + out.extend(l.output_slots().into_iter().map(|(role, value, address, local_path)| Slot { + at: SlotPos::Layer { layer, role }, + value, + address, + local_path, + })); + } + out + } + + /// Resolves only the recorded location; never rediscovers or reclassifies a written value. + pub fn slot_mut(&mut self, at: &SlotPos) -> Option<&mut String> { + match at { + SlotPos::Group { group, child } => { + let g = self.service_groups.get_mut(*group)?; + match child { + Some(c) => Some(&mut g.children.get_mut(*c)?.source_url), + None => Some(&mut g.url), + } + } + SlotPos::Tile { tile, role } => self.tiles.get_mut(*tile)?.output_slot_mut(*role), + SlotPos::Terrain => match &mut self.view.terrain.as_mut()?.source { + view::TerrainSource::Tiles { url, encoding: _ } => Some(url), + view::TerrainSource::Layer(_) => None, + }, + SlotPos::Layer { layer, role } => self.layers.get_mut(*layer)?.output_slot_mut(*role), + } + } +} + +impl Layer { + /// Classifies whole owning fields; a source serving as provenance and a path occurs once. + pub fn output_slots(&self) -> Vec<(LayerRole, &str, bool, bool)> { + let Layer { + id: _, + name: _, + visible: _, + dataset, + renderer: _, + labels: _, + transparency: _, + min_scale: _, + max_scale: _, + definition_query: _, + time: _, + selection: _, + expanded: _, + selectable: _, + graphics: _, + blend: _, + mask: _, + display_filters: _, + aggregation: _, + extrusion: _, + line_strokes: _, + fill_layers: _, + no_fill: _, + stroke_levels: _, + marker_rotation: _, + symbol_by: _, + reference_scale: _, + priority: _, + design: _, + raster, + source, + postgis, + gpkg, + connection: _, + anonymous: _, + instance_id: _, + projected: _, + display: _, + display_rejected: _, + display_levels: _, + projected_extents: _, + projected_vertices: _, + thinned: _, + projected_rev: _, + projected_for: _, + view_memory: _, + geometry_kinds: _, + geometry_rev: _, + diagram: _, + } = self; + let mut out = Vec::new(); + let rustgis_data::Dataset { + name: _, + geometry_type: _, + fields: _, + features: _, + source: provenance, + crs: _, + crs_incomplete: _, + zm_dropped: _, + windowed: _, + } = dataset; + if let Some(v) = provenance { + out.push((LayerRole::Dataset, v.as_str(), true, paths::is_local(v))); + } + if let Some(s) = source { + out.extend(s.output_slots()); + } + if let Some(r) = raster { + out.extend(r.output_slots()); + } + if let Some(g) = gpkg { + out.push((LayerRole::GpkgPath, g.path.as_str(), false, paths::is_local(&g.path))); + } + if let Some(pg) = postgis { + out.push((LayerRole::PostgisEditDatabase, pg.origin.database.as_str(), true, false)); + } + out + } + + /// Allocation-free resolution, valid while the layer's structure is held unchanged. + pub fn output_slot_mut(&mut self, role: LayerRole) -> Option<&mut String> { + match role { + LayerRole::Dataset => self.dataset.source.as_mut(), + LayerRole::Streamed + | LayerRole::GdalPath + | LayerRole::DuckDbDatabase + | LayerRole::PostgisDatabase + | LayerRole::MysqlDatabase + | LayerRole::SqlServerDatabase => self.source.as_mut()?.output_slot_mut(role), + LayerRole::Raster | LayerRole::ZarrStore => self.raster.as_mut()?.output_slot_mut(role), + LayerRole::GpkgPath => Some(&mut self.gpkg.as_mut()?.path), + LayerRole::PostgisEditDatabase => Some(&mut self.postgis.as_mut()?.origin.database), + } + } +} + +impl Map { + /// Bridge identity is independent of the address value; duplicate ids remain ambiguous. + pub fn slot_identity(&self, at: &SlotPos) -> Option { + match at { + SlotPos::Group { group, child } => { + let g = self.service_groups.get(*group)?; + Some(SlotAt::Group { + id: g.id, + child: match child { + Some(i) => Some(g.children.get(*i)?.layer), + None => None, + }, + }) + } + SlotPos::Tile { tile, role } => Some(SlotAt::Tile { id: self.tiles.get(*tile)?.id.clone(), role: *role }), + SlotPos::Terrain => Some(SlotAt::Terrain), + SlotPos::Layer { layer, role } => Some(SlotAt::Layer { id: self.layers.get(*layer)?.id, role: *role }), + } + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/map/src/raster.rs b/crates/map/src/raster.rs index 8d9235125..ec5cbab55 100644 --- a/crates/map/src/raster.rs +++ b/crates/map/src/raster.rs @@ -1551,6 +1551,53 @@ impl RasterLayer { } } +impl RasterLayer { + /// Only owning address/path strings are offered; embedded bytes and render state stay opaque. + pub(crate) fn output_slots(&self) -> Vec<(crate::LayerRole, &str, bool, bool)> { + let RasterLayer { + path, + file_name: _, + embedded, + subdataset: _, + crs: _, + time_zarr, + placement: _, + style: _, + data: _, + error: _, + masked: _, + opening: _, + in_place: _, + restyle: _, + look: _, + pixels_said: _, + error_said: _, + failed_at: _, + failures: _, + lost_said: _, + look_lost: _, + dropped: _, + } = self; + let mut out = Vec::new(); + if let Some(p) = path { + out.push((crate::LayerRole::Raster, p.as_str(), true, embedded.is_none() && crate::paths::is_local(p))); + } + if let Some(ZarrTime { store, variable: _, index: _, dim: _, offset: _, dates: _, crs: _, shown: _ }) = time_zarr { + out.push((crate::LayerRole::ZarrStore, store.as_str(), true, crate::paths::is_local(store))); + } + out + } + + /// Looks up a prepared address/path field without allocation. + pub(crate) fn output_slot_mut(&mut self, role: crate::LayerRole) -> Option<&mut String> { + match role { + crate::LayerRole::Raster => self.path.as_mut(), + crate::LayerRole::ZarrStore => Some(&mut self.time_zarr.as_mut()?.store), + _ => None, + } + } +} + #[cfg(test)] mod tests { use rustgis_data::raster::{Crs, GeoTransform}; diff --git a/crates/map/src/slots.rs b/crates/map/src/slots.rs new file mode 100644 index 000000000..2b73c5851 --- /dev/null +++ b/crates/map/src/slots.rs @@ -0,0 +1,50 @@ +//! Stable leaf locations for a synchronous output transaction. Resolving a location neither +//! discovers slots nor changes structure; addresses and local paths share a single location. + +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub enum SlotPos { + Group { group: usize, child: Option }, + Tile { tile: usize, role: TileRole }, + Terrain, + Layer { layer: usize, role: LayerRole }, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum LayerRole { + Dataset, + Streamed, + Raster, + ZarrStore, + GdalPath, + DuckDbDatabase, + GpkgPath, + PostgisEditDatabase, + PostgisDatabase, + MysqlDatabase, + SqlServerDatabase, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum TileRole { + Url, + TimeUrl, + TimeAt(usize), + Legend(usize), +} + +/// A slot's identity across a bridge exchange. Positions are deliberately absent, so reordering +/// layers preserves identity while changing their owner, role or id does not. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub enum SlotAt { + Layer { id: u32, role: LayerRole }, + Tile { id: String, role: TileRole }, + Group { id: u32, child: Option }, + Terrain, +} + +pub struct Slot<'a> { + pub at: SlotPos, + pub value: &'a str, + pub address: bool, + pub local_path: bool, +} diff --git a/crates/map/src/source.rs b/crates/map/src/source.rs index 4da28b4b5..0d4270e9a 100644 --- a/crates/map/src/source.rs +++ b/crates/map/src/source.rs @@ -34,6 +34,8 @@ pub const VIEW_LIMIT: usize = if cfg!(target_arch = "wasm32") { 100_000 } else { /// The most features a view reads from a file by its web address (each one is fetched). pub const REMOTE_VIEW_LIMIT: usize = if cfg!(target_arch = "wasm32") { 30_000 } else { 60_000 }; +const RECONNECT_REQUIRED: &str = "this database source came from a peer; reconnect explicitly before reading it"; + /// A streamed layer's source. #[derive(Clone, Default, Serialize, Deserialize)] pub struct SourceLayer { @@ -93,6 +95,9 @@ pub struct SourceLayer { /// The last open failed because the server refused the login (no password given this session, or a wrong one). #[serde(skip)] pub login: bool, + /// Runtime-only: imported source selection awaits an explicit reconnect or exact peer retention. + #[serde(skip)] + reconnect_required: bool, /// The most features a view reads (0 = [`VIEW_LIMIT`]). #[serde(skip)] pub limit: usize, @@ -688,6 +693,10 @@ impl SourceLayer { /// Opens the source again (after a project loads). A source by its web address is opened off /// the frame: `Ok` while it is (see [`SourceLayer::is_pending`]), open once it has landed. pub fn open(&mut self) -> Result<(), String> { + if self.reconnect_required { + self.error = Some(RECONNECT_REQUIRED.into()); + return Err(RECONNECT_REQUIRED.into()); + } if let Some(url) = self.path.clone().filter(|p| rustgis_data::source::is_remote_name(p)) { return self.open_remote(&url); } @@ -927,6 +936,25 @@ impl SourceLayer { self.pending.is_some() || self.opening.is_some() } + /// Suspends automatic reads without changing the serialized source definition or any password. + pub fn require_reconnect(&mut self) { + self.reconnect_required = true; + self.handle = None; + self.opening = None; + self.pending = None; + self.error = Some(RECONNECT_REQUIRED.into()); + } + + /// Whether the source still needs its owner to authorize a reconnect. + pub fn reconnect_required(&self) -> bool { + self.reconnect_required + } + + /// Releases a source after its owner explicitly reconnects or validates exact peer retention. + pub fn allow_reconnect(&mut self) { + self.reconnect_required = false; + } + /// Opens the source afresh, forgetting a failed or never-started open (and the window read /// from the old one): a source by its web address read again once what signs it has changed /// (its address, or the login behind it). Its symbology and place in the project stay. @@ -1211,6 +1239,89 @@ pub fn generation() -> u64 { GENERATION.load(Ordering::Relaxed) } +impl SourceLayer { + /// Persisted source address and local driver paths; handles and embedded data are opaque. + pub(crate) fn output_slots(&self) -> Vec<(crate::LayerRole, &str, bool, bool)> { + let SourceLayer { + path, + file_name: _, + table: _, + shapefile_crs: _, + embedded, + handle: _, + window: _, + render_lod: _, + pending: _, + projection: _, + ready_cache: _, + byte_budget: _, + estimated_row_bytes: _, + full_read: _, + saved_selection: _, + window_fields: _, + query: _, + opening: _, + error: _, + login: _, + reconnect_required: _, + limit: _, + reads: _, + gdal, + duckdb, + postgis, + mysql, + sqlserver, + } = self; + use crate::{LayerRole as R, paths}; + let mut out = Vec::new(); + if let Some(p) = path { + out.push(( + R::Streamed, + p.as_str(), + true, + embedded.is_none() + && gdal.is_none() + && duckdb.is_none() + && postgis.is_none() + && mysql.is_none() + && sqlserver.is_none() + && paths::is_local(p), + )); + } + if let Some(g) = gdal { + out.push((R::GdalPath, g.path.as_str(), false, paths::is_local(&g.path))); + } + if let Some(d) = duckdb + && let Some(p) = &d.database + { + out.push((R::DuckDbDatabase, p.as_str(), true, paths::is_local(p))); + } + if let Some(pg) = postgis { + out.push((R::PostgisDatabase, pg.database.as_str(), true, false)); + } + if let Some(my) = mysql { + out.push((R::MysqlDatabase, my.database.as_str(), true, false)); + } + if let Some(ms) = sqlserver { + out.push((R::SqlServerDatabase, ms.database.as_str(), true, false)); + } + out + } + + /// Resolves a prepared field without examining its written contents. + pub(crate) fn output_slot_mut(&mut self, role: crate::LayerRole) -> Option<&mut String> { + match role { + crate::LayerRole::Streamed => self.path.as_mut(), + crate::LayerRole::GdalPath => Some(&mut self.gdal.as_mut()?.path), + crate::LayerRole::DuckDbDatabase => self.duckdb.as_mut()?.database.as_mut(), + crate::LayerRole::PostgisDatabase => Some(&mut self.postgis.as_mut()?.database), + crate::LayerRole::MysqlDatabase => Some(&mut self.mysql.as_mut()?.database), + crate::LayerRole::SqlServerDatabase => Some(&mut self.sqlserver.as_mut()?.database), + _ => None, + } + } +} + #[cfg(test)] mod tests { /// Reopening this same layer must invalidate populated decisions even without `reset`. diff --git a/crates/map/src/tiles.rs b/crates/map/src/tiles.rs index 4932d425d..77e506993 100644 --- a/crates/map/src/tiles.rs +++ b/crates/map/src/tiles.rs @@ -570,6 +570,66 @@ pub fn host_of(url: &str) -> Option<&str> { Some(host.split(':').next().unwrap_or(host)) } +impl TileLayer { + /// All persisted tile addresses, with the archive URL also classified as a local path. + pub(crate) fn output_slots(&self) -> Vec<(crate::TileRole, &str, bool)> { + let TileLayer { + name: _, + kind: _, + url, + bounds: _, + subdomains: _, + min_zoom: _, + max_zoom: _, + tile_size: _, + tms: _, + attribution: _, + visible: _, + opacity: _, + overlay: _, + over: _, + blend: _, + id: _, + layers: _, + style_layers: _, + hidden_layers: _, + time_url, + time_at: _, + time_urls, + time_hidden: _, + error: _, + feature_info: _, + legends, + connection: _, + anonymous: _, + expanded: _, + selected: _, + single_image: _, + } = self; + use crate::TileRole as R; + let mut out = vec![(R::Url, url.as_str(), self.archive_file().is_some())]; + if let Some(v) = time_url { + out.push((R::TimeUrl, v.as_str(), false)); + } + out.extend(time_urls.iter().enumerate().map(|(i, (_, v))| (R::TimeAt(i), v.as_str(), false))); + out.extend(legends.iter().enumerate().map(|(i, v)| { + let rustgis_data::ogc::Legend { url, min_scale: _, max_scale: _ } = v; + (R::Legend(i), url.as_str(), false) + })); + out + } + + /// Resolves a fixed location, independent of the address's current spelling. + pub(crate) fn output_slot_mut(&mut self, role: crate::TileRole) -> Option<&mut String> { + match role { + crate::TileRole::Url => Some(&mut self.url), + crate::TileRole::TimeUrl => self.time_url.as_mut(), + crate::TileRole::TimeAt(i) => Some(&mut self.time_urls.get_mut(i)?.1), + crate::TileRole::Legend(i) => Some(&mut self.legends.get_mut(i)?.url), + } + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/tiles/src/arcgis_auth.rs b/crates/tiles/src/arcgis_auth.rs index 98c38e3f3..24795839a 100644 --- a/crates/tiles/src/arcgis_auth.rs +++ b/crates/tiles/src/arcgis_auth.rs @@ -454,7 +454,7 @@ pub fn adopt_url_token(url: &str) -> Result { .split('&') .filter(|p| { let (k, v) = p.split_once('=').unwrap_or((p, "")); - if k.eq_ignore_ascii_case("token") { + if rustgis_data::address::normalized_name(k).as_deref() == Some("token") { // A token is pasted, not form-encoded: its `+` stays a `+`. token = Some(decode_with(v, false)); false diff --git a/crates/tiles/src/cache.rs b/crates/tiles/src/cache.rs index 481ef2503..1e404fbbf 100644 --- a/crates/tiles/src/cache.rs +++ b/crates/tiles/src/cache.rs @@ -145,7 +145,8 @@ pub fn clear_fetcher() { /// on whichever thread is starting them already, so one test's fetch may run on another's thread, /// later. Tests that end a login take turns too, as that empties the cache ([`clear`]), and so do /// tests that wait for or plant what the cache holds (an archive read, a style resolved), which -/// that would drop. The turn puts the native fetcher back when it ends. +/// that would drop. Tests that inspect clean failure text also take turns: a concurrent login +/// can redact even a single letter from that text. The turn puts the native fetcher back when it ends. #[cfg(test)] pub(crate) fn turn() -> Turn { static TURN: Mutex<()> = Mutex::new(()); @@ -267,13 +268,12 @@ pub fn generation() -> u64 { let now = alarm_at(Instant::now()); let next = { let mut a = alarms().lock().unwrap_or_else(|e| e.into_inner()); - if a.first().is_some_and(|at| *at <= now) { + let (over, next) = take_due(&mut a, now); + if over { // Every wait over is one draw: the generation goes up once for them all. - let later = a.split_off(&now.saturating_add(1)); - *a = later; GENERATION.fetch_add(1, Ordering::Relaxed); } - a.first().copied() + next }; if let Some(at) = next { ring(Duration::from_millis(at.saturating_sub(now))); @@ -281,6 +281,16 @@ pub fn generation() -> u64 { GENERATION.load(Ordering::Relaxed) } +/// Consumes every elapsed deadline together and returns the next one to rearm. Keeping the +/// clock as an input lets the ordering contract be tested without scheduler deadlines. +fn take_due(alarms: &mut std::collections::BTreeSet, now: u64) -> (bool, Option) { + let over = alarms.first().is_some_and(|at| *at <= now); + if over { + *alarms = alarms.split_off(&now.saturating_add(1)); + } + (over, alarms.first().copied()) +} + /// How long until the earliest wait pending; none when there is none. #[cfg(test)] pub(crate) fn next_alarm() -> Option { @@ -683,7 +693,7 @@ fn fetch_tile(url: String, host: String, want: Want, omit: &'static [u16]) { Entry::Ready(Arc::new(p), size) } Err(e) => { - log::debug!("{}", crate::logins::scrub(&format!("{}: {e}", answer.key))); + log::debug!("{}", crate::logins::scrub_error(&format!("{}: {e}", answer.key))); // Refused until a time its server gave: the map draws again then. if let Some(wait) = retry_after(&e) { notify_at(Instant::now() + wait); @@ -867,7 +877,7 @@ impl<'a> RequestDiagnostics<'a> { let url = crate::logins::inner(url); let url = if rustgis_data::login_address::is_wrapper(url) { "[address omitted]" } else { url }; let clean = |text: &str| { - if carried || text.contains(rustgis_data::login_address::PREFIX) { crate::logins::scrub(text) } else { text.to_string() } + if carried || text.contains(rustgis_data::login_address::PREFIX) { crate::logins::scrub_error(text) } else { text.to_string() } }; // A status the caller expects is an answer, not a failure for the history; a caller // that treats the refusal as an error says so itself. One bounded record per server says @@ -1617,7 +1627,9 @@ mod native { } Err("too many redirects".into()) })(); - diagnostics.record(answer).map_err(|failure| if carried_login { crate::logins::scrub(&failure.message) } else { failure.message }) + diagnostics + .record(answer) + .map_err(|failure| if carried_login { crate::logins::scrub_error(&failure.message) } else { failure.message }) } /// POSTs a JSON document on a thread of its own. @@ -1673,7 +1685,9 @@ mod native { crate::logins::check(url).map_err(str::to_string)?; Ok(buf) })(); - diagnostics.record(answer).map_err(|failure| if carried_login { crate::logins::scrub(&failure.message) } else { failure.message }) + diagnostics + .record(answer) + .map_err(|failure| if carried_login { crate::logins::scrub_error(&failure.message) } else { failure.message }) } /// Downloads on a thread of its own, so a large file doesn't hold up the tiles. @@ -1776,7 +1790,7 @@ mod native { /// [`get_detailed`], a status in `omit` not recorded as a failure. pub(super) fn get_detailed_omitting(agent: &ureq::Agent, url: &str, omit: &'static [u16]) -> Result, Failure> { fetch_answer(agent, url, true, omit).map_err(|f| Failure { - url: if url.contains(rustgis_data::login_address::PREFIX) { crate::logins::scrub(url) } else { url.to_string() }, + url: if url.contains(rustgis_data::login_address::PREFIX) { crate::logins::scrub_error(url) } else { url.to_string() }, ..f }) } @@ -1995,8 +2009,8 @@ mod native { let result = crate::logins::check(super::resource_of(url)).map_err(Failure::from).and(result); result.map_err(|mut f| { if carried_login { - f.message = crate::logins::scrub(&f.message); - f.url = crate::logins::scrub(&f.url); + f.message = crate::logins::scrub_error(&f.message); + f.url = crate::logins::scrub_error(&f.url); } f }) @@ -2043,6 +2057,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn network_diagnostics_capture_background_failures_once() { + let _turn = turn(); use rustgis_data::diagnostics::{History, NetworkKind}; let history = History::default(); let _scope = history.enter(); @@ -2063,6 +2078,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn network_diagnostics_interleaved_sessions_keep_their_requests_private() { + let _turn = turn(); use rustgis_data::diagnostics::History; use std::io::{BufRead, BufReader, Write}; let first = History::default(); @@ -2123,6 +2139,7 @@ mod tests { #[test] fn network_diagnostics_explicit_capture_outlives_dispatch_scope_not_owner() { + let _turn = turn(); use rustgis_data::diagnostics::History; let first = History::default(); let second = History::default(); @@ -2149,6 +2166,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn network_diagnostics_redact_downloads_without_retaining_bodies() { + let _turn = turn(); use rustgis_data::diagnostics::History; let history = History::default(); let _scope = history.enter(); @@ -2199,6 +2217,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn network_diagnostics_skip_gone_map_tiles_and_keep_server_errors() { + let _turn = turn(); let history = rustgis_data::diagnostics::History::default(); let _scope = history.enter(); let answer = |status: &str| format!("HTTP/1.1 {status}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"); @@ -2220,10 +2239,11 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn network_diagnostics_keep_headers_after_a_credential_header() { + let _turn = turn(); let history = rustgis_data::diagnostics::History::default(); let _scope = history.enter(); let base = serve(vec![ - "HTTP/1.1 403 Forbidden\r\nWWW-Authenticate: Basic realm=\"private-realm\"\r\nRetry-After: 30\r\ncf-ray: 8a1-BOG\r\nX-Amzn-RequestId: amzn-1155\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + "HTTP/1.1 403 Forbidden\r\nWWW-Authenticate: Basic realm=\"private-realm\"\r\nRetry-After: 30\r\ncf-ray: 8a1-BOG\r\nX-Amzn-RequestId: amzn-1155\r\nX-Amz-Cf-Id: trace-1155\r\nX-Amzn-Waf-Action: block\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" .into(), ]); let url = format!("{base}/refused"); @@ -2231,7 +2251,9 @@ mod tests { let records = network_records(&history, &url); assert_eq!(records.len(), 1); let details = &records[0].details; - for kept in ["retry-after: 30", "cf-ray: 8a1-BOG", "x-amzn-requestid: amzn-1155"] { + for kept in + ["retry-after: 30", "cf-ray: 8a1-BOG", "x-amzn-requestid: amzn-1155", "x-amz-cf-id: trace-1155", "x-amzn-waf-action: block"] + { assert!(details.contains(kept), "{kept} in {details:?}"); } assert!(!details.contains("private-realm"), "{details:?}"); @@ -2240,6 +2262,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn network_diagnostics_skip_routine_tile_404s_and_keep_other_statuses() { + let _turn = turn(); use rustgis_data::diagnostics::{History, Severity}; let history = History::default(); let _scope = history.enter(); @@ -2298,6 +2321,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn network_diagnostics_keep_observed_status_on_blocking_body_failure() { + let _turn = turn(); use rustgis_data::diagnostics::{History, NetworkKind}; let history = History::default(); let _scope = history.enter(); @@ -2315,6 +2339,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn network_diagnostics_preserve_connection_and_timeout_causes() { + let _turn = turn(); use rustgis_data::diagnostics::{History, NetworkKind}; let history = History::default(); let _scope = history.enter(); @@ -2349,6 +2374,7 @@ mod tests { /// history keeps no record of it. #[test] fn network_diagnostics_skip_failed_local_reads() { + let _turn = turn(); let history = rustgis_data::diagnostics::History::default(); let _scope = history.enter(); native::get(&native::agent(), "local-file:/nonexistent-1155.pmtiles", &[]).unwrap_err(); @@ -2357,6 +2383,7 @@ mod tests { #[test] fn network_diagnostics_ignore_cancellation_and_keep_browser_causes_truthful() { + let _turn = turn(); use rustgis_data::diagnostics::{History, NetworkKind}; let history = History::default(); let _scope = history.enter(); @@ -2384,6 +2411,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn public_downloads_refuse_names_that_resolve_locally() { + let _turn = turn(); let err = download_public_within("http://localhost:9/doc.json", 1024, std::time::Duration::from_secs(5)).unwrap_err(); assert!(err.contains("resolves to this computer or the local network"), "{err}"); } @@ -2411,6 +2439,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn downloads_stop_at_the_callers_budget() { + let _turn = turn(); for framing in ["Content-Length: 4\r\n", "", "Transfer-Encoding: chunked\r\n"] { let body = if framing.starts_with("Transfer") { "4\r\ntile\r\n0\r\n\r\n" } else { "tile" }; for (limit, ok) in [(0, false), (3, false), (4, true), (5, true)] { @@ -2431,6 +2460,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn download_budget_also_bounds_decompressed_bytes() { + let _turn = turn(); let data = vec![b'x'; 4096]; // The ZIP writer supplies the CRC-32 for a valid, generated gzip trailer. let zip = rustgis_data::zip::write(&[("body".into(), data.clone())]).unwrap(); @@ -2530,24 +2560,14 @@ mod tests { /// its elevation source waits), so the later is kept, and the app asked for it then. #[test] fn a_later_wait_isnt_lost_behind_an_earlier_one() { - let _alarm = super::alarm_turn(); - let asked = std::sync::Arc::new(Mutex::new(Vec::new())); - let noted = asked.clone(); - super::set_alarm(move |after| noted.lock().unwrap_or_else(|e| e.into_inner()).push(after)); - super::clear_alarms(); - let before = generation(); - notify_at(Instant::now() + Duration::from_millis(20)); - notify_at(Instant::now() + Duration::from_millis(150)); - std::thread::sleep(std::time::Duration::from_millis(50)); - let earlier = generation(); - assert!(earlier > before, "the earlier wait over"); - // The later is still pending, and asked of the app as the earlier was over. - assert!(super::next_alarm().is_some_and(|d| d <= Duration::from_millis(150)), "{:?}", super::next_alarm()); - let later = |asked: &Vec| asked.iter().any(|d| (Duration::from_millis(30)..=Duration::from_millis(150)).contains(d)); - assert!(later(&asked.lock().unwrap_or_else(|e| e.into_inner())), "{:?}", asked.lock().unwrap_or_else(|e| e.into_inner())); - std::thread::sleep(std::time::Duration::from_millis(150)); - assert!(generation() > earlier, "the later wait over"); - super::clear_alarm(); + let mut alarms = std::collections::BTreeSet::from([20, 150]); + assert_eq!(super::take_due(&mut alarms, 10), (false, Some(20))); + assert_eq!(super::take_due(&mut alarms, 50), (true, Some(150)), "rearm the later wait after the first expires"); + assert_eq!(super::take_due(&mut alarms, 100), (false, Some(150))); + assert_eq!(super::take_due(&mut alarms, 200), (true, None)); + assert_eq!(super::take_due(&mut alarms, 300), (false, None), "expired deadlines are consumed once"); + let mut delayed = std::collections::BTreeSet::from([20, 150]); + assert_eq!(super::take_due(&mut delayed, 200), (true, None), "a delayed poll correctly consumes both waits"); } /// A draw before a wait is over asks the app for it again: the app's request lasts one frame, @@ -3079,6 +3099,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn the_length_asked_for_follows_redirects_and_keeps_refusals() { + let _turn = turn(); let no_suffix = "HTTP/1.1 501 Not Implemented\r\nContent-Length: 0\r\nConnection: close\r\n\r\n".to_string(); let agent = native::agent(); let base = serve(vec![ @@ -3113,6 +3134,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn a_refused_download_says_who_refused_it() { + let _turn = turn(); let page = "Attention Required! | CloudflareSorry, you have been blocked"; let blocked = format!( "HTTP/1.1 403 Forbidden\r\nServer: cloudflare\r\nCF-RAY: a4c9-BOG\r\nContent-Type: text/html\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{page}", @@ -3134,6 +3156,7 @@ mod tests { #[cfg(not(target_arch = "wasm32"))] #[test] fn follows_only_allowed_redirects() { + let _turn = turn(); let redirect = |to: &str| format!("HTTP/1.1 302 Found\r\nLocation: {to}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"); let ok = "HTTP/1.1 200 OK\r\nContent-Length: 4\r\nConnection: close\r\n\r\ntile".to_string(); let agent = native::agent(); diff --git a/crates/tiles/src/cog.rs b/crates/tiles/src/cog.rs index f1dc6fe9a..09c867cf9 100644 --- a/crates/tiles/src/cog.rs +++ b/crates/tiles/src/cog.rs @@ -404,7 +404,7 @@ fn with_info(url: &str, then: Box, String>) + Send>) let r = crate::logins::check(&key) .map_err(str::to_string) .and(r) - .map_err(|e| if rustgis_data::login_address::is_wrapper(&key) { crate::logins::scrub(&e) } else { e }); + .map_err(|e| if rustgis_data::login_address::is_wrapper(&key) { crate::logins::scrub_error(&e) } else { e }); let waiting = { let mut i = infos().lock().unwrap_or_else(|e| e.into_inner()); let next = match &r { diff --git a/crates/tiles/src/cog/raster.rs b/crates/tiles/src/cog/raster.rs index 0555506c6..3193b350a 100644 --- a/crates/tiles/src/cog/raster.rs +++ b/crates/tiles/src/cog/raster.rs @@ -531,6 +531,7 @@ mod tests { /// the last `usize` is refused by number, not overflowed. #[test] fn a_window_of_the_band_past_the_last_usize_is_refused() { + let _turn = crate::cache::turn(); let file = "test://past-last-band.tif"; let info = info_of(example_with(&Example { size: 64, tile: 16, ..Example::default() })); super::super::infos().lock().unwrap().insert(file.to_string(), super::super::State::Ready(Arc::new(info))); @@ -581,6 +582,8 @@ mod tests { /// asked for again when it is drawn next; one its server refused is, until its wait is over. #[test] fn a_dropped_tile_isnt_remembered_as_failed() { + let _turn = crate::cache::turn(); + let _alarm = crate::cache::alarm_turn(); let key = |c: u32| ("test://dropped.tif".to_string(), 0usize, c); tile_landed(key(1), &Err(crate::cache::DROPPED.to_string())); assert!(!failed().lock().unwrap().contains_key(&key(1))); diff --git a/crates/tiles/src/failure.rs b/crates/tiles/src/failure.rs index 0f5c94eef..58cbf0b3a 100644 --- a/crates/tiles/src/failure.rs +++ b/crates/tiles/src/failure.rs @@ -68,9 +68,9 @@ impl Failure { /// the browser's `HTTP 403`). pub fn new(url: &str, message: impl Into) -> Self { let message = message.into(); - let message = if message.contains(rustgis_data::login_address::PREFIX) { crate::logins::scrub(&message) } else { message }; let reason = crate::cache::http_status(&message).filter(|s| *s >= 400).map_or(Reason::Other, Reason::Status); - let url = if url.contains(rustgis_data::login_address::PREFIX) { crate::logins::scrub(url) } else { url.to_string() }; + let message = crate::logins::scrub_error(&message); + let url = if url.contains(rustgis_data::login_address::PREFIX) { crate::logins::scrub_error(url) } else { url.to_string() }; Failure { url, message, reason, answer: None } } @@ -375,6 +375,7 @@ mod tests { #[test] fn a_cloudflare_block_is_named_with_its_page_title() { + let _turn = crate::cache::turn(); let body = b"Attention Required! | CloudflareSorry, you have been blocked"; let f = Failure::answered( "https://example.org/MapServer?f=json", @@ -462,6 +463,7 @@ mod tests { /// read, say nothing more. #[test] fn retry_after_on_429_and_503() { + let _turn = crate::cache::turn(); let base = || "http status: 503".to_string(); assert_eq!(with_retry_after(base(), 503, Some("30")), "http status: 503 (retry after 30 s)"); assert_eq!(with_retry_after("HTTP 429".into(), 429, Some(" 2 ")), "HTTP 429 (retry after 2 s)"); diff --git a/crates/tiles/src/logins.rs b/crates/tiles/src/logins.rs index b5d916f4b..7c92a3498 100644 --- a/crates/tiles/src/logins.rs +++ b/crates/tiles/src/logins.rs @@ -101,6 +101,8 @@ pub fn activate(origin: &str, owner: &str, method: Method, value: &str, insecure activate_with_random(origin, owner, method, value, insecure, fill_random) } +/// Validates the login before requesting entropy, then installs a fresh binding and its active +/// output redactions together; failed validation or entropy leaves the registry unchanged. fn activate_with_random( origin: &str, owner: &str, @@ -129,6 +131,15 @@ fn activate_with_random( && !payload.is_empty() { add_redactions(&mut redactions, payload); + // A server may echo the raw Basic password rather than its wire encoding. The complete + // header was bounded above, so decoding cannot allocate beyond that input's size. + if let Ok(decoded) = base64::engine::general_purpose::STANDARD.decode(payload) + && let Ok(decoded) = std::str::from_utf8(&decoded) + && let Some((_, password)) = decoded.split_once(':') + && !password.is_empty() + { + add_redactions(&mut redactions, password); + } } let mut entries = entries(); // A token is the only thing that scopes a bound address to its login, so it is random. A @@ -266,8 +277,24 @@ fn append_percent_encoded(result: &mut String, value: &str) { /// prevents overlapping values or a credential resembling `[redacted]` from leaking secrets. pub fn scrub(text: &str) -> String { let entries = entries(); + if entries.is_empty() && !text.contains(rustgis_data::login_address::PREFIX) { + return text.to_string(); + } let mut patterns: Vec<_> = entries.values().flat_map(|entry| &entry.redactions).collect(); patterns.sort_unstable_by_key(|pattern| std::cmp::Reverse(pattern.value.expose().len())); + scrub_patterns(text, &patterns) +} + +/// Cleans generated error text before active values can obscure its address syntax. +/// Runtime request bindings and successful data values must not use this output projection. +pub fn scrub_error(text: &str) -> String { + let unbound = scrub_patterns(text, &[]); + let (public, _) = rustgis_data::address::scrub_addresses(&unbound); + scrub(&public) +} + +/// Removes runtime capability headers and the supplied active patterns in one pass. +fn scrub_patterns(text: &str, patterns: &[&Redaction]) -> String { // Strip capabilities even after revocation, when there are no active credential patterns. let mut result = String::with_capacity(text.len()); let mut rest = text; @@ -526,10 +553,14 @@ mod tests { #[test] fn scrubs_header_and_basic_payload() { let _turn = crate::cache::turn(); - let value = basic("scrub-user", "scrub-password"); + let password = "scrub-password: /é"; + let value = basic("scrub-user", password); let binding = activate("https://scrub-login.example", "scrub", Method::Header("Authorization".into()), &value, false).unwrap(); let payload = value.strip_prefix("Basic ").unwrap(); assert_eq!(scrub(&format!("{value} {payload} {}", percent_encode(&value))), "[redacted] [redacted] [redacted]"); + assert_eq!(scrub(&format!("server echoed {password} {}", percent_encode(password))), "server echoed [redacted] [redacted]"); + assert_eq!(scrub("scrub-user"), "scrub-user"); + assert_eq!(scrub("a clean server message"), "a clean server message"); let url = bind(&binding, "https://scrub-login.example/tile").unwrap(); assert!(!scrub(&format!("failed: {url}")).contains(rustgis_data::login_address::PREFIX)); assert_eq!(format!("{:?}", apply(&url).unwrap().unwrap()), "Applied()"); diff --git a/crates/tiles/src/pmtiles.rs b/crates/tiles/src/pmtiles.rs index 1176b8548..8a5cb2dc5 100644 --- a/crates/tiles/src/pmtiles.rs +++ b/crates/tiles/src/pmtiles.rs @@ -512,7 +512,7 @@ fn finish(archive: &str, r: Result) { .map_err(str::to_string) .and(r) .map(Arc::new) - .map_err(|e| if rustgis_data::login_address::is_wrapper(archive) { crate::logins::scrub(&e) } else { e }); + .map_err(|e| if rustgis_data::login_address::is_wrapper(archive) { crate::logins::scrub_error(&e) } else { e }); let waiting = { let mut all = archives().lock().unwrap_or_else(|e| e.into_inner()); let old = match &r { diff --git a/crates/ui-egui/src/assistantpane.rs b/crates/ui-egui/src/assistantpane.rs index 3f5e04f14..8de0d33b2 100644 --- a/crates/ui-egui/src/assistantpane.rs +++ b/crates/ui-egui/src/assistantpane.rs @@ -1790,6 +1790,17 @@ fn inline(ui: &Ui, text: &str, t: &Tokens, size: f32, heading: bool) -> egui::te job } +/// A peer's project cannot inherit a pending approval, provider reply or queued user action. +pub(crate) fn peer_replaced(app: &mut RustGisApp) { + if let Some(agent) = &mut app.assistant.agent { + agent.project_replaced(); + } + app.assistant.inflight = None; + app.assistant.chunks = None; + app.assistant.queued = None; + app.assistant.always = false; +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/ui-egui/src/autosave.rs b/crates/ui-egui/src/autosave.rs index 513601a91..b56ab561d 100644 --- a/crates/ui-egui/src/autosave.rs +++ b/crates/ui-egui/src/autosave.rs @@ -43,6 +43,9 @@ pub struct Recovery { /// Copies written before it was known read as `false`. #[serde(default, skip_serializing_if = "std::ops::Not::not")] pub quit: bool, + /// Counted output notices shown when this copy is recovered. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub notices: Vec, } /// Where a shell keeps the recovery copy. Writing and clearing may happen in the background, but @@ -131,7 +134,7 @@ fn now_secs() -> u64 { /// a lossy spelling of the name. fn about(app: &RustGisApp) -> Recovery { let path = app.session.path.as_ref().and_then(|p| std::path::absolute(p).unwrap_or_else(|_| p.clone()).to_str().map(str::to_string)); - Recovery { name: app.session.project.name.clone(), path, time: now_secs(), quit: false } + Recovery { name: app.session.project.name.clone(), path, time: now_secs(), quit: false, notices: Vec::new() } } /// Reads the interval preference and the copy an earlier run left (from [`RustGisApp::with_services`]). @@ -290,6 +293,9 @@ pub(crate) fn recover(app: &mut RustGisApp, p: &Value) -> Result next_offer(app); let text = crate::i18n::fmt("Recovered unsaved work in {name}. Save to keep it.", &[("name", &app.session.project.name)]); app.session.notify(text, false); + for notice in offered.notices { + app.session.notify(notice, false); + } // Other copies wait for the next start: recovering one now would replace this work. let waiting = waiting(app); if waiting > 0 { @@ -667,8 +673,9 @@ mod folder { /// Writes this run's copy: the description line, then the project's text, in one atomic /// write, so a stop part way leaves the previous copy whole. fn write(dir: &Path, project: Project, about: &Recovery) -> Result<(), String> { - let text = project.recovery_text()?; - let header = serde_json::to_string(about).map_err(|e| e.to_string())?; + let (text, report) = project.recovery_text_with_report()?; + let about = Recovery { notices: report.notices(), ..about.clone() }; + let header = serde_json::to_string(&about).map_err(|e| e.to_string())?; let mut bytes = Vec::with_capacity(header.len() + 1 + text.len()); bytes.extend_from_slice(header.as_bytes()); bytes.push(b'\n'); diff --git a/crates/ui-egui/src/connectiontree.rs b/crates/ui-egui/src/connectiontree.rs index 25af0ba3d..55805112d 100644 --- a/crates/ui-egui/src/connectiontree.rs +++ b/crates/ui-egui/src/connectiontree.rs @@ -1038,13 +1038,23 @@ mod tests { let mut h = harness(); let api = "https://x.org/collections"; let tiled = r#"{"collections": [{"id": "relief", "links": [{"rel": "tilesets-map", "href": "https://x.org/relief/map/tiles"}]}]}"#; + let opened = h.state().session.opened; let step = |fallback: &str, origin: Option<&str>| { let slot = Slot::default(); *slot.lock().unwrap() = Some(Err("unreachable".to_string())); let params = json!({"url": api, "layer": "relief (map tiles)", "text": tiled, "name": "Relief"}); - let resolve = - crate::TilesResolve { params, list: "l".into(), doc: "l".into(), read: 0, vector: false, fallback: fallback.into(), slot }; - crate::ServiceStep::Tiles { resolve, dialog: false, origin: origin.map(str::to_string) } + let resolve = crate::TilesResolve { + params, + list: "l".into(), + doc: "l".into(), + crossed_origin: false, + read: 0, + withheld_addresses: 0, + vector: false, + fallback: fallback.into(), + slot, + }; + crate::ServiceStep::Tiles { resolve, dialog: false, origin: origin.map(str::to_string), opened } }; let before = h.state().session.sources.recent.clone(); h.state_mut().ui.service_steps.push(step("http://127.0.0.1/{z}/{x}/{y}", Some("Atlas"))); diff --git a/crates/ui-egui/src/lib.rs b/crates/ui-egui/src/lib.rs index c2e750df7..678298303 100644 --- a/crates/ui-egui/src/lib.rs +++ b/crates/ui-egui/src/lib.rs @@ -426,11 +426,16 @@ pub struct ServiceBrowse { pub struct TilesResolve { /// `service.add`'s parameters, given the template as `"tiles"` once read. pub params: Value, - /// The tileset list, whose query the template keeps; the document being read; how many have + /// The tileset list, whose safe query options the template may inherit; the document being read; how many have /// been. pub list: String, pub doc: String, + /// Runtime discovery lineage: once a document or template crosses origins, later hops + /// stay anonymous even if they return to a saved connection's origin. + pub crossed_origin: bool, pub read: u8, + /// Destinations that lost inherited query settings across both document reads. + pub withheld_addresses: u32, pub vector: bool, /// The usual-path template, when the advertised one can't be read. pub fallback: String, @@ -442,10 +447,10 @@ pub struct TilesResolve { /// `origin` is the connection a tree add is for, named in Recent once the layer is added. pub enum ServiceStep { /// An OGC API layer whose advertised tile template is being read. - Tiles { resolve: TilesResolve, dialog: bool, origin: Option }, + Tiles { resolve: TilesResolve, dialog: bool, origin: Option, opened: u64 }, /// A WCS coverage whose description is being read before it is asked for: `service.coverage`'s /// parameters and the description on its way. - Coverage { params: Value, slot: Slot, String>>, dialog: bool, origin: Option }, + Coverage { params: Value, slot: Slot, String>>, dialog: bool, origin: Option, opened: u64 }, } /// Fetches an address into a slot, off the frame. @@ -1643,7 +1648,7 @@ impl RustGisApp { // Not an undoable edit: saving names the project, as `project.saveAs` does, once the // file is written. let old = std::mem::replace(&mut self.session.project.name, stem.clone()); - let (text, stripped) = match self.session.project.saved_text(None) { + let (text, report) = match self.session.project.saved_text(None) { Ok(saved) => saved, Err(e) => { self.session.project.name = old; @@ -1656,14 +1661,8 @@ impl RustGisApp { return Err(e); } self.session.notify(crate::i18n::fmt("Downloaded {name}", &[("name", &file)]), false); - if stripped > 0 { - self.session.notify( - crate::i18n::fmt( - "{n} layer address(es) were saved without their keys, tokens or passwords: after RustGIS closes, add those layers again to give them", - &[("n", &stripped.to_string())], - ), - false, - ); + for notice in report.notices() { + self.session.notify(notice, false); } self.session.mark_saved(); self.ui.web_file = Some(file); @@ -2081,8 +2080,56 @@ impl RustGisApp { self.run(id, params) } + /// Imports a peer's project without going through User command dispatch or restore memory. + pub fn open_peer_text(&mut self, text: &str) -> Result<(), String> { + let report = self.session.open_peer_text(rustgis_engine::Peer::Embedder, text)?; + assistantpane::peer_replaced(self); + self.ui.web_file = None; + self.map.deleting = None; + self.retire_cog_probes(); + if let Some(steps) = self.ws.recording.take() { + self.session.notify( + crate::i18n::fmt( + "Recording stopped: another project opened, and its {n} steps were for the previous one.", + &[("n", &steps.len().to_string())], + ), + true, + ); + } + layoutview::follow_project(self); + for notice in report.notices() { + self.session.notify(notice, false); + } + Ok(()) + } + + /// Reports pending header reads retired by either ordinary or peer project replacement. + fn retire_cog_probes(&mut self) { + for (p, _) in std::mem::take(&mut self.ui.cog_probes) { + let url = p.get("url").and_then(Value::as_str).unwrap_or(""); + let url = serviceui::address_in_message(rustgis_engine::data::s3_address::inner(url)); + let why = i18n::fmt("{url}: its project was closed while it was read, so it wasn't added", &[("url", &url)]); + self.landed_in_dialog(&p, Err(&why)); + self.session.notify(why, false); + } + } + /// Runs a command by id: `ui.*` here, everything else in the engine. pub fn run(&mut self, id: &str, params: Value) -> Result { + self.run_with_prepared(id, params, AppDispatch::Direct) + } + + /// Reports a command failure once through the normal diagnostics and notice path. + fn command_failed(&mut self, id: &str, error: &str, notice_mark: u64) { + let text = rustgis_engine::Session::redacted(error); + log::info!("{id}: {}", text.as_str()); + if !self.session.notices_since(notice_mark).iter().any(|notice| notice.error && notice.text == text.as_str()) { + self.session.notify_redacted(text, true); + } + } + + /// Runs direct and resolved-service commands with the same app lifecycle. + fn run_with_prepared(&mut self, id: &str, params: Value, dispatch: AppDispatch) -> Result { let _diagnostics = self.session.diagnostics.enter(); if !matches!(id, "ui.menu" | "ui.ribbon" | "ui.language" | "ui.worldview") { self.ui.ribbon_peek = false; @@ -2106,24 +2153,24 @@ impl RustGisApp { let notice_mark = self.session.notice_mark(); // A plugin's command that adds data by address doesn't freeze the app: the download is // left for the frame loop (`ui.addUrl`), as in the browser. - let mut r = if id.starts_with("ui.") { - self.run_ui(id, ¶ms) - } else if id.starts_with(rustgis_engine::plugins::PREFIX) { - self.session.run_leaving_downloads(id, ¶ms) - } else { - self.session.run(id, ¶ms) + let resolved_lineage = match &dispatch { + AppDispatch::ResolvedTiles { crossed_origin, .. } => Some(*crossed_origin), + _ => None, + }; + let mut r = match dispatch { + AppDispatch::ResolvedTiles { template, crossed_origin } => { + rustgis_engine::cmd::service::add_resolved_tiles(&mut self.session, ¶ms, &template, crossed_origin) + } + AppDispatch::Direct if id.starts_with("ui.") => self.run_ui(id, ¶ms), + AppDispatch::Direct if id.starts_with(rustgis_engine::plugins::PREFIX) => self.session.run_leaving_downloads(id, ¶ms), + AppDispatch::Direct => self.session.run(id, ¶ms), }; r = self.deliver_gp_files(r); if r.is_ok() && id == "tool.set" { self.map.cancel_sketch(); } if let Err(e) = &r { - log::info!("{id}: {e}"); - // UI-only failures can return before an engine command emits a notice. - let text = rustgis_engine::Session::redacted(e); - if !self.session.notices_since(notice_mark).iter().any(|notice| notice.error && notice.text == text.as_str()) { - self.session.notify_redacted(text, true); - } + self.command_failed(id, e, notice_mark); } // Commands can replace the project, switch maps or change the renderer while the pane // is closed. A field picked for the old context must not overwrite the new renderer. @@ -2146,15 +2193,9 @@ impl RustGisApp { self.session.notify(text, true); } // A probe still under way is for an address added to the old project: dropped, so - // its layer never lands in the new one, and said, as a download landing late is. (A plugin - // command waiting for it went with the old project's waits.) - for (p, _) in std::mem::take(&mut self.ui.cog_probes) { - let url = p.get("url").and_then(Value::as_str).unwrap_or(""); - let url = serviceui::address_in_message(rustgis_engine::data::s3_address::inner(url)); - let why = i18n::fmt("{url}: its project was closed while it was read, so it wasn't added", &[("url", &url)]); - self.landed_in_dialog(&p, Err(&why)); - self.session.notify(why, false); - } + // its layer never lands in the new one, and said, as a download landing late is. A plugin + // command waiting for it went with the old project's waits. + self.retire_cog_probes(); } // Undo and redo can move vertices: an index chosen before may now name another vertex. // A step moved or deleted, from the pane, Command Search, the assistant or a script: the @@ -2200,7 +2241,30 @@ impl RustGisApp { self.ui.dynamic_text = None; } if r.is_ok() { - self.record_step(id, ¶ms, active); + if resolved_lineage == Some(true) { + if self.ws.recording.is_some() { + self.session.notify( + tl!( + "This service layer isn't recorded: its tile discovery crossed origins, and a task could use a different login." + ), + false, + ); + } + } else if resolved_lineage == Some(false) { + // A saved connection can replay its same-origin discovery with its current login; + // runtime bearer wrappers themselves must never become recorded parameters. + let mut replay = params.clone(); + if params.get("connection").and_then(Value::as_str).is_some_and(|id| !id.is_empty()) { + for key in ["url", "source_url", "tiles"] { + if let Some(url) = params.get(key).and_then(Value::as_str) { + replay[key] = json!(rustgis_engine::render::rustgis_tiles::logins::inner(url)); + } + } + } + self.record_step(id, &replay, active); + } else { + self.record_step(id, ¶ms, active); + } } self.note_last_project(); // Sources added, saved or made favourites are kept with the preferences. @@ -4307,6 +4371,13 @@ impl RustGisApp { } } + /// Reports query losses once when discovery finishes, including a refused next document. + fn notice_service_query_loss(&mut self, count: u32) { + if count > 0 { + self.session.notify(i18n::fmt(rustgis_engine::data::ogc::QUERY_WITHHELD_NOTICE, &[("n", &count.to_string())]), false); + } + } + /// Tells how a service layer added outside the add-service dialog (or in it) went: in the /// dialog's own line, or as a notice for the Connections tree. fn report_step(&mut self, dialog: bool, r: Result) { @@ -4318,52 +4389,121 @@ impl RustGisApp { } } + /// Refuses answers from a replaced project and reports the retired read without starting follow-up work. + fn service_read_current(&mut self, opened: u64, address: &str, pending: Option<&str>) -> bool { + if opened == self.session.opened { + return true; + } + let url = serviceui::address_in_message(address); + let why = i18n::fmt("{url}: its project was closed while it was read, so it wasn't added", &[("url", &url)]); + if pending.is_some() && self.dialogs.info.as_deref() == pending { + self.dialogs.info = None; + } + self.session.notify(why, false); + false + } + /// Moves the service layers waiting on a document along: an OGC API layer's tile template /// (read from its tileset list, then the tileset when the list doesn't carry it; the layer is /// added with the template read, or the usual path) and a WCS coverage's description (the /// coverage is then asked for the view and downloaded as From URL… does). fn poll_service_steps(&mut self) { use rustgis_engine::data::ogc::{TilesStep, resolve_tiles}; - for step in std::mem::take(&mut self.ui.service_steps) { + let steps = std::mem::take(&mut self.ui.service_steps); + // A read in the new project can display the same title as a retired read. Preserve + // progress belonging to that live generation, even before its answer has landed. + let current: std::collections::HashSet = steps + .iter() + .filter_map(|step| { + let (params, dialog, opened, message) = match step { + ServiceStep::Tiles { resolve, dialog, opened, .. } => { + (&resolve.params, *dialog, *opened, "Reading the tileset of {name}") + } + ServiceStep::Coverage { params, dialog, opened, .. } => (params, *dialog, *opened, "Describing {name}"), + }; + (dialog && opened == self.session.opened) + .then(|| i18n::fmt(message, &[("name", params.get("name").and_then(Value::as_str).unwrap_or(""))])) + }) + .collect(); + for step in steps { match step { - ServiceStep::Tiles { mut resolve, dialog, origin } => { + ServiceStep::Tiles { mut resolve, dialog, origin, opened } => { + let pending = dialog.then(|| { + i18n::fmt( + "Reading the tileset of {name}", + &[("name", resolve.params.get("name").and_then(Value::as_str).unwrap_or(""))], + ) + }); + if !self.service_read_current(opened, &resolve.doc, pending.as_deref().filter(|message| !current.contains(*message))) { + continue; + } let Some(r) = resolve.slot.lock().unwrap_or_else(|e| e.into_inner()).take() else { - self.ui.service_steps.push(ServiceStep::Tiles { resolve, dialog, origin }); + self.ui.service_steps.push(ServiceStep::Tiles { resolve, dialog, origin, opened }); continue; }; let next = r.ok().and_then(|body| resolve_tiles(&resolve.list, &resolve.doc, &body, resolve.vector)); + let next = next.map(|resolved| { + resolve.withheld_addresses = resolve.withheld_addresses.saturating_add(resolved.withheld_addresses); + resolved.step + }); let template = match next { Some(TilesStep::Next(next)) if resolve.read < 2 => { resolve.read += 1; - resolve.doc = next.clone(); - match serviceui::request_from( - self, - resolve.params.get("connection").and_then(Value::as_str), - resolve.params.get("url").and_then(Value::as_str).unwrap_or(""), - &next, - ) { - Ok(url) => resolve.slot = fetch_into(&url), + // The resolver already carried the current document's binding, or + // its absence. A saved connection must not restore it after a hop. + match rustgis_engine::render::rustgis_tiles::address::check(&next, Some(&resolve.doc)) { + Ok(()) => { + resolve.crossed_origin |= rustgis_engine::render::rustgis_tiles::logins::origin(&resolve.doc) + != rustgis_engine::render::rustgis_tiles::logins::origin(&next); + resolve.doc = next; + resolve.slot = fetch_into(&resolve.doc); + } Err(e) => { + self.notice_service_query_loss(resolve.withheld_addresses); self.report_step(dialog, Err(e)); continue; } } - self.ui.service_steps.push(ServiceStep::Tiles { resolve, dialog, origin }); + self.ui.service_steps.push(ServiceStep::Tiles { resolve, dialog, origin, opened }); continue; } Some(TilesStep::Template(t)) => t, - _ => resolve.fallback.clone(), + _ => rustgis_engine::render::rustgis_tiles::logins::inherit( + &resolve.doc, + rustgis_engine::render::rustgis_tiles::logins::inner(&resolve.fallback), + ), }; + resolve.crossed_origin |= rustgis_engine::render::rustgis_tiles::logins::origin(&resolve.doc) + != rustgis_engine::render::rustgis_tiles::logins::origin(&template); + self.notice_service_query_loss(resolve.withheld_addresses); let mut p = resolve.params; p["tiles"] = json!(template); let title = p["name"].as_str().unwrap_or("").to_string(); - let r = connectiontree::run_named(self, "service.add", p, origin.as_deref()).map(|_| serviceui::added(&title)); + let before = self.session.sources.generation; + let r = self.run_with_prepared( + "service.add", + p, + AppDispatch::ResolvedTiles { template, crossed_origin: resolve.crossed_origin }, + ); + if r.is_ok() { + connectiontree::name_recent(self, before, origin.as_deref()); + } + let r = r.map(|_| serviceui::added(&title)); self.report_step(dialog, r); self.map.invalidate(); } - ServiceStep::Coverage { mut params, slot, dialog, origin } => { + ServiceStep::Coverage { mut params, slot, dialog, origin, opened } => { + let pending = dialog + .then(|| i18n::fmt("Describing {name}", &[("name", params.get("name").and_then(Value::as_str).unwrap_or(""))])); + if !self.service_read_current( + opened, + params.get("url").and_then(Value::as_str).unwrap_or(""), + pending.as_deref().filter(|message| !current.contains(*message)), + ) { + continue; + } let Some(r) = slot.lock().unwrap_or_else(|e| e.into_inner()).take() else { - self.ui.service_steps.push(ServiceStep::Coverage { params, slot, dialog, origin }); + self.ui.service_steps.push(ServiceStep::Coverage { params, slot, dialog, origin, opened }); continue; }; let title = params["name"].as_str().unwrap_or("").to_string(); @@ -4374,7 +4514,11 @@ impl RustGisApp { let connection = params.as_object_mut().and_then(|p| p.remove("connection")); let get = c.get("url").and_then(Value::as_str).unwrap_or("").to_string(); let mut p = json!({"url": get, "name": title, "coverage": true}); - if let Some(connection) = connection { + if params.get("anonymous").and_then(Value::as_bool) == Some(true) + || c.get("anonymous").and_then(Value::as_bool) == Some(true) + { + p["anonymous"] = json!(true); + } else if let Some(connection) = connection { p["connection"] = connection; } connectiontree::run_add_url(self, p, origin.as_deref()) @@ -5450,6 +5594,93 @@ mod download_service_tests { use super::{RustGisApp, Services, complete_gp_file}; use serde_json::json; + /// The real asynchronous completion receiver owns the forwarding notice, even when the + /// layer is added successfully. Input is already downloaded into the usual completion slot. + #[test] + fn tile_resolver_completion_reports_query_settings_it_withheld() { + for (query, count) in [("collections=roads", 0), ("token=RGSYNTH_ui_20261103&collections=roads", 1)] { + let mut app = RustGisApp::new(rustgis_engine::Session::new(rustgis_engine::Project::default())); + let list = format!("https://h.invalid/svc/tiles?{query}"); + let capabilities = json!({"title":"roads", "links":[{"rel":"tilesets-map", "href":list}]}).to_string(); + let body = + br#"{"links":[{"rel":"item", "type":"image/png", "href":"https://elsewhere.invalid/{tileMatrix}/{tileRow}/{tileCol}"}]}"# + .to_vec(); + let resolve = super::TilesResolve { + params: json!({"url":"https://h.invalid/svc", "type":"ogcapi", "layer":"map", "text":capabilities}), + list: list.clone(), + doc: list, + crossed_origin: false, + read: 0, + withheld_addresses: 0, + vector: false, + fallback: "https://h.invalid/svc/tiles/{z}/{y}/{x}".into(), + slot: std::sync::Arc::new(std::sync::Mutex::new(Some(Ok(body)))), + }; + app.ui.service_steps.push(super::ServiceStep::Tiles { resolve, dialog: true, origin: None, opened: app.session.opened }); + app.poll_service_steps(); + assert!(app.ui.service_steps.is_empty()); + let notices = app.session.notices.iter().filter(|n| n.text.contains("address(es) the service listed")).count(); + assert_eq!(notices, count); + assert_eq!(app.session.map().tiles.len(), 1); + assert_eq!(app.session.map().tiles[0].url, "https://elsewhere.invalid/{z}/{y}/{x}?collections=roads"); + assert!(app.dialogs.error.is_none(), "{:?}", app.dialogs.error); + } + } + + #[test] + fn retired_service_steps_clear_only_their_own_pending_dialog() { + for coverage in [false, true] { + for newer in 0..3 { + let mut app = RustGisApp::new(rustgis_engine::Session::new(rustgis_engine::Project::default())); + let opened = app.session.opened; + let params = json!({"url":"https://retired.invalid/service", "name":"Old read"}); + let slot = std::sync::Arc::new(std::sync::Mutex::new(None)); + let step = |opened, slot: std::sync::Arc, String>>>>| { + if coverage { + super::ServiceStep::Coverage { params: params.clone(), slot, dialog: true, origin: None, opened } + } else { + super::ServiceStep::Tiles { + resolve: super::TilesResolve { + params: params.clone(), + list: "https://retired.invalid/list".into(), + doc: "https://retired.invalid/list".into(), + crossed_origin: false, + read: 0, + withheld_addresses: 0, + vector: false, + fallback: "https://retired.invalid/{z}/{y}/{x}".into(), + slot: slot.clone(), + }, + dialog: true, + origin: None, + opened, + } + } + }; + app.ui.service_steps.push(step(opened, slot.clone())); + app.session.run("project.new", &json!({})).unwrap(); + let pending = if coverage { "Describing Old read" } else { "Reading the tileset of Old read" }; + if newer == 2 { + app.ui.service_steps.push(step(app.session.opened, Default::default())); + } + app.dialogs.info = Some(if newer == 1 { "New operation" } else { pending }.into()); + *slot.lock().unwrap() = Some(Ok(b"late completion".to_vec())); + app.poll_service_steps(); + assert_eq!(app.ui.service_steps.len(), usize::from(newer == 2)); + assert!(app.session.map().tiles.is_empty() && app.session.map().layers.is_empty()); + assert!(app.session.notices.iter().any(|n| n.text.contains("its project was closed while it was read"))); + assert_eq!( + app.dialogs.info.as_deref(), + match newer { + 1 => Some("New operation"), + 2 => Some(pending), + _ => None, + } + ); + } + } + } + #[test] fn rejected_download_does_not_announce_success_or_rename_project() { let mut app = RustGisApp::with_sample() @@ -6034,3 +6265,53 @@ mod look_tests { assert!(app.use_theme("Gone").is_err(), "a name the list does not hold"); } } + +#[cfg(test)] +mod peer_output_tests { + use super::*; + + /// Drives the actual receiving wrapper, including the UI state a direct Session call misses. + #[test] + fn peer_project_uses_import_authority_and_resets_old_ui_work() { + let mut app = RustGisApp::with_sample(); + app.session.project.maps[0].layers[0].dataset.source = Some("https://h.invalid/?token=RGSYNTH_4".into()); + let (text, _) = app.session.project.saved_text(None).unwrap(); + app.session.trusted_tasks.insert("prior".into()); + app.assistant.queued = Some("old project action".into()); + app.ui.web_file = Some("prior.rgis".into()); + let generation = app.session.opened; + app.ui.cog_probes.push(( + json!({"url":"https://peer-probe.invalid/elevation.tif?token=RGSYNTH_peer_probe","name":"Elevation"}), + Default::default(), + )); + app.dialogs.info = Some(crate::serviceui::downloading("Elevation")); + let probe_mark = app.session.notice_mark(); + app.open_peer_text(&text).unwrap(); + assert_ne!(app.session.opened, generation); + assert!(app.ui.cog_probes.is_empty()); + assert!(app.dialogs.info.is_none(), "a retired peer probe cannot keep the reading message open"); + let retired = app + .session + .notices_since(probe_mark) + .iter() + .filter(|n| n.text.contains("peer-probe.invalid/elevation.tif") && n.text.contains("project was closed")) + .count(); + assert_eq!(retired, 1, "the peer replacement must say which pending read was omitted"); + assert!(app.session.notices_since(probe_mark).iter().all(|n| !n.text.contains("RGSYNTH_peer_probe"))); + let clean_mark = app.session.notice_mark(); + app.open_peer_text(&text).unwrap(); + assert!( + app.session.notices_since(clean_mark).iter().all(|n| !n.text.contains("project was closed")), + "a clean peer replacement does not replay a retired read" + ); + assert!(app.session.trusted_tasks.is_empty()); + assert!(app.ui.web_file.is_none() && app.assistant.queued.is_none()); + assert!(!serde_json::to_string(&app.session.project).unwrap().contains("RGSYNTH_4")); + } +} + +/// Resolved tile discovery shares the normal command bookkeeping. +enum AppDispatch { + Direct, + ResolvedTiles { template: String, crossed_origin: bool }, +} diff --git a/crates/ui-egui/src/serviceui.rs b/crates/ui-egui/src/serviceui.rs index 581bb17d8..9351a935e 100644 --- a/crates/ui-egui/src/serviceui.rs +++ b/crates/ui-egui/src/serviceui.rs @@ -98,7 +98,8 @@ pub(crate) fn add_layer_from( origin: Option<&str>, connection: Option<&str>, ) -> Result { - let connection = connection_for(app, connection, &src.url); + let anonymous = l.get("anonymous").and_then(Value::as_bool) == Some(true); + let connection = connection_for(app, connection, &src.url).filter(|_| !anonymous); let name = l.get("name").and_then(Value::as_str).unwrap_or("").to_string(); let title = l.get("title").and_then(Value::as_str).unwrap_or(&name).to_string(); let offer = l.get("offer").cloned().unwrap_or_default(); @@ -117,7 +118,13 @@ pub(crate) fn add_layer_from( // The layer's bounds go along, for its features' axes (#1053). let bounds = l.get("bbox").cloned().unwrap_or_default(); let mut params = json!({"url": items, "name": title, "service": true, "bounds": bounds}); - if let Some(connection) = connection { + let anonymous = l.get("anonymous").and_then(Value::as_bool) == Some(true) + || rustgis_engine::render::rustgis_tiles::logins::origin(&src.url) + != rustgis_engine::render::rustgis_tiles::logins::origin(items); + if anonymous { + params["url"] = json!(rustgis_engine::data::login_address::inner(items)); + params["anonymous"] = json!(true); + } else if let Some(connection) = connection { params["connection"] = json!(connection); } // An area asked for keeps the view, even on an empty map, as `service.add` does. @@ -137,9 +144,15 @@ pub(crate) fn add_layer_from( if let Some(connection) = connection { p["connection"] = json!(connection); } - let describe = request_from(app, connection, &src.url, &describe)?; + let describe = if anonymous { + p["anonymous"] = json!(true); + p["url"] = json!(rustgis_engine::data::login_address::inner(&src.url)); + rustgis_engine::data::login_address::inner(&describe).to_string() + } else { + request_from(app, connection, &src.url, &describe)? + }; let slot = crate::fetch_into(&describe); - app.ui.service_steps.push(crate::ServiceStep::Coverage { params: p, slot, dialog, origin }); + app.ui.service_steps.push(crate::ServiceStep::Coverage { params: p, slot, dialog, origin, opened: app.session.opened }); return Ok(i18n::fmt("Describing {name}", &[("name", &title)])); } let mut p = json!({"url": src.url, "layer": name, "text": src.text, "documents": src.documents, "source_url": l.get("source_url")}); @@ -149,6 +162,10 @@ pub(crate) fn add_layer_from( if let Some(connection) = connection { p["connection"] = json!(connection); } + if anonymous { + p["anonymous"] = json!(true); + p["url"] = json!(rustgis_engine::data::login_address::inner(&src.url)); + } if l.get("time").is_some() && !time.is_empty() { p["time"] = json!(time); } @@ -159,16 +176,33 @@ pub(crate) fn add_layer_from( p["name"] = json!(title); let fallback = offer.get("url").and_then(Value::as_str).unwrap_or("").to_string(); let vector = offer.get("type").and_then(Value::as_str) == Some("vectorTiles"); - let list = request_from(app, connection, &src.url, list)?; + let list = if anonymous { + rustgis_engine::data::login_address::inner(list).to_string() + } else { + request_from(app, connection, &src.url, list)? + }; let slot = crate::fetch_into(&list); - let resolve = crate::TilesResolve { params: p, list: list.clone(), doc: list, read: 0, vector, fallback, slot }; - app.ui.service_steps.push(crate::ServiceStep::Tiles { resolve, dialog, origin }); + let resolve = crate::TilesResolve { + params: p, + list: list.clone(), + doc: list.clone(), + crossed_origin: anonymous + || rustgis_engine::render::rustgis_tiles::logins::origin(&src.url) + != rustgis_engine::render::rustgis_tiles::logins::origin(&list), + read: 0, + withheld_addresses: 0, + vector, + fallback, + slot, + }; + app.ui.service_steps.push(crate::ServiceStep::Tiles { resolve, dialog, origin, opened: app.session.opened }); Ok(i18n::fmt("Reading the tileset of {name}", &[("name", &title)])) } None => crate::connectiontree::run_named(app, "service.add", p, origin.as_deref()).map(|_| added(&title)), } } +/// Starts grouped discovery under its original request, retaining the import target and byte budget. pub fn start(app: &mut RustGisApp, p: &Value, refresh: bool) -> Result<(), String> { if !p.is_object() { return Err(tl!("Give the service's web address").into()); @@ -219,7 +253,7 @@ pub fn start(app: &mut RustGisApp, p: &Value, refresh: bool) -> Result<(), Strin let url = task.params.get("url").and_then(Value::as_str).unwrap_or_default(); let kind = task.params.get("type").and_then(Value::as_str); let caps = rustgis_engine::data::ogc::capabilities_url(url, kind); - let caps = match task.request(app, &caps) { + let caps = match task.request(&caps) { Ok(caps) => caps, Err(e) => { app.session.download_failed(task.token); @@ -236,14 +270,18 @@ pub fn start(app: &mut RustGisApp, p: &Value, refresh: bool) -> Result<(), Strin impl Import { /// Anonymous service provenance forbids selecting a login, including for same-origin links. - fn request(&self, app: &mut RustGisApp, url: &str) -> Result { + fn request(&self, url: &str) -> Result { if self.params.get("anonymous").and_then(Value::as_bool) == Some(true) { return Ok(rustgis_engine::data::login_address::inner(url).to_string()); } let base = self.params.get("url").and_then(Value::as_str).unwrap_or_default(); - request_from(app, self.connection.as_ref().and_then(Value::as_str), base, url) + // The plan already resolved this exact destination and its login (or absence). + // Re-selecting from the original saved connection would undo a foreign hop. + rustgis_engine::render::rustgis_tiles::address::check(url, Some(base)).map_err(|e| crate::connectionui::translated(&e))?; + Ok(url.to_string()) } + /// Consumes the engine plan verbatim: foreign-hop anonymity cannot reselect a login. fn discover(&mut self, app: &mut RustGisApp) -> Result<(), String> { let listing = app.session.run("service.groupPlan", &self.params)?; if listing.get("request").and_then(|p| p.get("anonymous")).and_then(Value::as_bool) == Some(true) { @@ -254,7 +292,7 @@ impl Import { self.connection = None; } if let Some(next) = listing.get("next").and_then(Value::as_str) { - let next = self.request(app, next)?; + let next = self.request(next)?; self.slot = fetch_into(&next); self.document = Some(next); } else { @@ -265,20 +303,22 @@ impl Import { .flatten() .filter_map(|l| l.get("offer").and_then(|o| o.get("url")).and_then(Value::as_str).map(str::to_string)) .collect(); - self.next(app)?; + self.next()?; } Ok(()) } - fn next(&mut self, app: &mut RustGisApp) -> Result<(), String> { + /// Fetches one exact planned collection within the remaining shared byte budget. + fn next(&mut self) -> Result<(), String> { self.current = self.remaining.pop_front(); if let Some(url) = &self.current { - let url = self.request(app, url)?; + let url = self.request(url)?; self.slot = fetch_into_limited(&url, rustgis_engine::cmd::service::MAX_GROUP_BYTES.saturating_sub(self.bytes)); } Ok(()) } + /// Accumulates supplied documents or feature responses before the engine import/refresh. fn landed(&mut self, app: &mut RustGisApp, bytes: Vec) -> Result<(), String> { if let Some(document) = self.document.take() { let text = crate::service_text(bytes).map_err(|_| tl!("The service's answer isn't UTF-8 text").to_string())?; @@ -298,7 +338,7 @@ impl Import { } let text = String::from_utf8(bytes).map_err(|_| tl!("The service's answer isn't UTF-8 text").to_string())?; self.responses.push(json!({"url":self.current.take(),"text":text})); - self.next(app)?; + self.next()?; Ok(()) } } @@ -509,6 +549,83 @@ fn noted(app: &mut RustGisApp, landed: &Landed, note: Option) { mod tests { use super::*; + /// Discovery's anonymous provenance survives both deferred protocols and their final params. + #[test] + fn anonymous_service_offers_keep_descriptions_and_tiles_unbound() { + use rustgis_engine::render::rustgis_tiles::{cache, logins}; + let _turn = crate::fetching::turn(); + let caps = r#" + socSoil carbon + 5 49.25.8 50 + "#; + let describe = br#" + 49.2 550 5.8"#; + for coverage in [false, true] { + for anonymous in [true, false] { + let mut app = RustGisApp::new(rustgis_engine::Session::new(rustgis_engine::Project::default())); + let url = "https://anonymous-offer.invalid/collections"; + let kind = if coverage { "wcs" } else { "ogcapi" }; + let id = app.run("connection.new", json!({"kind":kind, "address":url, "auth":"header", "keyName":"X-Key"})).unwrap()["id"] + .as_str() + .unwrap() + .to_string(); + app.run("connection.login", json!({"id":id, "apiKey":"RGSYNTH_anonymous_offer"})).unwrap(); + let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let capture = seen.clone(); + cache::set_fetcher(move |url, done| { + capture.lock().unwrap().push(cache::split_download_limit(&url).0.to_string()); + done(Err("held description".into())); + }); + let text = if coverage { + caps.to_string() + } else { + json!({"collections":[{"id":"relief","links":[{"rel":"tilesets-map","href":"https://anonymous-offer.invalid/list"}]}]}) + .to_string() + }; + let src = ServiceSource { url: url.into(), kind, text, documents: vec![] }; + let listed = app.session.run("service.layers", &json!({"url":url,"type":kind,"text":src.text,"connection":id})).unwrap(); + let mut layer = listed["layers"][0].clone(); + layer["anonymous"] = json!(anonymous); + add_layer_from(&mut app, &src, &layer, "", None, false, None, Some(&id)).unwrap(); + { + let seen = seen.lock().unwrap(); + assert_eq!(seen.len(), 1); + assert_eq!(logins::apply(&seen[0]).unwrap().is_some(), !anonymous, "coverage={coverage}, anonymous={anonymous}"); + } + let params = match &app.ui.service_steps[0] { + crate::ServiceStep::Coverage { params, slot, .. } => { + *slot.lock().unwrap() = Some(Ok(describe.to_vec())); + params + } + crate::ServiceStep::Tiles { resolve, .. } => { + assert_eq!(resolve.crossed_origin, anonymous); + *resolve.slot.lock().unwrap() = Some(Ok(br#"{"links":[{"rel":"item","type":"image/png","href":"https://anonymous-offer.invalid/{tileMatrix}/{tileRow}/{tileCol}"}]}"#.to_vec())); + &resolve.params + } + }; + assert_eq!(params.get("connection").is_some(), !anonymous); + assert_eq!(params.get("anonymous").and_then(Value::as_bool).unwrap_or(false), anonymous); + // Deliver through the actual completion handler and its admitted final command. + app.poll_service_steps(); + assert!(app.ui.service_steps.is_empty()); + if coverage { + let seen = seen.lock().unwrap(); + assert_eq!(seen.len(), 2, "{:?}", app.session.notices); + assert!(seen[1].contains("REQUEST=GetCoverage")); + assert_eq!(logins::apply(&seen[1]).unwrap().is_some(), !anonymous); + let (_, _, params, _) = app.ui.downloads.first().expect("the coverage download was dispatched"); + assert_eq!(params.get("connection").and_then(Value::as_str), (!anonymous).then_some(id.as_str())); + assert_eq!(params.get("anonymous").and_then(Value::as_bool).unwrap_or(false), anonymous); + } else { + let tile = app.session.map().tiles.first().expect("the resolved tiles were added"); + assert_eq!(tile.anonymous, anonymous); + assert_eq!(tile.connection, if anonymous { "" } else { id.as_str() }); + assert_eq!(logins::apply(&tile.url).unwrap().is_some(), !anonymous); + } + } + } + } + /// A linked page reached through another origin is fetched anonymously even when it points /// back at the first page's origin: the login follows the referring page, not the first one. #[test] diff --git a/crates/ui-egui/tests/autosave.rs b/crates/ui-egui/tests/autosave.rs index 1115841bf..30d64ec44 100644 --- a/crates/ui-egui/tests/autosave.rs +++ b/crates/ui-egui/tests/autosave.rs @@ -6,6 +6,7 @@ use std::sync::{Arc, Mutex}; use egui::accesskit::Role; use egui_kittest::kittest::Queryable; +use rustgis_engine::project::{Task, TaskStep}; use rustgis_ui_egui::{FolderStore, RecoveryStore, RustGisApp, Services}; use serde_json::json; @@ -138,12 +139,53 @@ fn unsaved_work_is_kept_and_offered_back_after_a_crash() { assert!(app.session.map().layer_by_name("Inspections").is_some()); assert_eq!(app.session.path.as_deref(), Some(std::path::absolute(&project).unwrap().as_path()), "Save writes back to its file"); assert!(app.session.dirty(), "recovered work is unsaved"); + assert!( + !app.session.notices.iter().any(|n| n.text.contains("without credentials or URL fragments")), + "a clean recovery reports no credential loss" + ); // The copy is this run's now, unchanged, until the work is saved. assert!(!file(&dir.0, PENDING).exists() && file(&dir.0, LIVE).is_file()); h.state_mut().run("project.save", json!({})).unwrap(); settle(&mut h); assert!(!file(&dir.0, LIVE).exists(), "saved work needs no copy"); assert!(std::fs::read_to_string(&project).unwrap().contains("Inspections")); + drop(h); + + // This task is stored, never run: recovering its credential needs no network request. + // Nothing in this scenario is explicitly saved, so the saved-text cache cannot restore it. + let dir = TempDir::new("crash-credentials"); + let mut h = launch(&dir.0, &prefs); + let secret = "RGSYNTH_autosave_20261105"; + h.state_mut().session.project.tasks.push(Task { + name: "Credential recovery".into(), + description: String::new(), + steps: vec![TaskStep { + title: "Log in".into(), + instructions: String::new(), + command: "connection.login".into(), + params: json!({"id": "autosave-unused", "apiKey": secret}), + }], + }); + edit(&mut h, "Credential recovery"); + settle(&mut h); + assert!(!std::fs::read_to_string(file(&dir.0, LIVE)).unwrap().contains(secret), "the recovery file contains no token"); + assert_eq!(h.state().session.project.tasks.last().unwrap().steps[0].params["apiKey"], secret, "autosave preserves the live token"); + drop(h); + + let mut h = launch(&dir.0, &prefs); + h.step(); + h.get_by_role_and_label(Role::Button, "Recover").click(); + settle(&mut h); + let app = h.state(); + assert!(app.session.map().layer_by_name("Credential recovery").is_some()); + assert!(!serde_json::to_string(&app.session.project).unwrap().contains(secret), "the recovered state contains no token"); + let losses: Vec<_> = app.session.notices.iter().filter(|n| n.text.contains("without credentials or URL fragments")).collect(); + assert_eq!(losses.len(), 1, "recovery reports the credential loss once"); + assert_eq!( + losses[0].text, + "1 executable step(s) were written without credentials or URL fragments; supply missing values again before replay" + ); + assert!(!losses[0].error); } #[test] diff --git a/docs/contributing.md b/docs/contributing.md index d8fcc18af..1942aca95 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -33,6 +33,14 @@ them together. Reply on the comment that it's left for the release quality pass reply with ``), so it can be found. Bugs, crashes, silent data loss and breaks of the rules in AGENTS.md are fixed in the pull request. +Changes to saved or generated output must classify each new owning field: address, executable +parameters, generated text or user data. Use the shared durable-output transaction and keep +credential removal separate from execution and approval identity. Review every writer that +consumes the owner, including packages and peer exchange. Reuse a focused regression that checks +the written output, restoration of live fields on failure, counted notices and clean-input +silence. Opaque data needs an explicit preservation policy; recursive removal by key name can +destroy user data. + ## Developer checks `cargo xtask ci` runs `cargo fmt --all --check`, diff --git a/docs/mcp.md b/docs/mcp.md index efca93465..f0a72aeb0 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -158,7 +158,11 @@ Passwords, keys and tokens in the arguments, a `run` script's lines included, ar they are written. On Linux and macOS the folder and files are readable by you only; on Windows they're protected by your user profile's permissions. A batch's calls share the batch's `duration_s` and are marked `"batch": true`. An error whose text mentions a credential (a token, a -password, a bearer…) is logged as withheld rather than risk keeping the credential. Days older than 30 are removed. +password, a bearer…) is logged as withheld rather than risk keeping the credential. Log days older +than 30 days are pruned when a logged call creates a new day's file, not by a timer; older files +can remain while logging is idle. You can remove unwanted day files from this folder yourself. +Updating does not rewrite old logs. If an older log exposed a credential, rotate it with its +service before sharing the log. ### MCP over HTTP diff --git a/docs/python.md b/docs/python.md index bc43cf8ee..e068cb644 100644 --- a/docs/python.md +++ b/docs/python.md @@ -139,6 +139,15 @@ The project, with the data that travels with it, goes to that app through the br (`postMessage`); nothing goes to a server. The app runs in your browser, so with the default address your data stays on your machine. Point `url=` only at a copy of the app you trust. +The widget receives public addresses and cleaned executable parameters. Recognized credentials +in supported fields are removed, so protected layers can remain empty there; a warning counts +affected layer addresses. User data, foreign notebook source and opaque parameters are kept +unchanged and may contain credentials: review them before sharing the project with a widget. +When the widget sends a project back, Python can retain credentials only for +unchanged, uniquely matched sources from its last exchange that still match its current project. +Changing or replacing a source can require signing in again in Python. Exchange notices are +available through `m.notices()` on the `Map` or `MapWidget`. + ## The app in a Qt application `rustgis.qt.MapWidget` is the same app in a Qt widget, for a desktop tool or a script that opens a @@ -293,9 +302,13 @@ last view in memory; run `layer.loadAll` on it before `to_geojson` or `to_geopan of them. Open an existing project with `Session(project="atlas.rgis")` or `s.open("atlas.rgis")`. -`s.project_json()` returns its JSON text; `s.run("project.openText", text=...)` opens project -JSON without an intermediate file. `to_png()` returns PNG bytes even when it also writes a -file; omit the path to keep the image in memory. +`s.project_json()` returns JSON with the same credential removal as Save, with notices available +through `s.notices()`. Like Save, this explicit serialization retains a bounded, process-local +credential-restoration snapshot for reopening its exact text; repeated identical text replaces +the same snapshot. It refuses unfinished imports instead of writing runtime reservations. +The widget's private peer exchange uses a separate path that never fills this save cache. +`s.run("project.openText", text=...)` opens project JSON without an intermediate file. `to_png()` +returns PNG bytes even when it also writes a file; omit the path to keep the image in memory. ## From source diff --git a/docs/user-guide/data.md b/docs/user-guide/data.md index fd083ea20..d21dc9a53 100644 --- a/docs/user-guide/data.md +++ b/docs/user-guide/data.md @@ -1221,13 +1221,19 @@ routes, areas, observations…) to draw into. Add Folder connects a folder to th also lists the project's layouts, styles and tasks. A task's steps each run a command; New Task From History turns the geoprocessing history into a task you can replay. -Saved projects, map and layer files, and packages keep no password, key, token or signature in -layer web addresses. The open project keeps them for this session. RustGIS remembers the 32 most -recent distinct saved snapshots whose addresses needed stripping. Opening a plain `.rgis` file -whose bytes exactly match one of those snapshots restores its secrets; saving more distinct -snapshots evicts the oldest. Changed files, packages, imports and files saved by another process -never restore secrets. After a snapshot is evicted or RustGIS closes, add those layers again with -their credentials. +Saved projects, templates, map and layer files, and packages remove recognized credentials from +supported addresses and recognized executable parameters. The open project keeps its credentials. Removal +notices describe what a shared workflow or video will need again; an unchecked-content notice +means opaque content was kept and needs your review. Unrecognized credentials can remain in +addresses without a notice, so review addresses and preserved content before sharing. + +RustGIS remembers up to 32 distinct saved snapshots for explicit project reopening, with at most +1 MiB of restoration data per snapshot and 8 MiB in total. A snapshot must match, and only its +recorded, unchanged fields can regain credentials. Larger restoration records are not remembered; +a notice says so. Imports, packages and peer exchanges do not use this memory. After eviction or +closing RustGIS, sign in again or add the protected sources with their credentials. Feature data +and user text remain data, so review them before sharing. + A layer added from a saved web-service connection records the connection's id (`"connection"` in the `.rgis` file; never its login), so the project needs no token in its addresses. See [Web-service logins](#web-service-logins) for what happens when it opens where that connection or diff --git a/docs/user-guide/web-data.md b/docs/user-guide/web-data.md index 3ddf0e001..2accba00e 100644 --- a/docs/user-guide/web-data.md +++ b/docs/user-guide/web-data.md @@ -495,6 +495,14 @@ rustgis-cli run --cmd 'portal.files={"dataset":"giswqs/geospatial"}' rustgis-cli run --sample --cmd 'portal.add={"url":"https://data.geopf.fr/telechargement/download/LiDARHD-NUALID/NUALHD_1-0__LAZ_LAMB93_KE_2025-06-06/LHD_FXX_0652_6864_PTS_LAMB93_IGN69.copc.laz","name":"LiDAR HD 0652-6864"}' ``` +Service discovery forwards only a small set of query options to addresses supplied by the +service: `collections`, `datetime`, `subset`, `subset-crs`, `crs`, `bgcolor` and `transparent`. +Other settings, including keys, are inherited only by sub-addresses RustGIS constructs within +the original source path. A counted notice says when a listed address did not receive settings. +If it needs authentication, supply an explicitly authorized address for that source rather than +relying on a token being copied from the discovery address. Destination addresses keep settings +they already contain. + ### Layers that change with time A WMS or WMTS layer with a time dimension, from any service, goes on the diff --git a/docs/user-guide/workspace.md b/docs/user-guide/workspace.md index 5cf4a33dc..bea32fc55 100644 --- a/docs/user-guide/workspace.md +++ b/docs/user-guide/workspace.md @@ -32,9 +32,12 @@ The browser app doesn't ask yet; it warns before you leave a page with unsaved c default), RustGIS keeps your changes as a [recovery copy](#autosave-and-recovery) and offers them back the next time it starts; with autosave off, they are lost, so save before you quit. -Saved layer addresses omit passwords, access keys, tokens and signatures. The open session -keeps them, but after restarting RustGIS a protected source may need you to sign in to its -connection again or add it again with credentials. +Supported layer and video address fields, and recognized task, model, notebook and history +parameters, omit recognized passwords, access keys, tokens and signatures. Notices count what was removed or could not +be checked. The open session keeps its credentials, but a saved workflow or protected source may +need them again when reopened. User notes, feature data and opaque content are preserved; see +[Sharing projects and generated items](https://github.com/opengeos/RustGIS/blob/main/SECURITY.md#sharing-projects-and-generated-items) +before sharing a file. Review addresses before sharing too: an unrecognized credential can remain unchanged. ### Autosave and recovery