Query execution cleanup

VIBES.org

@@ -8837,11 +8837,106 @@ Tests should not rely on the repo config.
 :END:
 
 Add a top-level config section for tasks commands, name it =tasks=. Move the current top-level =capture= config into =tasks.capture=. Define a new section for the =tasks.list= that will contain all of its configuration & defaults.
-*** TODO Streamline query execution
+*** REWORKED Streamline query execution
+:LOGBOOK:
+- State "REWORKED"   from "TODO"       [2026-06-23 Tue 20:10]
+:END:
 The commands use a common ETL pipeline where they load some data from files, apply filters/ordering/mutators and lastly present the results in some way. Sometimes the documents are then saved.
-It would be nice to make query execution as streamlined as possible, ideally, =org.logic.query= would expose a function =run_query= that takes a list of documents, a list of query stages to run and the context for the query to run in, and compiles and executes the query and returns the result.
+It would be nice to make query execution as streamlined as possible, ideally, =org.query.runner= would expose a function =run_query= that takes a list of documents, a list of query stages to run and the context for the query to run in, and compiles and executes the query and returns the result.
 That streamlined query execution should be used in all places that extract or transform the org document data - the data should be loaded, then processed with the =run_query= function and used as needed.
 Error handling should be left to the caller (=run_query= should not accept customizable error types/builders).
+**** DONE src/org/commands/agenda/views.py
+:PROPERTIES:
+:context: 
+:END:
+:LOGBOOK:
+- State "DONE"       from "TODO"       [2026-06-23 Tue 19:15]
+:END:
+
+In file src/org/commands/agenda/views.py around line 69, near the ~class AgendaViewContext:~
+
+#+begin_example
++def _build_section_query(filter_query: str, order_by: str | None) -> str:
++    """Build one agenda section filter/order query text."""
++    return build_filter_order_query_text(filter_query, order_by)
+
+#+end_example
+
+These wrappers aren't very useful, please clean this up in all the commands that now have these.
+
+**** REWORKED src/org/commands/tasks/common.py
+:PROPERTIES:
+:context: 
+:END:
+:LOGBOOK:
+- State "REWORKED"   from "TODO"       [2026-06-23 Tue 19:48]
+:END:
+
+In file src/org/commands/tasks/common.py around line 385, near the ~def resolve_headings_by_query(~
+
+#+begin_example
+     selector_query: str,
+ ) -> list[Heading]:
+     """Resolve matching headings across files from selector query."""
+-    compiled_query = compile_query_or_raise(
+-        selector_query,
+-        lambda message: typer.BadParameter(f"Invalid task selector query: {message}"),
+-    )
+-
+     logger.info("Task selector query: %s", selector_query)
+     matches_by_identity: dict[int, Heading] = {}
+     for filename in filenames:
+         document = load_document(filename)
+         logger.info("Running task selector query against file: %s", filename)
+-        results = execute_query_or_raise(
+-            compiled_query,
+-            [document],
+-            {},
+-            lambda message: typer.BadParameter(f"Task selector query failed: {message}"),
+-        )
++        try:
++            results = run_query([document], [selector_query], {})
++        except QueryParseError as exc:
++            raise typer.BadParameter(f"Invalid task selector query: {exc}") from exc
++        except QueryRuntimeError as exc:
++            raise typer.BadParameter(f"Task selector query failed: {exc}") from exc
+
+         for value in results:
+             if not isinstance(value, Heading):
+
+#+end_example
+
+This whole function should use the usual ETL pipeline.
+Ideally, this function should be removed and the callers should use the usual ETL pipeline.
+**** DONE src/org/commands/tasks/remove.py
+:PROPERTIES:
+:context: 
+:END:
+:LOGBOOK:
+- State "DONE"       from "TODO"       [2026-06-23 Tue 19:50]
+:END:
+
+In file src/org/commands/tasks/remove.py around line 99, near the ~def _selected_delete_roots(headings: list[Heading]) -> list[Heading]:~
+
+#+begin_example
++    matches_by_identity: dict[int, Heading] = {}
++    for value in results:
++        if not isinstance(value, Heading):
++            raise typer.BadParameter("Task selector query must return task headings")
++        matches_by_identity[id(value)] = value
++    selected_headings = list(matches_by_identity.values())
++    if not selected_headings:
+
+#+end_example
+
+Sounds like this could be extracted to be a separate function that's used by the commands that need to handle the identity.
+
+**** DONE Cache query compilation results
+:LOGBOOK:
+- State "DONE"       from "TODO"       [2026-06-23 Tue 20:09]
+:END:
+It is often the case that the same queries are recompiled multiple times. We could improve performance by caching the query compilation results and checking if the query was already compiled.
+Add the query compilation cache.
 
 *** TODO Org file repository abstraction
 :PROPERTIES:
@@ -9343,6 +9438,9 @@ org tasks query all-todos
 - Show stats widgets when query return widget objects
 - Support mixed data (title string + histogram widget, etc)
 
+*** Limit & offset applied outside of the query
+- Define variables in context
+- Apply =results[offset:offset + limit]= outside of the query.
 ** Board command improvements                                                             :UX:Board:
 *** Board plan command
 Shows long term plan including completeness of tasks that have children and gannt-style chart for execution.

docs/query_language.md

@@ -10,7 +10,7 @@ A query evaluates over a **stream of values**. Each stage receives a stream and
 - `.` is identity: it forwards the current stream item unchanged.
 - Many expressions can emit zero, one, or many output values per input value.
 
-At the CLI, the initial stream contains one value: the loaded roots collection.
+At the CLI, the initial stream contains the loaded Org documents.
 
 ```bash
 poetry run org tasks query '.[] | .children | length' examples/ARCHIVE_small
@@ -112,12 +112,21 @@ Some functions are no-arg and are written without parentheses:
 
 ```text
 reverse
+fold
 length
 sum
 any
 all
 ```
 
+`fold` collects the current stream into one list value. This differs from bracket fold syntax,
+which runs a subquery per input item and collects each subquery result stream separately.
+
+```text
+[1, 2, 3][] | fold
+.[] | (.children[] | .title_text | fold) | length
+```
+
 ### Tuple expression
 
 Comma builds tuples from expression outputs:
@@ -733,7 +742,7 @@ Notes:
 .[ $offset : $offset + $limit ]
 
 # find most recently modified tasks
-.[][] | sort_by(.repeats + .deadline + .closed + .scheduled | max) | .title_text
+.[] | sort_by(.repeats + .deadline + .closed + .scheduled | max) | .title_text
 
 # same time, two days from now
 now + days(2)

docs/tasks_query_command.md

@@ -46,13 +46,13 @@ Shared top-level config still applies for `mapping`, `exclude`, `todo_states`, `
 1) Fetch headings from all tasks
 
 ```bash
-poetry run org tasks query '.[][] | .title_text' examples/ARCHIVE_small
+poetry run org tasks query '.[] | .title_text' examples/ARCHIVE_small
 ```
 
 2) Fetch headings of completed tasks
 
 ```bash
-poetry run org tasks query '.[][] | select(.todo in $done_states) | .title_text' \
+poetry run org tasks query '.[] | select(.todo in $done_states) | .title_text' \
   examples/ARCHIVE_small
 ```
 
@@ -67,7 +67,7 @@ Prepare a document for team onboarding to Ejabberd.
 3) Fetch a paged window of headings
 
 ```bash
-poetry run org tasks query '[ .[][] | .title_text ][ $offset : $offset + $limit ]' \
+poetry run org tasks query '[ .[] | .title_text ][ $offset : $offset + $limit ]' \
   --offset 5 \
   --limit 5 \
   examples/ARCHIVE_small
@@ -76,7 +76,7 @@ poetry run org tasks query '[ .[][] | .title_text ][ $offset : $offset + $limit
 4) Fetch headings for tasks tagged `Debugging`, then page results
 
 ```bash
-poetry run org tasks query '[ .[][] | select("Debugging" in .tags) | .title_text ][ $offset : $offset + $limit ]' \
+poetry run org tasks query '[ .[] | select("Debugging" in .tags) | .title_text ][ $offset : $offset + $limit ]' \
   --offset 2 \
   --limit 8 \
   examples/ARCHIVE_small
@@ -85,7 +85,16 @@ poetry run org tasks query '[ .[][] | select("Debugging" in .tags) | .title_text
 5) Fetch tasks that are not completed yet
 
 ```bash
-poetry run org tasks query '[ .[][] | select(not(.todo in $done_states)) ][ $offset : $offset + $limit ]' \
+poetry run org tasks query '[ .[] | select(not(.todo in $done_states)) ][ $offset : $offset + $limit ]' \
+  --offset 0 \
+  --limit 5 \
+  examples/ARCHIVE_small
+```
+
+If you want to page across the whole current stream at that point, use `fold` first:
+
+```bash
+poetry run org tasks query '.[] | select(not(.todo in $done_states)) | fold | .[$offset:$offset + $limit]' \
   --offset 0 \
   --limit 5 \
   examples/ARCHIVE_small
@@ -119,7 +128,7 @@ SCHEDULED: <2023-10-19 czw>
 6) Export markdown with a custom Pygments theme
 
 ```bash
-poetry run org tasks query '.[][] | .title_text' \
+poetry run org tasks query '.[] | .title_text' \
   --out gfm \
   --out-theme monokai \
   examples/ARCHIVE_small
@@ -128,5 +137,5 @@ poetry run org tasks query '.[][] | .title_text' \
 7) Export PDF by piping raw binary output to a file
 
 ```bash
-poetry run org tasks query '.[][]' --out pdf examples/ARCHIVE_small > report.pdf
+poetry run org tasks query '.[]' --out pdf examples/ARCHIVE_small > report.pdf
 ```

src/org/commands/agenda/ui.py

@@ -13,7 +13,8 @@
 from rich.text import Text
 
 from org.logic.time import local_now, parse_date_argument, shift_datetimes_by_unit
-from org.query.runner import execute_query_or_raise
+from org.query.engine.errors import QueryParseError, QueryRuntimeError
+from org.query.runner import run_query
 from org.tui.bits import (
     heading_title_to_text,
     task_priority_to_text,
@@ -787,15 +788,16 @@ def _apply_section_query(
     view_name: str,
 ) -> list[Heading]:
     """Apply section query against nodes with $date in context, returning matched nodes."""
-    results = execute_query_or_raise(
-        spec.query,
-        nodes,
-        {"date": day},
-        lambda message: typer.BadParameter(
-            "Agenda filter/order-by query failed "
-            f"(section={spec.name}, view={view_name}): {message}",
-        ),
-    )
+    try:
+        results = run_query(nodes, [spec.query_text], {"date": day})
+    except QueryParseError as exc:
+        raise typer.BadParameter(
+            f"Invalid agenda filter/order-by (view={view_name}, section={spec.name}): {exc}",
+        ) from exc
+    except QueryRuntimeError as exc:
+        raise typer.BadParameter(
+            f"Agenda filter/order-by query failed (section={spec.name}, view={view_name}): {exc}",
+        ) from exc
     return [cast("Heading", node) for node in results]
 
 

src/org/commands/agenda/views.py

@@ -8,21 +8,20 @@
 import typer
 
 import org.config.app
-from org.query.runner import compile_filter_order_query
+from org.query.engine.errors import QueryParseError, QueryRuntimeError
+from org.query.runner import build_filter_order_query_text, run_query
 
 
 if TYPE_CHECKING:
-    from org.query.engine.compiler import CompiledQuery
-
     from .command import AgendaArgs
 
 
 @dataclass(frozen=True)
 class AgendaSectionSpec:
-    """Compiled agenda section specification."""
+    """Agenda section query specification."""
 
     name: str
-    query: CompiledQuery
+    query_text: str
     style: str
     timeline: bool
 
@@ -35,11 +34,6 @@ class AgendaViewContext:
     name: str
 
 
-def _compile_section_query(filter_query: str, order_by: str | None) -> CompiledQuery:
-    """Compile one agenda section filter/order query text."""
-    return compile_filter_order_query(filter_query, order_by)
-
-
 def _fallback_agenda_view() -> org.config.app.AgendaViewConfig:
     """Return built-in fallback agenda view definition."""
     return org.config.app.AgendaViewConfig(
@@ -61,15 +55,16 @@ def _compile_view_section_specs(view: org.config.app.AgendaViewConfig) -> list[A
     section_specs: list[AgendaSectionSpec] = []
     for section in view.sections:
         try:
-            query = _compile_section_query(section.filter, section.order_by)
-        except Exception as err:
+            query_text = build_filter_order_query_text(section.filter, section.order_by)
+            run_query([], [query_text], {})
+        except (QueryParseError, QueryRuntimeError) as err:
             raise typer.BadParameter(
                 f"Invalid agenda filter/order-by (view={view.name}, section={section.name}): {err}",
             ) from err
         section_specs.append(
             AgendaSectionSpec(
                 name=section.name,
-                query=query,
+                query_text=query_text,
                 style=section.style,
                 timeline=section.timeline,
             ),