Refactors document loading

VIBES.org

@@ -8944,20 +8944,128 @@ Add the query compilation cache.
 :context: 
 :END:
 
-Add a new package =org.db= that will store the org file repository abstraction.
-In that package, add a repository-pattern abstraction over one or more org files. The abstraction should:
-- automatically load all the documents from files maintaining their last modification time,
-- watch for external file changes and re-load a document as needed (file modification time more recent than currently stored one)
-- allow explicitly reloading a document from file,
-- allow saving the documents explicitly (dump file contents, reload and update the modification state),
-- allow overwriting potentially externally modified document contents without generating errors,
-- allow querying the repository using the query language: =repository.query(query_stages, context)=.
-- cache discovered todo and done states for other commands,
-
-The abstraction should handle file conflicts with explicit errors (attempting to save a file that was modified outside of the CLI, file no longer exists, etc). These errors will be then handled by the commands in various ways (interactive commands should display a prompt asking for confirmation about overwriting or reloading and reapplying an action; non-interactive commands should print the error to the output).
-
-All commands should be updated to instantiate this repository and use its querying capabilities for all of their querying needs.
-All commands should be updated to handle the file errors more or less gracefully.
+1. Add a repository-pattern abstraction over one or more org files to the =org.db= package. The abstraction should:
+   - automatically load all the documents from files maintaining their last modification time,
+   - watch for external file changes and re-load a document as needed (file modification time more recent than currently stored one)
+   - allow explicitly reloading a document from file,
+   - allow saving the documents explicitly (dump file contents, reload and update the modification state),
+   - allow optionally overwriting potentially externally modified document contents without generating errors,
+   - cache discovered todo and done states for the commands,
+   - allow querying the repository using the query language: =repository.query(query_stages, context)=.
+   - provide the means to fetch headings by identity,
+   - provide the means to modify & save specific headings,
+   - provide the means to move headings between documents,
+   - provide the means to add & remove headings from a document.
+
+2. Most if not all functionality of the current =org.db.load= module should be moved to the repository implementation and made uniform.
+
+3. The repository should:
+   - detect which documents have been modified by the commands (by way od the =document.dirty= flag) and which files have been modified outside of the CLI (stored & file modification times). This detection does not need to be done async, it's OK if the detection is carried out on all read & write operations.
+   - handle file conflicts with explicit errors (attempting to save a file that was modified outside of the CLI, file no longer exists, etc). These errors will then be handled by the commands in various ways (interactive commands should display a prompt asking for confirmation about overwriting or reloading and reapplying an action; non-interactive commands should print the error to the output).
+   - ideally, saving the files would be an explicit operation that automatically derives which documents to save (automatically detect which documents need saving and if their associated files were modified outside of the CLI). No implicit document saving should be performed by the respository, it should be an explicit operation that the commands run on the repository.
+   - ideally, when querying the repository data, outside-CLI modifications should be detected, and files that can be reloaded without conflicts (no in-app modifiations) should be automatically and implicitly reloaded.
+   - upon saving a modified document, reload the file contents of that document, so that the modification state is reset.
+
+4. All commands should be updated to instantiate this repository and use its querying capabilities for all of their querying needs. Ideally, the app would load the configuration file, pass it to the command, build the command arguments, create a repository based on the config and arguments, run the command logic using the config, arguments and the org repository, and finally use the repository to query for the data, modify it and save the files if needed. Interactive commands should maintain the repository in their state and rely on its saving & reloading features.
+
+5. All commands should be updated to handle the file errors more or less gracefully.
+**** DONE Code review comment
+:PROPERTIES:
+:context: [[file:~/projects/software/org-cli/src/org/commands/board/actions.py::repository: OrgRepository | None = None]]
+:END:
+:LOGBOOK:
+- State "DONE"       from "TODO"       [2026-06-25 Thu 19:25]
+:END:
+
+Filename: /home/k/projects/software/org-cli/src/org/commands/board/actions.py
+
+#+begin_example
+    repository: OrgRepository | None = None
+
+#+end_example
+
+Looks like this is always available and should never be a =None=. Please adjust that type and refactor the places that check for the existance of board session repository (=_require_repository= etc).
+**** DONE src/org/commands/agenda/actions.py
+:PROPERTIES:
+:context: 
+:END:
+:LOGBOOK:
+- State "DONE"       from "TODO"       [2026-06-25 Thu 19:25]
+:END:
+
+In file src/org/commands/agenda/actions.py around line 90, near the ~class AgendaSession:~
+
+#+begin_example
++    repository: OrgRepository | None = None
+
+#+end_example
+
+This also should always be available. Adjust that for all the commands that use this pattern and make sure that the associated =_require_repository= functions are not necessary.
+
+**** TODO Code review comment
+:PROPERTIES:
+:context: [[file:~/projects/software/org-cli/src/org/db/repository.py::raise HeadingAmbiguousError(
+ f"--parent is ambiguous, multiple headings with title '{selector}'",
+ )]]
+:END:
+
+Filename: /home/k/projects/software/org-cli/src/org/db/repository.py
+
+#+begin_example
+            raise HeadingAmbiguousError(
+                f"--parent is ambiguous, multiple headings with title '{selector}'",
+            )
+
+#+end_example
+
+There's no guarantee that this selector is a title and comes from =--parent=. Please reword to just say that multiple headings match selector.
+**** TODO src/org/commands/agenda/command.py
+:PROPERTIES:
+:context: 
+:END:
+
+In file src/org/commands/agenda/command.py around line 140, near the ~def run_agenda(args: AgendaArgs, config: org.config.app.AppConfig) -> None:~
+
+#+begin_example
++            render=ui.RenderContext(
++                color_enabled=color_enabled,
++                done_states=done_states,
++                todo_states=todo_states,
+
+#+end_example
+
+I think it makes more sense for the RenderContext to also take the repository and use the todo and done states from there. This way, if the files change outside of the app and additional states are configured, the render config will immediately pick it up without having to restart the CLI.
+**** TODO src/org/commands/agenda/command.py
+:PROPERTIES:
+:context: 
+:END:
+
+In file src/org/commands/agenda/command.py around line 136, near the ~def run_agenda(args: AgendaArgs, config: org.config.app.AppConfig) -> None:~
+
+#+begin_example
++            nodes=nodes,
+
+#+end_example
+
+Both the Agenda and Board commands operate on subsets of all the nodes from the repository. Would it make sense to introduce a =RepositoryView= abstraction? I.e. a class that behaves exactly like a repository with file reloading logic, but it would also implicitly filter down the results based on a configured query. This way the agenda can receive a repository view of nodes filtered down by the CLI switches and then create further filtered-down views for the different days and sections of the agenda.
+
+**** TODO tests/cli_common/test_filter_construction.py
+:PROPERTIES:
+:context: 
+:END:
+
+In file tests/cli_common/test_filter_construction.py around line 389, near the ~def test_load_and_process_data_logs_query_context(caplog: pytest.LogCaptureFixtu~
+
+#+begin_example
+-        load_and_process_data(args, AppConfig(config_path=".org-cli.yaml"))
++        build_repository_query_plan(
++            args,
++            AppConfig(config_path=".org-cli.yaml"),
++            include_ordering=True,
+
+#+end_example
+
+The test name no longer matches what it checks.
 
 *** CANCELLED Use Textual header & footer widgets in the interactive commands
 :LOGBOOK:

pyproject.toml

@@ -104,7 +104,7 @@ lines-after-imports = 2
     "SLF001",  # private member access is tolerated in tests/support code
 ]
 
-"src/org/pipeline/format.py" = [
+"src/org/db/format.py" = [
     "S603",    # pandoc subprocess invocation is an intentional feature
     "S112",    # best-effort serialization intentionally skips unreadable attributes
 ]

src/org/commands/agenda/actions.py

@@ -9,17 +9,22 @@
 from typing import TYPE_CHECKING
 
 import typer
+from org_parser.document import Heading
 from org_parser.time import Clock, Timestamp
 
 from org.commands.tasks.capture import TasksCaptureArgs, capture_task
 from org.commands.tasks.common import (
     duration_to_org_text,
     iter_descendants,
-    load_document,
     parse_clock_duration,
-    save_document,
     todo_states_for_heading,
 )
+from org.db.errors import RepositoryError
+from org.db.repository import (
+    OrgRepository,
+    build_repository_query_plan,
+    cli_error_from_repository_error,
+)
 from org.logic.archive import archive_heading_subtree_and_save
 from org.logic.edit import edit_heading_subtree_in_external_editor
 from org.logic.search import filter_nodes_by_search
@@ -30,7 +35,7 @@
     resolve_heading_locator,
 )
 from org.logic.time import advance_timestamp_by_repeater, local_now, set_timestamp_fields
-from org.pipeline.load import load_and_process_data
+from org.query.engine.errors import QueryParseError, QueryRuntimeError
 
 from .ui import (
     AgendaColumnWidths,
@@ -49,7 +54,7 @@
 if TYPE_CHECKING:
     from collections.abc import Callable
 
-    from org_parser.document import Document, Heading
+    from org_parser.document import Document
 
     from org.config.app import AppConfig
 
@@ -81,9 +86,20 @@ class AgendaSession:
     search_text: str
     view_ctx: AgendaViewContext
     app_config: AppConfig
+    repository: OrgRepository
     run_external: Callable[[Callable[[], None]], None] | None = None
 
 
+@dataclass(frozen=True)
+class AgendaSessionData:
+    """Prepared agenda session inputs shared with the app layer."""
+
+    nodes: list[Heading]
+    render: RenderContext
+    view_ctx: AgendaViewContext
+    repository: OrgRepository
+
+
 def refresh_session(
     session: AgendaSession,
     preserve_identity: HeadingLocator | None,
@@ -194,15 +210,21 @@ def _shift_timestamp_by_hours(timestamp: Timestamp, hour_delta: int) -> None:
     set_timestamp_fields(timestamp, shifted_start, shifted_end)
 
 
-def _save_document_changes(document: Document) -> None:
+def _save_document_changes(session: AgendaSession, document: Document) -> None:
     """Persist one mutated document to disk."""
     logger.info("Saving agenda edit file: %s", document.filename)
-    save_document(document)
+    session.repository.save_document(document.filename or "")
 
 
 def _reload_session_nodes(session: AgendaSession) -> None:
     """Reload nodes through standard processing pipeline after mutations."""
-    nodes, _, _ = load_and_process_data(session.args, session.app_config)
+    plan = build_repository_query_plan(session.args, session.app_config, include_ordering=True)
+    repository = session.repository
+    results = repository.query(plan.stages, plan.context)
+    nodes = [value for value in results if isinstance(value, Heading)]
+    limit = session.args.max_results
+    if limit is not None:
+        nodes = nodes[session.args.offset : session.args.offset + limit]
     session.all_nodes = nodes
     session.nodes = filter_nodes_by_search(nodes, session.search_text)
 
@@ -236,9 +258,13 @@ def archive_selected_task(session: AgendaSession) -> None:
         return
     session.status_message = ""
     try:
-        archive_result = archive_heading_subtree_and_save(row.node, {})
-    except typer.BadParameter as err:
-        session.status_message = str(err)
+        archive_result = archive_heading_subtree_and_save(
+            row.node,
+            {},
+            session.repository,
+        )
+    except (RepositoryError, typer.BadParameter) as err:
+        session.status_message = str(cli_error_from_repository_error(err))
         return
     preserve_identity = heading_locator(archive_result.heading)
     _reload_session_nodes(session)
@@ -348,7 +374,7 @@ def apply_state_change_with_value(session: AgendaSession, new_state: str) -> Non
             heading.id,
             heading.deadline,
         )
-    _save_document_changes(heading.document)
+    _save_document_changes(session, heading.document)
     preserve_identity = heading_locator(heading)
     _reload_session_nodes(session)
     refresh_session(session, preserve_identity)
@@ -366,7 +392,7 @@ def apply_shift_date(session: AgendaSession, *, day_delta: int) -> None:
         session.status_message = status
         return
     heading = row.node
-    _save_document_changes(heading.document)
+    _save_document_changes(session, heading.document)
     preserve_identity = heading_locator(heading)
     _reload_session_nodes(session)
     refresh_session(session, preserve_identity)
@@ -384,7 +410,7 @@ def apply_shift_time(session: AgendaSession, *, hour_delta: int) -> None:
         session.status_message = status
         return
     heading = row.node
-    _save_document_changes(heading.document)
+    _save_document_changes(session, heading.document)
     preserve_identity = heading_locator(heading)
     _reload_session_nodes(session)
     refresh_session(session, preserve_identity)
@@ -413,9 +439,9 @@ def apply_refile_with_value(session: AgendaSession, destination_input: str) -> N
         session.status_message = "Refile cancelled"
         return
     try:
-        destination_document = load_document(destination_path)
-    except typer.BadParameter as err:
-        session.status_message = str(err)
+        destination_document = session.repository.get_document(destination_path)
+    except (RepositoryError, typer.BadParameter) as err:
+        session.status_message = str(cli_error_from_repository_error(err))
         return
     heading = row.node
     source_document = heading.document
@@ -429,9 +455,9 @@ def apply_refile_with_value(session: AgendaSession, destination_input: str) -> N
         session.status_message = "Task already in destination file"
         return
     _move_heading_to_document(heading, destination_document)
-    _save_document_changes(destination_document)
+    _save_document_changes(session, destination_document)
     if source_document is not destination_document:
-        _save_document_changes(source_document)
+        _save_document_changes(session, source_document)
     preserve_identity = heading_locator(heading)
     _reload_session_nodes(session)
     refresh_session(session, preserve_identity)
@@ -461,7 +487,7 @@ def apply_clock_entry_with_value(session: AgendaSession, duration_input: str) ->
     )
     duration_text = duration_to_org_text(duration)
     row.node.clock_entries.append(Clock(timestamp=timestamp, duration=duration_text))
-    _save_document_changes(row.node.document)
+    _save_document_changes(session, row.node.document)
     preserve_identity = heading_locator(row.node)
     _reload_session_nodes(session)
     refresh_session(session, preserve_identity)
@@ -539,11 +565,11 @@ def apply_capture_task(session: AgendaSession, template_name: str) -> None:
         return
     capture_result.heading.scheduled = scheduled
     try:
-        _save_document_changes(capture_result.document)
+        _save_document_changes(session, capture_result.document)
         _reload_session_nodes(session)
         refresh_session(session, heading_locator(capture_result.heading))
-    except typer.BadParameter as err:
-        session.status_message = str(err)
+    except (RepositoryError, QueryParseError, QueryRuntimeError, typer.BadParameter) as err:
+        session.status_message = str(cli_error_from_repository_error(err))
         return
     session.status_message = f"Task captured and scheduled for {scheduled}"
 
@@ -595,16 +621,14 @@ def apply_search_text(session: AgendaSession, search_text: str) -> None:
 def create_agenda_session(
     args: AgendaArgs,
     config: AppConfig,
-    nodes: list[Heading],
-    render: RenderContext,
-    view_ctx: AgendaViewContext,
+    data: AgendaSessionData,
 ) -> AgendaSession:
     """Create interactive session state for agenda."""
     session = AgendaSession(
         args=args,
-        all_nodes=list(nodes),
-        nodes=nodes,
-        render=render,
+        all_nodes=list(data.nodes),
+        nodes=data.nodes,
+        render=data.render,
         start_date=resolve_agenda_start_date(args.date),
         days=args.days,
         now=local_now(),
@@ -616,8 +640,9 @@ def create_agenda_session(
         scroll_offset=0,
         status_message="",
         search_text="",
-        view_ctx=view_ctx,
+        view_ctx=data.view_ctx,
         app_config=config,
+        repository=data.repository,
     )
     refresh_session(session, None)
     return session