refactor(memory): run Dream and Skill evolution in agent sessions

[xiami762]

Summary

• Rework persistent Memory into explicit User, Global, Project, and Daily layers backed by Markdown files under the configured Flocks data directory.
• Keep memory_search as the retrieval interface while using the standard read/write/edit/glob/grep tools to maintain curated Memory.
• Replace the separate Dream/Learn pipelines with one hidden self-improve Agent and one integrated prompt that can improve Memory or at most one Flocks-managed Skill.

Key Changes

Memory hierarchy and lifecycle

• Use <data>/memory as the canonical Memory root (normally ~/.flocks/data/memory; this is not ~/.flocks/memory):
  • USER.md — user identity/context, communication preferences, working style, and technical level.
  • MEMORY.md — global environment/tool knowledge, lessons and corrections, and external references.
  • projects/<project_id>/MEMORY.md — project context, project-specific lessons and corrections, and references.
  • daily/YYYY-MM-DD.md — lifecycle-owned append-only evidence.
• Inject Memory into normal Sessions in this order: User, Global, then the registered current Project.
• Keep Daily files out of normal prompt injection and reject Agent write/edit operations against them; only lifecycle code may append Daily entries.
• Remove the /new Session-to-Markdown snapshot hook.
• Make the Memory Flush and self-improvement prompts choose exactly one canonical destination:
  • user information/preferences -> USER.md
  • project-only declarative knowledge -> Project MEMORY.md
  • cross-project environment knowledge/lessons/references -> Global MEMORY.md
  • repeatable procedures -> Skill
  • secrets, guesses, transient/one-off/rediscoverable facts, duplicates, or weak evidence -> no change
• Require read-before-edit and protect curated files from accidental overwrite.

Search and indexing

• Make memory_search global across selectable sources:
  • memory searches User, Global, all Project, and Daily Memory.
  • session searches persisted user/assistant Session text across projects.
• Use hybrid vector + FTS retrieval for Memory when embeddings are available, with FTS fallback; Session History remains FTS-only.
• Scope the rebuildable Memory index by (scope, scope_id, path).
• Maintain Session History search incrementally on message create/update/delete and reconcile historical records when required.
• Reconcile indexed Memory files before search because standard file tools and external editors can bypass MemoryManager; file hashes keep the check cheap when content is unchanged.

Unified Dream / self-improvement

• Remove the retired self-enhance Agent, separate /learn command, Skill-learning trigger pipeline, proposal persistence, and dual evolution state.

• Add one hidden disposable self-improve Agent running through the normal SessionLoop with memory_enabled=False.

• Trigger self-improvement only:

  • manually with /dream for the current Global or Project target
  • on the existing scheduled Dream cadence (24-hour per target, checked every 30 minutes)

• Do not trigger Dream at startup.

• Feed the Agent incremental user/assistant Session evidence, bounded tool traces, target-mapped Daily evidence, and a complete-entry-trimmed Skill catalog. Curated Memory is inspected from disk instead of duplicated into the prompt.

• Give the Agent read, glob, grep, bash, and Skill-loading support, with writes restricted to the exact curated Memory paths and user Skill root.

• Require an explicit create/edit/no-op decision. A run may create or evolve at most one complete Skill, and may edit only Skills marked:

  metadata:
    managed_by: flocks

• Preserve built-in, project, source-controlled, and otherwise unmanaged Skills; validate and restore guarded files if the Agent violates the write contract.

• Advance the checkpoint only after the Agent run, output validation, and required index synchronization succeed. Skip Memory index synchronization when no Memory file changed.

• Show manual /dream progress and return a structured result with the target, evidence reviewed, and changed Memory/Skill files. Scheduled runs remain silent.

Tests and cleanup

• Add isolated coverage for Memory scopes, flush routing, file guards, search/index lifecycle, evolution prompts, scheduling, checkpoints, Skill protection, commands, and manual Dream status.
• Remove legacy script-style Memory/OpenClaw/external integration smoke tests that touched real providers or the user's configured data directory and could report false-positive pytest results.

Impact Scope

• User-visible behavior: /dream now reports progress and its final changes. Memory files use the hierarchy above, and /learn, memory_get, and memory_write are no longer available.
• Compatibility / migration: Existing Markdown Memory remains the source of truth. Missing User/Global/Project files are created from the new templates. Derived search indexes are rebuildable; no Session content migration is required.
• Configuration / environment: No new required environment variables, secrets, or deployment steps. Memory still honors the configured data root. Evolution scheduling remains under Memory evolution configuration.
• Dependencies: No new third-party dependencies.
• Performance / resources: Message mutations maintain Session FTS incrementally. Each Memory search performs a hash-based file reconciliation. Scheduled self-improvement may invoke the configured model once per eligible target and interval, with bounded input.
• Security / permissions: Daily files are Agent-read-only, curated writes are path-guarded, prompt evidence is treated as untrusted and redacted, and Skill writes are limited to Flocks-managed user Skills.

Business Logic to Review

• Project registration and routing: a Session with a registered project receives that Project Memory; otherwise it receives only User and Global Memory.
• The one-destination Memory decision rules shared by flush and self-improvement prompts, especially Global vs Project routing and the no-op boundary.
• Daily append-only enforcement and curated Memory overwrite protection across file tools and sandbox path resolution.
• Memory index scope migration/rebuild, per-search reconciliation, and Session FTS consistency across message CRUD and historical backfill.
• Dream evidence cursors, target mapping, input budgets, Session/Daily overlap handling, checkpoint ordering, retries, and partial-failure behavior.
• Hidden Session lifecycle and cleanup, recursion prevention, tool permissions, Skill snapshot/restore validation, and the one-Skill-per-run invariant.
• Manual /dream status cleanup on both success and failure.

Why This Approach

• Markdown plus standard file tools keeps Memory transparent, reviewable, and editable without introducing a separate CRUD abstraction.
• A single self-improvement Agent matches the actual decision boundary: evidence may imply a Memory update, a reusable Skill, or no change.
• Reusing the normal SessionLoop gives Dream the same prompt/tool/runtime contracts as other Agents while disabling recursive Memory injection.
• Search indexes and checkpoints remain derived operational state; curated Markdown stays authoritative and recoverable.

Test Plan

• .venv/bin/ruff check passes for changed Python files except flocks/server/app.py, which retains pre-existing repository E402 findings.
• .venv/bin/pytest tests/memory tests/command/test_evolution_commands.py tests/server/test_input_dispatcher.py tests/session/test_status.py tests/sandbox/test_sandbox_file_tools.py tests/tool/test_memory_file_write.py -q — 126 passed.
• cd webui && npm run lint
• cd webui && npm run test:run -- src/components/common/SessionChat.test.ts src/pages/Session/index.test.tsx src/features/session-chat/sseActions.test.ts — 218 passed.
• cd webui && npm run build

Compatibility, Migration & Rollback

• This intentionally removes memory_get, memory_write, /learn, the previous automatic Skill-learning triggers, and the legacy Session Memory hook. Integrations should use memory_search, standard file tools, and /dream.
• The default location remains ~/.flocks/data/memory; this PR does not move Memory to ~/.flocks/memory.
• No new secrets, environment variables, or package migrations are required.
• On rollback, revert the application changes and restart Flocks. Curated Markdown files remain usable; derived indexes/checkpoints can be discarded and rebuilt.