dev/feature/xyc/memory_0722#3497
Open
Jasonxia007 wants to merge 69 commits into
Open
Conversation
…PR-0) Add foundational context management module under sdk/nexent/core/agents/context/ to support W8/W12/W13 workstreams for progressive component reduction, history projections, and unified context policy. Core components: - ContextItem: Fine-grained context unit with type, authority tier, and fidelity levels - ContextItemHandler: Pluggable handler interface for scoring and reducing context items - ItemHandlerRegistry: Registry mapping context item types to their handlers - 10 built-in handlers (system_prompt, tool, skill, memory, knowledge_base, etc.) - ContextProjector: Converts ContextComponent to ContextItem with proper authority/fidelity - Policy models: SelectionDecision, MemoryDecision for traceable policy decisions - ReductionResult: Immutable record of context reduction operations - Reason codes: 11 standardized codes for decision traceability Testing: - 77 unit tests covering all data models, handlers, and registry - All tests pass with 0.59s execution time This is infrastructure-only (no existing code modified) and provides the foundation for subsequent PRs implementing context projection, policy engines, and handlers.
…PR-1) - Add use_context_items config field (default False for backward compatibility) - Add context_items field to ContextEvidence for traceability - Implement project_context_items() method in ContextManager - Integrate projection logic into assemble_final_context() - Store _source_component reference in metadata for semantic equivalence - Use component.to_messages() to produce formatted text (not JSON dumps) - Add 15 projection tests covering all 7 component types - Add 3 ManagedContextRuntime integration tests - Remove test __init__.py files to fix namespace collision All 112 tests pass. Oracle verified complete and correct.
…e (PR-2) Implement W12 DB History Projection to persist and reconstruct ReAct execution details from conversation history. Database layer: - Add 5 nullable columns: run_id, step_id, tool_call_id, event_time on conversation_message_t and conversation_message_unit_t - Add indexes on run_id and tool_call_id for query performance - Extend conversation_db.py with optional history projection params - Add get_message_units_by_run() and get_max_run_id_for_conversation() SDK layer: - Create HistoryProjector with dependency injection for DB queries - Support 3 projection modes: model_context, resume, chat - Produce HISTORY_TURN, TOOL_CALL_RESULT, WORKING_MEMORY ContextItems - Wire into ContextManager.use_context_items=True path via config Service layer: - Track run_id/step_id/tool_call_id/event_time in _stream_agent_chunks() - Pass tracking fields through save_message/save_message_unit wrappers - Compute run_id before message creation for proper persistence Tests: 21 new tests, 114 total context tests passing, zero regressions.
…ration tests Production wiring: - Thread conversation_id through execution chain: agent_service → create_agent_run_info → AgentConfig → ManagedContextRuntime → assemble_final_context → HistoryProjector - Inject HistoryProjector into ContextManagerConfig before get_or_create_context_manager() so both run-scoped and conversation-scoped ContextManager paths receive it - use_context_items remains False by default (opt-in) Integration tests (7 new, 28 total): - TestChatProjectionCompleteness: unit type coverage, ordering, metadata, source_refs completeness - TestEndToEndIntegration: component + history projection → FinalContext, graceful failure handling, conversation_id gating 121 context tests + 28 HistoryProjector tests passing, zero regressions.
…eline Blocker 0: Call register_all() in ContextManager.__init__ so ItemHandlerRegistry is populated before any projection occurs. Blocker 1: Add to_messages() method to ContextItemHandler base class with default implementation, plus type-specific overrides in HistoryTurnHandler, ToolCallResultHandler, and WorkingMemoryHandler. Blocker 2: Remove mock of ItemHandlerRegistry.get in integration test so the end-to-end path exercises real handlers. 198 tests passing (121 context agent + 77 context module), zero regressions.
…ort regression Move register_all() call from ContextManager.__init__ to lazy initialization in project_context_items() to avoid breaking isolated test loading. Changes: - Remove register_all() from __init__ (line 301-302) - Add _ensure_handlers_registered() helper with idempotency flag - Call helper at start of project_context_items() - Handlers now register only when actually needed This fixes the 124-test regression caused by eager import in __init__ breaking test_agent_context/test_pure_functions.py isolated loading. All 254 context tests now pass (121 context + 77 module + 56 agent_context).
Add comprehensive OTel tracing to provide visibility into the context assembly pipeline: - ManagedContextRuntime: trace prepare_step() and prepare_final_answer() - ContextManager: trace assemble_final_context(), project_context_items(), compress_if_needed(), and _do_generate_summary() with nested LLM spans - HistoryProjector: trace project() with nested DB query spans All spans include relevant attributes (conversation_id, purpose, counts) and span events for key milestones. Backward compatible - spans are no-ops when monitoring is disabled. Testing: All 254 existing tests pass, no syntax errors.
…ct_items attributes Critical fix: - Move 'with self._lock:' block inside trace_operation context manager - Ensures all compression work (including LLM calls) is properly traced - Fixes broken parent-child span hierarchy for compression path Minor improvements: - Add component_count as span attribute to project_context_items - Remove unused 'as span' from assemble_final_context All 254 tests pass.
Previously _last_uncompressed_est was set from input_messages which are already compressed by prepare_step, making est_raw_i always equal est_i and save% structurally ~0%. Now pull the truly-uncompressed token count from ContextManager.get_token_counts()['last_uncompressed'], recorded in compress_if_needed from the raw memory before compression. Co-Authored-By: Claude <noreply@anthropic.com>
Add test scripts and fixtures used during context-manager refactor development. Only .py and .md files are tracked; .out run artifacts remain ignored via .git/info/exclude. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…step.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…_renderer.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The token_estimation module lives at core/utils/, not agents/utils/. From agent_context/ sub-package, three dots (...utils) are needed instead of two (..utils). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…remove monolith Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…ent_context package Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The v2 PreviousCompressor and CurrentCompressor do not fall through from incremental to fresh when LLM returns None - they return PreviousCompressResult/CurrentCompressResult(summary_text=None) immediately. Updated tests P3, C4 to match this behavior. Updated P4 and C6_asymmetry to patch _summarize_pairs with PreviousCompressResult instead of raw tuples. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The context_runtime package lives at core/context_runtime/, not agents/context_runtime/. From agent_context/ sub-package, three dots (...context_runtime) are needed instead of two (..context_runtime). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
- New test_stats_export.py: 21 tests covering all pure functions (100%) - New test_step_renderer.py: 24 tests covering truncation, rendering, compress_history_offline with mock LLM (49%→86%) - Extended test_pure_functions.py: tests for _is_context_length_error, has_invoked_tools, message_role, trim_pairs_to_budget (92%→97%) Overall agent_context package coverage: 71% → 81% Co-Authored-By: Claude <noreply@anthropic.com>
…100%) Co-Authored-By: Claude <noreply@anthropic.com>
Resolved .gitignore conflict by keeping .tokensave entry (used for token storage)
Fixed TypeError when agent_request.history is explicitly None instead of absent. Changed getattr(agent_request, 'history', []) to handle None case properly. This bug prevented message persistence when is_debug=false, causing all event log fields (run_id, step_id, tool_call_id, event_time) to remain NULL.
…loud floating point comparison - Add 5 tests for get_message_units_by_run (with/without run_id, empty result, string coercion, dict mapping) - Add 4 tests for get_max_run_id_for_conversation (found, none, string coercion, zero edge case) - Update stubs to include run_id and step_id attributes - Fix floating point comparison in test_handlers.py using pytest.approx (SonarCloud S1244) This improves patch coverage for conversation_db.py and fixes SonarCloud reliability rating.
Add comprehensive integration test that verifies: - Real LLM model execution through the SDK - OpenTelemetry instrumentation captures spans correctly - Context management with use_context_items=True works end-to-end - LangFuse receives and displays traces properly Test includes: - test_context_items_with_real_model: Basic context management with real model - test_context_compression_with_real_model: Compression with low token threshold - test_history_projector_integration: History projection with real model Requires OPENAI_API_KEY environment variable to run.
- Fix OpenAIModel initialization (add model_id parameter) - Fix ActionStep initialization (use correct parameters) - Fix HistoryProjector initialization (use query_units_fn) - Fix ManagedContextRuntime.prepare_step call (remove conversation_id) - Add local_only marker to pytest.ini - Mark all SDK integration tests as local_only to skip in CI These tests require OPENAI_API_KEY and network access, so they should only run locally, not in CI environments.
- Remove CONTEXT_MANAGEMENT_DEV_PLAN.md from git (keep locally) - Add to .gitignore to prevent future commits This is a temporary development document that should not be tracked in version control.
…emove WorkingMemoryHandler - Merge tool+execution_logs into single tool_call row with JSON unit_content - Remove run_id from conversation_message_t and conversation_message_unit_t - Remove tool_call_id and event_time from conversation_message_unit_t - Rename step_id to step_index for clarity - Replace run_id grouping with message_id-based algorithm in HistoryProjector - Remove WorkingMemoryHandler and WORKING_MEMORY context item type - Remove resume projection from HistoryProjector - Remove dead components field from ManagedRunContext - Fix parameter ordering in create_agent_run_info() - Add TOOL_CALL handling in frontend for both streaming and history - Rewrite migration to be idempotent with proper DROP/ADD COLUMN - Update all affected tests (backend + SDK)
- Add 3 tests for tool_call merge logic in _stream_agent_chunks - Fix string-to-int type mismatch in test_conversation_db.py
…p/nexent into feat/W5_structured_agent_execution_event_log
Update feature branch with mainstream develop branch.
✨ [WIP] Memory implementation Phase 2
✨ [WIP] Memory implementation Phase 4
# Conflicts: # backend/agents/agent_run_manager.py # backend/agents/create_agent_info.py # backend/apps/config_app.py # backend/apps/runtime_app.py # backend/apps/skill_app.py # backend/apps/skill_repository_app.py # backend/consts/model.py # backend/database/conversation_db.py # backend/database/db_models.py # backend/database/model_management_db.py # backend/database/skill_db.py # backend/database/skill_repository_db.py # backend/services/agent_repository_service.py # backend/services/agent_service.py # backend/services/conversation_management_service.py # backend/services/skill_repository_service.py # backend/services/skill_service.py # frontend/app/[locale]/agent-space/components/MineApplyListingModal.tsx # frontend/app/[locale]/agent-space/page.tsx # frontend/app/[locale]/agents/components/agentConfig/SkillBuildModal.tsx # frontend/app/[locale]/agents/components/agentConfig/SkillDetailModal.tsx # frontend/app/[locale]/agents/components/agentConfig/SkillDraftPanel.tsx # frontend/app/[locale]/agents/components/agentInfo/AgentGenerateDetail.tsx # frontend/app/[locale]/chat/internal/chatInterface.tsx # frontend/app/[locale]/chat/streaming/chatStreamHandler.tsx # frontend/app/[locale]/knowledges/components/document/DocumentList.tsx # frontend/app/[locale]/skill-space/components/CreateNewSkillCard.tsx # frontend/app/[locale]/skill-space/components/MineSkillsView.tsx # frontend/app/[locale]/skill-space/components/RepositoryView.tsx # frontend/app/[locale]/skill-space/components/ReviewSkillList.tsx # frontend/app/[locale]/skill-space/components/SkillRepositoryCard.tsx # frontend/app/[locale]/skill-space/components/SkillRepositoryControls.tsx # frontend/app/[locale]/skill-space/components/SkillRepositoryDetailModal.tsx # frontend/app/[locale]/skill-space/components/SkillReviewStatusModal.tsx # frontend/app/[locale]/skill-space/components/skillRepositoryShared.ts # frontend/app/[locale]/skill-space/page.tsx # frontend/const/chatConfig.ts # frontend/hooks/agent/useSkillList.ts # frontend/lib/agentRepositoryMine.ts # frontend/public/locales/zh/common.json # frontend/services/agentConfigService.ts # frontend/services/skillService.ts # frontend/types/agentConfig.ts # frontend/types/skillRepository.ts # sdk/benchmark/acon_eval/run_acon_qa.py # sdk/benchmark/manual_cases/summary_inspector.py # sdk/benchmark/manual_cases/test_benchmark.py # sdk/nexent/core/agents/agent_context/manager.py # sdk/nexent/core/agents/agent_context/summary_step.py # sdk/nexent/core/agents/agent_model.py # sdk/nexent/core/agents/context/__init__.py # sdk/nexent/core/agents/nexent_agent.py # sdk/nexent/core/agents/summary_config.py # sdk/nexent/core/context_runtime/contracts.py # sdk/nexent/core/context_runtime/managed/runtime.py # sdk/nexent/core/tools/search_memory_tool.py # sdk/nexent/core/tools/store_memory_tool.py # test/backend/agents/test_create_agent_info.py # test/backend/app/test_skill_app.py # test/backend/app/test_skill_repository_app.py # test/backend/database/test_conversation_db.py # test/backend/database/test_skill_db.py # test/backend/database/test_skill_repository_db.py # test/backend/services/test_agent_service.py # test/backend/services/test_skill_repository_service.py # test/backend/services/test_skill_service.py # test/backend/utils/test_context_component_types.py # test/sdk/core/agents/test_agent_context/loader.py # test/sdk/core/agents/test_agent_context/unit/test_component_management.py # test/sdk/core/agents/test_context_component.py # test/sdk/core/agents/test_nexent_agent_component_integration.py # test/sdk/core/tools/test_search_memory_tool.py # test/sdk/core/tools/test_store_memory_tool.py
🐛 Bugfix: add pdfinfo in data-process image to resovle files with images
…mory_0722 # Conflicts: # backend/services/agent_service.py # test/backend/services/test_conversation_management_service.py
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| }) | ||
| resp.append(combined) | ||
|
|
||
| return {"result": resp} |
| num=topk, | ||
| return_score=False | ||
| ) | ||
| return {"result": results} |
Comment on lines
+415
to
+418
| return { | ||
| "result": [], | ||
| "error": str(e) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.