implement task evaluation framework - #187

Open

Kavya Sree Kaitepalli (KavyaSree2610) wants to merge 21 commits into

mainfrom

kkaitepalli/eval-agent

Commits on Aug 31, 2026

feat: implement evaluation task framework with feedback mechanism
Kavya Sree Kaitepalli
committed
73de905
Add SweBenchVerifiedTask for evaluation and improve EvalTask interface
Kavya Sree Kaitepalli
committed
c9d7b9b
Add unit tests for evaluation tasks and orchestrator
Kavya Sree Kaitepalli
committed
0070b61
refactor: update training function references to use run_training_loop and add training_iterations parameter
Kavya Sree Kaitepalli
committed
90ed586
modify file structure
Kavya Sree Kaitepalli
committed
b2b08ff
Implement task registry for EvalTask instances
Kavya Sree Kaitepalli
committed
a3237ae

Commits on Sep 2, 2026

Refactor evalTask module and improve functionality
Kavya Sree Kaitepalli
committed
4a952ab
Update development mode package installation to include training dependencies
Kavya Sree Kaitepalli
committed
085f462
Add unit tests
Kavya Sree Kaitepalli
committed
25e0b91
Refactor evalTask and related modules: implement build_feedback method, remove analyzer, and update tests
Kavya Sree Kaitepalli
committed
00d095b

Commits on Sep 3, 2026

Refactor
Kavya Sree Kaitepalli
committed
021c7ef
Resolve copilot comments
Kavya Sree Kaitepalli
committed
998040a

Commits on Sep 7, 2026

Single training should be validated by all the instances in the task
Bala (0xba1a)
committed
db258da

Commits on Sep 8, 2026

Clean and Prune code to make it simple
Bala (0xba1a)
committed
8905390
Clean docstrings, fix minor bugs and rewrite test cases
Bala (0xba1a)
committed
fdef49a
Increase the eval agent timeout
Bala (0xba1a)
committed
b0b968f

Commits on Sep 9, 2026

Update Azure CLI installation step to include package update
Kavya Sree Kaitepalli
committed
8ace0db
Enhance Azure CLI installation step with keyring setup and version check
Kavya Sree Kaitepalli
committed
955a40a
Update logs to reach appropriate log files and fix the shared repo permission issue
Bala (0xba1a)
committed
b91b333

Commits on Sep 16, 2026

Update requirements and enhance MemoryTool for read-only mode
Kavya Sree Kaitepalli
committed
087c67e
Refactor eval method signatures to include training_repo_dir parameter for enhanced evaluation context and improve prompt
Kavya Sree Kaitepalli
committed
432e103