Repository navigation
Add llm evaluation against human accessibility audits - #5
Open
reiganarmstrong wants to merge 152 commits into
Open
reiganarmstrong wants to merge 152 commits into
reiganarmstrong wants to merge 152 commits into
Conversation
Document the CLI workflow, eligibility classifications, current Luna proof-of-concept status, and remaining work. Expand the domain glossary for the human-audit reference workbook and non-model eligibility classes.
Use Human finding mapping names for saved artifacts, local lookup paths, and manifest fields. Require approved mappings throughout execution and reporting, and remove tests for unsupported review behavior. BREAKING CHANGE: Remove old CLI aliases and storage names. Saved runs using the previous format require fresh preparation.
Move normalized Programmatic findings into artifacts and add concise run READMEs with clear preview and live instructions. Keep mapping preparation focused on CSVs and instructions, rebuilding approval evidence from the original audit and snapshots.
Move detailed evaluator documentation into its package README and provide a complete quick start in the root README. Clarify the human audited Excel file terminology in documentation and local variables, and include the editor environment setting.
reiganarmstrong
marked this pull request as ready for review
October 25, 2026 20:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a CLI eval tool that compares findings from the va-dat tool and those in a provided human accessibility audit. This is especially useful for comparing the price to performance ratios between various llms in the va-dat tool. Details of the CLI tool are documented in the following links: