Skip to content

PoC: AI Skills - #25976

Draft
thykel wants to merge 101 commits into
devfrom
ai-assistance/health-check
Draft

thykel wants to merge 101 commits into
devfrom
ai-assistance/health-check

Conversation

@thykel

@thykel thykel commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Note

Proof of concept. It is not ready for a merge.

What this is

An AI Assistance menu in the app header, on every page. It offers skills from assistants that an admin sets up. Every skill opens a conversation, uses the permissions of the user who clicked, and writes nothing until that user applies a proposal.

  • Where skills are offered. A skill is offered on "Work packages" or "Everywhere". On a work package (full view, or split view in lists, boards and calendars), every matching skill is offered. On other pages, only "Everywhere" skills, without proposals: their context is the user and, on a project page, the project. A chat belongs to its place (a work package, a project, or neither), and History lists all chats with their place. The permission is "Use AI Assistance" in the project of the place, or in any project on pages without one.
  • Skills. A skill is a label, an icon, a skill prompt, a proposal scope and conditions. What it does is in its prompt. With "Answer at once when opened" (on by default), opening the skill starts a chat with a start line ("Tom started Refresh work package"), and the skill answers at once to a fixed request, for example "Refresh work package" or "Write a test plan". "Run again" starts it again in the same chat. Without the setting, the skill opens an empty chat for the user's own questions, for example "Chat". The user can ask follow-up questions in every skill.
  • Answers and proposals. Every answer is text with any number of proposals, shown as cards. A proposal is a finding (a severity, an area, evidence, and an optional change) or a plain change with a short text. A finding without evidence is dropped. The user applies or dismisses each card, with an optional reason. The model gets the earlier turns, with a short list of what it proposed before.
  • Consistency checks by the model. There are no built-in rules. The seeded "Refresh work package" prompt asks for findings: comments, meeting outcomes or relations that the description, subject, status, assignee, accountable or priority does not show yet, the state of the descendants, duplicates, linked pull requests and inactivity. For this, the context also holds the linked pull requests, the default-status flag, the last activity, the descendants' parent and update time, and which relations are closed. Every call uses temperature 0.2, so the answers stay similar from run to run.
  • Read-only MCP tools. For data outside the context, the model can call OpenProject's own read-only MCP tools, for example to search or count other work packages, or to look up users, versions, types and custom field values. The tools run in-process, through RubyLLM's tool loop, as the user who asked, so each tool's visibility rules apply. There is no token and no HTTP call, and write tools are never offered. Each skill can opt out with "Use OpenProject MCP tools" (on by default). The code limits the calls: an identical call gets the earlier result, every call after 20 calls or after three repeated calls gets "answer now", and the answer fails after 25. While an answer loads, the dialog lists the tool calls so far. "Details" shows each call with its arguments, time and result size.
  • MCP tool changes (also for external MCP clients): search_work_packages takes open (true or false), a list of status ids and a list of assignee ids; the new count_work_packages counts work packages, optionally per status, type, priority, project or assignee, and lists every requested assignee, also with 0; search_users takes a group, by id or name; search_custom_field_items also returns the values of plain list fields.
  • Model. Llm::Runtime (from dev) resolves the model of the "AI Assistance" feature: the assistant's own model, then the feature binding on the LLM connection, then the connection default. Tests on the demo data: Claude Sonnet 4.5 and 5.5 used the tools well and answered every test question correctly. gpt-4o-mini often skipped the tools and answered "none", so it is not enough for skills with tools.
  • Applying. Changes are Automations actions (subject, description, status, priority, dates, assignee, accountable, comment). They are applied through Automations::UpdateWorkPackageService in strict mode, with a lockVersion check and a journal cause. The activity entry names the assistant and the user who applied the change, for example "proposed by OP assistant and applied by Jane Doe". For the owner of the chat, it also links to the chat. Other users see the text without the link, because chats are private. A chat whose skill is no longer offered opens read-only. Each skill has a proposal scope: Changes, Comment only, or Read-only.
  • Admin. Administration → Artificial Intelligence (AI) → Assistants lists one section per assistant (global, group, personal, each with an icon and a link to its owner), with its skills as rows. Assistants and skills can be copied: a copy is inactive until someone switches it on. An assistant has an assistant prompt that all its skills share, and each skill has its own skill prompt, both stored in the database and editable. The code adds the output rules (answer format, language, findings and plain proposals, what the proposal scope allows, how to use the tools, and a guard against instructions in the context and the tool results), so a prompt cannot break the answer contract. The skill page shows the full system prompt, which changes while the admin types and follows the tools checkbox. The icon of a skill is chosen from a menu of icons, next to its label. The basic data seeder creates the global "OP assistant" with two skills: Refresh work package and Chat.
  • Menu. The menu has one section for each assistant, with the assistant's name and a dimmed owner icon as its heading: personal assistants first, then group assistants, then global ones. History is at the end.
  • One answer at a time. A double click or a second tab does not start a second answer: a unique index allows one pending answer per chat, starting a skill that is still answering opens that chat, the reply job runs once per answer, and a result after the time limit is dropped.
  • Skill conditions. A skill can be limited to work package types, statuses and projects, and to the user's roles in the project, as Automations conditions are. The menu offers only skills whose conditions match.
  • Personal assistants. Users with the global permission "Manage own AI assistants" create their own assistants and skills at My account → AI assistants. A personal assistant shows only in its owner's menu and has no model of its own: it uses the "AI Assistance" feature binding or the connection default. The admin list shows every assistant with its owner, and the admin can edit, deactivate and delete personal assistants.
  • Group assistants. An assistant has an optional owner, a principal: none (global), a user (personal) or a group. Only the group's direct members see a group assistant. The development data seeds a "COMMS" team with the admin user as its member, the team's "COMMS assistant" ("Write a test plan"), and "Tom's assistant", a personal assistant of the admin user ("My next steps").
  • Transparency. Each answer has a "Details" timeline with the prompt, the answer, errors, and each tool call with its arguments and full result, and the model's reasoning before tool calls. Answers name people instead of guessing pronouns. Chats are listed in History.
  • Privacy. Chats and personal assistants belong to one user, and only that user sees them. They are deleted with the user, and group assistants with their group (Principals::DeleteJob, covered by its integration spec).

Plan and review split

The concept, the decisions and the proposed split into stacked pull requests are in docs/development/concepts/ai-assistance/README.md. Each commit is one step of that split. Steps 21–26 replaced runs, rules and the three skill types with conversations. For review, they are best folded into the earlier steps that they change. The MCP tool commits come after them, and the MCP tool changes can be reviewed on their own. The header button uses only extension points that a module has (a view hook, a dynamic Stimulus controller, and Angular services from the plugin context), so it can move into a plugin as it is.

How to try

A fresh install of this branch, a pull preview included, seeds a demo (temporary, not for dev):

  • OPAssistant (global): "Chat", offered everywhere.
  • COMMS assistant (the COMMS team): "Assign to the least busy developer". The team is Tom Admin (the admin user) and Kabiru, Bruno, Judith and Ihor Developer, with 3, 1, 0 and 2 open work packages in the demo project.
  • My assistant (Tom Admin): "Refresh work package". The demo project's conference work package has comments and meeting outcomes that its subject, description and fields do not show yet.

Steps:

  1. On a pull preview, the ai_assistance feature flag is on. Elsewhere, enable it in Administration → Experimental.
  2. Configure an LLM connection with a default chat model, and enable the AI features. For good answers with tools, bind a Sonnet-class model to the "AI Assistance" feature of the connection.
  3. Sign in as the admin, open the demo project's conference work package (Add i18n-js to OpenProject #2), click the AI Assistance icon in the header, and choose "Refresh work package". Then try "Assign to the least busy developer".
  4. On any other page, choose "Chat". To see proposals, add a comment that changes the scope, or close the work package while a child is still open.
  5. On a work package, choose "Chat" and ask, for example, "How many open work packages of this project does each person have?" The loading text lists the tool calls, and "Details" shows them after the answer.
  6. To change the prompts or copy a skill, go to Administration → Artificial Intelligence (AI) → Assistants.
  7. For personal assistants, add "Manage own AI assistants" to a global role (for all users: the Standard global role), then open My account → AI assistants.
  8. For group assistants, choose a group as the owner when creating an assistant in Administration.

Not in this pull request

  • Enterprise gating, also for the chat's MCP tools. The Enterprise mcp_server feature and the admin's MCP tool switches apply to external MCP clients only.
  • Removing external images from AI answers (still open). format_text keeps images from any host, and the image policy is img-src *, so an injected instruction could make an answer load a URL that carries data. The planned fix runs the answer through Sanitize again with relative image sources only.
  • Tool results in later turns: a follow-up question calls the tools again. search_time_entries is not offered.
  • A trigger model: chats do not record a trigger yet, and Automations cannot start a skill.
  • Assistants for subgroups (membership is not inherited), sharing a personal assistant, and a project-level menu.
  • Agent skills, such as "Break down into tasks" or a coding agent. The design for them (where the line between core and an agent is, delegated tokens, attribution and containment) is in 2026-10-14-agents.md.
  • Browser specs for the header button and the chat dialog. The root seeder spec is skipped on this branch, because the demo data changes its counts. Model, service, request and component specs cover the chat, the proposals and the admin pages.

🤖 Generated with Claude Code

thykel and others added 15 commits October 12, 2026 22:23
The Automations update service drops actions that fail validation and
applies the rest. Callers that need all-or-nothing pass strict: true.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One private conversation per user and work package. Answers can propose
changes, which the user applies like the fixes of a finding. The prompt
is an instance setting, not code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
2 Warnings
⚠️ Attention developer & reviewer: Files with potential user references found:

  • modules/llm/db/migrate/20261012120001_create_ai_assistants.rb (migration with user reference)

Please make sure:

  1. You've added proper relationships (has_many, belongs_to, …) and their inverse in your models
  2. You deal with model destruction dependencies: dependent: :destroy or dependent: :delete_all
  3. You add behavior and tests for the Principal::DeleteJob (app/workers/principals/delete_job.rb)
  4. You replace references to users with deleted user in Principal::ReplaceReferencesService (app/services/principals/replace_references_service.rb) by adding to the replacements initailizer (config/initializers/replace_references_service.rb) for the core, or using the replace_principal_references helper in modules/plugins.
  5. You test the above behaviors with an integration test (e.g., like this one to confirm deletion of users is possible.

This helps prevent dangling database objects when users are deleted and resulting bugs.

⚠️ Attention developer & reviewer: migration looks up an index by hardcoded name:

  • modules/llm/db/migrate/20261015130000_add_one_pending_answer_per_ai_chat.rb

remove_index ..., name: and rename_index MAY fail when the live name differs from
the canonical one (for example, old databases pgloader prefixes MySQL indexes with idx_<oid>_; pre-squash
databases may still use the original migration name).

Please use Migration::Utils instead:

require Rails.root.join("db/migrate/migration_utils/utils")

class SomeMigration < ActiveRecord::Migration[8.0]
  include Migration::Utils

  def up
    remove_index_on :workflows, "wkfs_role_type_old_status", %w[role_id type_id old_status_id]
    rename_index_on :workflows, "wkfs_role_type_old_status", "wkfs_role_type_variant_old_status",
                    %w[role_id type_id old_status_id]
  end
end

Those helpers match the given name, or the name with pgloader prefix, then fall back to columns.

Generated by 🚫 Danger

thykel and others added 2 commits October 12, 2026 22:33
…s to AI::Assistance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Seed QA assistance with Check health, Chat and Generate test cases, with prompts
- Share the LLM call, the event log, the user locale, polling and the feature guard
- Move lost references, comment-only and the proposal state into AI::Proposal
- One list of changeable attributes (ChangeOptions::ATTRIBUTE_KEYS)
- Run Comment after Notify, so Notify cannot replace a proposed comment
- Neutral apply errors for chats, and a translated review state in Details
- Remove unused code, columns (uuid, error_message) and the failed run status
- Use Primer utilities instead of inline styles where they exist
- Share the admin form views, validate the skill type in the model
- Update the concept doc and outdated specs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Deploying openproject with ⚡ PullPreview

Field Value
Latest commit 8f294ad
Job deploy
Status 🗑️ Preview destroyed
Preview URL Destroyed

View logs

thykel and others added 6 commits October 12, 2026 23:06
…nsistent

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run service spec created a skill with an unknown key, which the skill_key
validation now refuses. The run spec called terminal?, which the cleanup removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Users with the global permission manage_own_ai_assistants create their own
assistants and skills at My account > AI assistants. A personal assistant shows
only in its owner's menu, after the global ones, and uses the default model of
the LLM connection. It is deleted with its owner.

The admin list shows every assistant with its owner and filters by owner. The
admin and the My account controllers share their actions and views.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread app/controllers/admin/ai_assistants_controller.rb Fixed
thykel and others added 2 commits October 13, 2026 18:32
Add the associations on User that Danger asks for, as for AI text transform
runs, and cover the deletion in the Principals::DeleteJob integration spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rules report pull request findings in that area, but the review schema
did not offer it to the model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thykel and others added 2 commits October 15, 2026 18:56
A pronoun guessed from a name can be wrong, so the output rules of
every skill ask the model to refer to people by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…heck

dev adds a migration with the version 20261012120000, which the AI
assistants migration had. The AI one is now 20261012120001.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Temporary, for previews, and not meant for dev:

- The development people (departments, users, Tom Admin, the COMMS
  team and the AI assistants) are seeded also outside development.
  They need no factory_bot, unlike the rest of the development data.
- After the demo data, the COMMS developers' workloads and a Refresh
  work package demo are seeded: the conference work package with the
  comments and meeting outcomes that its subject, description and
  fields do not show yet.
- Pull previews switch on the ai_assistance feature flag. The LLM
  connection is still configured by hand.
- The root seeder spec is skipped, because its counts no longer match.

A fresh seed without development data was checked: all assistants,
the team, the workloads and the Refresh case are there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Warning

Flaky specs

  • rspec ./spec/features/components/lookbook_accessibility_spec.rb[1:14:1]
🤖 Ask Copilot to investigate

Copy the prompt below into a new comment on this PR to delegate the investigation to GitHub Copilot. It will look into the flakiness and open a separate pull request with you as reviewer.

@copilot The following spec(s) are flaky in CI (first seen on PR #25976, linked for reference only):

- `rspec ./spec/features/components/lookbook_accessibility_spec.rb[1:14:1]`

Treat this as a standalone task, unrelated to PR #25976. Create a new branch from origin/dev and open a new pull request targeting dev — do not stack it on PR #25976 or reuse that branch.

Follow the playbook in docs/development/testing/handling-flaky-tests/README.md to find the root cause and fix the underlying race — do not skip, delete, or weaken the spec to make it pass; disabling is a last resort per the playbook, and only with a bug ticket. Verify the fix by running the spec(s) repeatedly (e.g. `script/bulk_run_rspec --run-count 10`).

If you cannot reproduce the flake or are not confident in a fix after reasonable investigation, do not fabricate a change or skip the spec to force CI green. Instead, leave the pull request in draft and document what you tried, the suspected cause, and any leads in its description, then assign @thykel to take over.

Once the fix is verified, title the PR after the spec(s) it fixes, and use the PR description to explain the root cause, how the change resolves it, and the before/after results. Label the PR `flaky-spec`, assign @thykel, and request a review from @thykel.
On every commit, set @thykel as the sole co-author with a `Co-authored-by:` trailer (use their GitHub no-reply email so it links to their account), so it is traceable who dispatched the fix.

thykel and others added 15 commits October 18, 2026 22:14
GLM 5.3 often gives a finding a severity and the area "none". The
answer format dropped such findings, also with a valid change, so the
number of proposals changed from run to run.

- Derive the area of a finding from its changed fields, else use the
  model's area; a finding with the area "none" is kept without one.
- Tell the model that a finding uses a listed area, and that the newest
  comment or meeting wins when two disagree.
- Log each dropped proposal with its reason in the "stored" event.
- Ask the model once more, with the reasons, when an answer loses all
  its proposals.
- Store proposals in a fixed order within a severity: by the first
  changed field, then by area.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the separate count_work_packages MCP tool with a generic,
optional grouping feature of SearchTool. A search tool can declare
groups (`group :status, column: ..., association: ...` or
`group :assignee, with: :method`). It then accepts `group_by` and
returns `groups: [{name, count}]` next to `total`. The new
`level_of_detail: "count"` returns only the total and the groups.

search_work_packages declares the groups status, type, priority,
project and assignee. The assignee logic is unchanged. The tool
descriptions, the AI Assistance tool rule, the seeded prompt and the
docs now name search_work_packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…heck

Register the ai_assistance feature flag and LLM feature in the
openproject-llm engine, now that dev moved the LLM connection there,
and build proposal actions from the STI automation action templates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves the AI Assistance app code, specs, factory, spec support and its
five migrations under modules/llm, next to the LLM connection it uses.
Class names, table names, route helpers and i18n keys stay the same.

The core-file hunks move into the engine and module files:

- routes, the admin and My account menus, both permissions, the
  inflections and the header button hook now live in the engine or in
  modules/llm/config
- the Principal and User associations become engine patches
- the en.yml additions move to modules/llm/config/locales/en.yml
- BasicData::AiAssistantSeeder becomes BasicData::Llm::AiAssistantSeeder,
  with its data and the member role permission in the module common.yml

Staying in core: the journal cause, the Automations actions, the MCP
tool changes, and the demo-data seeders that the core seeders call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- define ai_assistants on User and Group: a has_many patched onto
  Principal does not reach its already loaded STI subclasses
- inherit the module seeder from BasicData::ModelSeeder
- put require_module_spec_helper on its own line in the moved specs
- expect the search_work_packages tool title in the chat poll spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dev renamed core-stimulus/helpers/my-work-entry-menus to
my-time-management-entry-menus, which broke the frontend build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The admin pages list and open only global and group assistants. A
personal assistant belongs to its user, who manages it in My account.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restores the Angular op-wp-ai-assistance-slot from before 578d590:
a labelled "AI Assistance" dropdown in the full view, and an icon
button in the split view. It loads the menu from /ai_assistance/menu
with the work package as the place.

The app header button is removed: its view hook, component, Stimulus
controller and engine wiring. The menu is now offered on work packages
only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A skill that answers at once opens with its answer and a "Chat about
this" button. The button shows the question field and the New chat and
Send buttons. Once the user asked a question, and for chat skills, the
dialog shows the full form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Put on hold, Escalate and Assign to me: automations with a button,
seeded with the rest of the AI Assistance demo on every install of
this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- sort the AI Assistance locale keys
- split the methods that RuboCop finds too large into smaller ones,
  without changing their behavior
- use params.expect for the record ids in the assistant controllers
- break the multi-line render calls in the chat templates

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thykel thykel changed the title PoC: AI Assistants PoC: AI Skills Oct 25, 2026
@github-actions

Copy link
Copy Markdown

Warning

Flaky specs

  • rspec ./modules/resource_management/spec/features/work_package_allocated_time_field_spec.rb[1:2]
🤖 Ask Copilot to investigate

Copy the prompt below into a new comment on this PR to delegate the investigation to GitHub Copilot. It will look into the flakiness and open a separate pull request with you as reviewer.

@copilot The following spec(s) are flaky in CI (first seen on PR #25976, linked for reference only):

- `rspec ./modules/resource_management/spec/features/work_package_allocated_time_field_spec.rb[1:2]`

Treat this as a standalone task, unrelated to PR #25976. Create a new branch from origin/dev and open a new pull request targeting dev — do not stack it on PR #25976 or reuse that branch.

Follow the playbook in docs/development/testing/handling-flaky-tests/README.md to find the root cause and fix the underlying race — do not skip, delete, or weaken the spec to make it pass; disabling is a last resort per the playbook, and only with a bug ticket. Verify the fix by running the spec(s) repeatedly (e.g. `script/bulk_run_rspec --run-count 10`).

If you cannot reproduce the flake or are not confident in a fix after reasonable investigation, do not fabricate a change or skip the spec to force CI green. Instead, leave the pull request in draft and document what you tried, the suspected cause, and any leads in its description, then assign @thykel to take over.

Once the fix is verified, title the PR after the spec(s) it fixes, and use the PR description to explain the root cause, how the change resolves it, and the before/after results. Label the PR `flaky-spec`, assign @thykel, and request a review from @thykel.
On every commit, set @thykel as the sole co-author with a `Co-authored-by:` trailer (use their GitHub no-reply email so it links to their account), so it is traceable who dispatched the fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants