Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/release-plugin.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ on:
description: "Run even if no reference changes detected"
type: boolean
required: false

default: false

permissions:
Expand Down
25 changes: 25 additions & 0 deletions context/sources-json-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,22 @@ The pipeline:
}
```

For repos that publish an npm package, add `npm_package`:

```json
{
"id": "fastedge-sdk-js",
"github_url": "https://github.com/G-Core/FastEdge-sdk-js",
"ref": "latest-release",
"trigger": "schedule",
"contract_path": "fastedge-plugin-source/",
"intent_dir": "agent-intent-skills/fastedge-sdk-js/",
"generator_agent": "claude",
"reviewer_agent": "openai",
"npm_package": "@gcoredev/fastedge-sdk-js"
}
```

| Field | Type | Required | Description |
| ----------------- | ------ | -------- | ----------- |
| `id` | string | yes | Unique identifier. Used in baseline tags and traceability frontmatter. Kebab-case. |
Expand All @@ -87,6 +103,13 @@ The pipeline:
| `intent_dir` | string | yes | Path to synthesis intent files in the plugin repo, relative to `sources.json`. Must end with `/`. Directory must exist. |
| `generator_agent` | string | yes | AI agent that generates content. See **Agent Values** below. |
| `reviewer_agent` | string | yes | AI agent that reviews content. **Must differ from `generator_agent`**. |
| `npm_package` | string | no | npm package name published by this repo (e.g. `@gcoredev/fastedge-sdk-js`). When present, the pipeline resolves the latest published version via `npm view <package> version` and substitutes that version into source material before the generator and reviewer run, so generated blueprints always reference the current release. Must be a valid npm package name (scoped `@scope/name` or unscoped `name`, lowercase). Absence means no version substitution occurs. |

### npm_package semantics

When `npm_package` is set, `run_agents` calls `npm view <package> version` once before the parallel entry loop. The resolved version is forwarded to both the generator and reviewer as `--sdk-version`. If the lookup fails (registry unreachable, package not found, npm unavailable), the run aborts with an error — it does **not** silently continue with stale source versions, because doing so would advance the baseline tag and suppress retries on the next scheduled run.

**Known gap**: change detection is based on git commit SHA. A new npm publish without a corresponding git commit does not trigger regeneration. See the open issue for tracking options.

### Removed in v2

Expand Down Expand Up @@ -261,6 +284,7 @@ Validation runs in two places:
4. All `contract_path` values end with `/`
5. All `intent_dir` values end with `/` and the directory exists
6. No v1 fields (`updates[]`, `sparse_paths`) are present
7. `npm_package`, when present, is a valid npm package name (`@scope/name` or `name`, lowercase, no whitespace)

A validation failure aborts the workflow before any sparse checkout begins.

Expand Down Expand Up @@ -334,6 +358,7 @@ Each source repo has a `fastedge-plugin-source/CONVENTIONS.md` documenting the n
- `intent_dir` must end with `/` and directory must exist (Rule 5)
- `generator_agent` must differ from `reviewer_agent` (Rule 3)
- No v1 fields (`updates[]`, `sparse_paths`) allowed (Rule 6)
- `npm_package`, when present, must be a valid npm package name (Rule 7)

**To add webhook support** in the source repo's release workflow:

Expand Down
18 changes: 16 additions & 2 deletions scripts/sync/invoke-agent.sh
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ PROMPT
}

_run_generator() {
local REFERENCE_FILE="" SECTION="" SOURCE_DIR="" SOURCE_FILES="" REPO_ID="" REF="" COMMIT="" OUTPUT_FILE="" INTENT_FILE=""
local REFERENCE_FILE="" SECTION="" SOURCE_DIR="" SOURCE_FILES="" REPO_ID="" REF="" COMMIT="" OUTPUT_FILE="" INTENT_FILE="" SDK_PACKAGE="" SDK_VERSION=""

while [[ $# -gt 0 ]]; do
case "$1" in
Expand All @@ -363,6 +363,8 @@ _run_generator() {
--commit) COMMIT="$2"; shift 2 ;;
--output-file) OUTPUT_FILE="$2"; shift 2 ;;
--intent-file) INTENT_FILE="$2"; shift 2 ;;
--sdk-package) SDK_PACKAGE="$2"; shift 2 ;;
--sdk-version) SDK_VERSION="$2"; shift 2 ;;
*) echo "ERROR: Unknown argument: $1" >&2; exit 1 ;;
esac
done
Expand Down Expand Up @@ -391,6 +393,12 @@ _run_generator() {
local source_content
source_content=$(load_source_files "$SOURCE_DIR" "$SOURCE_FILES")

# Substitute pinned SDK version with the resolved release version
if [[ -n "$SDK_PACKAGE" && -n "$SDK_VERSION" ]]; then
source_content=$(sed "s|\"${SDK_PACKAGE}\": *\"[^\"]*\"|\"${SDK_PACKAGE}\": \"^${SDK_VERSION}\"|g" <<< "$source_content")
echo "INFO: Substituted ${SDK_PACKAGE} version → ^${SDK_VERSION} in source material" >&2
Comment thread
godronus marked this conversation as resolved.
fi

local today
today=$(date -u +"%Y-%m-%d")

Expand Down Expand Up @@ -487,14 +495,16 @@ PROMPT
}

_run_reviewer() {
local INPUT_FILE="" SOURCE_DIR="" SOURCE_FILES="" OUTPUT_FILE=""
local INPUT_FILE="" SOURCE_DIR="" SOURCE_FILES="" OUTPUT_FILE="" SDK_PACKAGE="" SDK_VERSION=""

while [[ $# -gt 0 ]]; do
case "$1" in
--input-file) INPUT_FILE="$2"; shift 2 ;;
--source-dir) SOURCE_DIR="$2"; shift 2 ;;
--source-files) SOURCE_FILES="$2"; shift 2 ;;
--output-file) OUTPUT_FILE="$2"; shift 2 ;;
--sdk-package) SDK_PACKAGE="$2"; shift 2 ;;
--sdk-version) SDK_VERSION="$2"; shift 2 ;;
*) echo "ERROR: Unknown argument: $1" >&2; exit 1 ;;
esac
done
Expand All @@ -517,6 +527,10 @@ _run_reviewer() {
local source_content
source_content=$(load_source_files "$SOURCE_DIR" "$SOURCE_FILES")

if [[ -n "$SDK_PACKAGE" && -n "$SDK_VERSION" ]]; then
source_content=$(sed "s|\"${SDK_PACKAGE}\": *\"[^\"]*\"|\"${SDK_PACKAGE}\": \"^${SDK_VERSION}\"|g" <<< "$source_content")
fi

local prompt
prompt=$(_build_reviewer_prompt "$generated_content" "$source_content")

Expand Down
18 changes: 17 additions & 1 deletion scripts/sync/process-repos.sh
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,17 @@ run_agents() {
repo_id=$(jq -r ".repos[$idx].id" "$SOURCES_FILE")
contract_path=$(jq -r ".repos[$idx].contract_path" "$SOURCES_FILE")
intent_dir=$(jq -r ".repos[$idx].intent_dir" "$SOURCES_FILE")
local npm_package sdk_version=""
npm_package=$(jq -r ".repos[$idx].npm_package // empty" "$SOURCES_FILE")
if [[ -n "$npm_package" ]]; then
sdk_version=$(npm view "$npm_package" version 2>/dev/null) || sdk_version=""
Comment thread
godronus marked this conversation as resolved.
if [[ -n "$sdk_version" ]]; then
echo "INFO: Resolved npm version for ${npm_package}: ${sdk_version}" >&2
else
echo "ERROR: Could not resolve npm version for ${npm_package} — aborting to avoid advancing baseline with stale versions" >&2
return 1
Comment thread
godronus marked this conversation as resolved.
fi
fi

local manifest_file="${checkout_dir}/${contract_path}manifest.json"

Expand Down Expand Up @@ -214,16 +225,20 @@ run_agents() {
echo "MISSING_INTENT=true" >> "$result_file"
fi

local section_args=() intent_args=() source_files_args=()
local section_args=() intent_args=() source_files_args=() sdk_args=()
[[ -n "$section" ]] && section_args=(--section "$section")
[[ -n "$intent_file" ]] && intent_args=(--intent-file "$intent_file")
[[ -n "$source_files_list" ]] && source_files_args=(--source-files "$source_files_list")
if [[ -n "$npm_package" && -n "$sdk_version" ]]; then
sdk_args=(--sdk-package "$npm_package" --sdk-version "$sdk_version")
fi

bash "${SCRIPT_DIR}/invoke-agent.sh" \
--role generator \
"${section_args[@]}" \
"${intent_args[@]}" \
"${source_files_args[@]}" \
"${sdk_args[@]}" \
Comment thread
godronus marked this conversation as resolved.
--reference-file "$reference_file" \
--source-dir "$checkout_dir" \
--repo-id "$repo_id" \
Expand All @@ -240,6 +255,7 @@ run_agents() {
--input-file "$gen_output" \
--source-dir "$checkout_dir" \
"${source_files_args[@]}" \
"${sdk_args[@]}" \
--output-file "$rev_output" < /dev/null || {
echo "ERROR: Reviewer failed for '${source_key}'" >&2
echo "FAILED=1" >> "$result_file"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"version": "2.0",
"repos": [
{
"id": "repo-a",
"github_url": "https://github.com/example/repo-a",
"ref": "latest-release",
"trigger": "schedule",
"contract_path": "fastedge-plugin-source/",
"intent_dir": "agent-intent-skills/fastedge-test/",
"generator_agent": "claude",
"reviewer_agent": "codex",
"npm_package": "Invalid Package Name!"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"version": "2.0",
"repos": [
{
"id": "repo-a",
"github_url": "https://github.com/example/repo-a",
"ref": "latest-release",
"trigger": "schedule",
"contract_path": "fastedge-plugin-source/",
"intent_dir": "agent-intent-skills/fastedge-test/",
"generator_agent": "claude",
"reviewer_agent": "codex",
"npm_package": "@gcoredev/fastedge-sdk-js"
}
]
}
174 changes: 174 additions & 0 deletions scripts/sync/tests/test-process-repos.sh
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,180 @@ fi
# Restore stub so any future tests added below see the expected default
run_agents() { OVERALL_VERDICT="ACCEPT"; CHANGED_FILES="plugins/test/ref.md"; }

# ── (8) npm_package: sdk args forwarded to both generator and reviewer ────────
#
# Exercises run_agents directly with a real manifest + npm_package in sources.json.
# A stub npm returns a known version; a stub invoke-agent.sh records which roles
# received --sdk-package. Asserts both generator and reviewer calls carry it.

eval "$_real_run_agents"

_ra8_work="${TMPWORK}/run-agents-npm"
_ra8_checkout="${_ra8_work}/checkout"
_ra8_staging="${_ra8_work}/staging"
_ra8_mocks="${_ra8_work}/mocks"
_ra8_contract="fastedge-plugin-source/"
mkdir -p "${_ra8_checkout}/${_ra8_contract}" "$_ra8_staging" "$_ra8_mocks"

cat > "${_ra8_checkout}/${_ra8_contract}manifest.json" <<'MANIFEST'
{
"target_mapping": {
"key-a": {"reference_file": "docs/a.md"}
},
"sources": {
"key-a": {"files": ["src/a.rs"]}
}
}
MANIFEST

cat > "${_ra8_work}/sources.json" <<EOF
{
"version": "2.0",
"repos": [
{
"id": "test-repo-npm",
"github_url": "https://github.com/test/test-repo-npm",
"ref": "latest-release",
"trigger": "schedule",
"contract_path": "${_ra8_contract}",
"intent_dir": "agent-intent/",
"generator_agent": "claude",
"reviewer_agent": "kimi",
"npm_package": "@gcoredev/fastedge-sdk-js"
}
]
}
EOF

# Stub invoke-agent.sh: records role name to SDK_ARGS_LOG only when both
# --sdk-package and --sdk-version match the expected values exactly.
cat > "${_ra8_mocks}/invoke-agent.sh" <<'STUB'
#!/usr/bin/env bash
role="" output_file="" sdk_pkg="" sdk_ver=""
while [[ $# -gt 0 ]]; do
case "$1" in
--role) role="$2"; shift 2 ;;
--output-file) output_file="$2"; shift 2 ;;
--sdk-package) sdk_pkg="$2"; shift 2 ;;
--sdk-version) sdk_ver="$2"; shift 2 ;;
*) shift ;;
esac
done
if [[ "$sdk_pkg" == "@gcoredev/fastedge-sdk-js" && "$sdk_ver" == "3.1.4" ]]; then
echo "$role" >> "${SDK_ARGS_LOG}"
fi
[[ "$role" == "generator" ]] && echo "# Generated" > "$output_file"
[[ "$role" == "reviewer" ]] && printf 'VERDICT=ACCEPT\n\nLooks good.\n' > "$output_file"
exit 0
STUB
chmod +x "${_ra8_mocks}/invoke-agent.sh"

# Stub npm: returns a known version
cat > "${_ra8_mocks}/npm" <<'NPM'
#!/usr/bin/env bash
echo "3.1.4"
NPM
chmod +x "${_ra8_mocks}/npm"

export SDK_ARGS_LOG="${_ra8_work}/sdk-args.log"
export SOURCES_FILE="${_ra8_work}/sources.json"
RESOLVED_REF="v3.1.4"
COMMIT="abc123"
SCRIPT_DIR="$_ra8_mocks"
_ra8_old_path="$PATH"
export PATH="${_ra8_mocks}:$PATH"

run_agents 0 "$_ra8_staging" "$_ra8_checkout" 2>/dev/null
rc8=$?

export PATH="$_ra8_old_path"
SCRIPT_DIR="$_ra_orig_script_dir"
export SOURCES_FILE="${TMPWORK}/sources.json"

if [[ "$rc8" -eq 0 ]] \
&& grep -qF "generator" "${SDK_ARGS_LOG}" \
&& grep -qF "reviewer" "${SDK_ARGS_LOG}"; then
pass "(8) npm_package: sdk args forwarded to both generator and reviewer"
else
fail "(8) npm_package: sdk args forwarded to both generator and reviewer" \
"exit=${rc8}, log=$(cat "${SDK_ARGS_LOG:-/dev/null}" 2>/dev/null)"
fi

run_agents() { OVERALL_VERDICT="ACCEPT"; CHANGED_FILES="plugins/test/ref.md"; }

# ── (9) npm lookup failure → run_agents returns 1 ────────────────────────────
#
# Same setup as (8) but the stub npm exits 1 (simulating registry/auth failure).
# Asserts run_agents hard-fails so the baseline is never advanced with stale output.

eval "$_real_run_agents"

_ra9_work="${TMPWORK}/run-agents-npm-fail"
_ra9_checkout="${_ra9_work}/checkout"
_ra9_staging="${_ra9_work}/staging"
_ra9_mocks="${_ra9_work}/mocks"
_ra9_contract="fastedge-plugin-source/"
mkdir -p "${_ra9_checkout}/${_ra9_contract}" "$_ra9_staging" "$_ra9_mocks"

cat > "${_ra9_checkout}/${_ra9_contract}manifest.json" <<'MANIFEST'
{
"target_mapping": {
"key-a": {"reference_file": "docs/a.md"}
},
"sources": {
"key-a": {"files": ["src/a.rs"]}
}
}
MANIFEST

cat > "${_ra9_work}/sources.json" <<EOF
{
"version": "2.0",
"repos": [
{
"id": "test-repo-npm-fail",
"github_url": "https://github.com/test/test-repo-npm-fail",
"ref": "latest-release",
"trigger": "schedule",
"contract_path": "${_ra9_contract}",
"intent_dir": "agent-intent/",
"generator_agent": "claude",
"reviewer_agent": "kimi",
"npm_package": "@gcoredev/fastedge-sdk-js"
}
]
}
EOF

# Stub npm: fails (simulates registry or auth error)
cat > "${_ra9_mocks}/npm" <<'NPM'
#!/usr/bin/env bash
exit 1
NPM
chmod +x "${_ra9_mocks}/npm"

export SOURCES_FILE="${_ra9_work}/sources.json"
RESOLVED_REF="v1.0.0"
COMMIT="abc123"
SCRIPT_DIR="$_ra9_mocks"
_ra9_old_path="$PATH"
export PATH="${_ra9_mocks}:$PATH"

run_agents 0 "$_ra9_staging" "$_ra9_checkout" 2>/dev/null
rc9=$?

export PATH="$_ra9_old_path"
SCRIPT_DIR="$_ra_orig_script_dir"
export SOURCES_FILE="${TMPWORK}/sources.json"

if [[ "$rc9" -eq 1 ]]; then
pass "(9) npm lookup failure → run_agents returns 1"
else
fail "(9) npm lookup failure → run_agents returns 1" "exit=${rc9}"
fi

run_agents() { OVERALL_VERDICT="ACCEPT"; CHANGED_FILES="plugins/test/ref.md"; }

# ── Summary ───────────────────────────────────────────────────────────────────

echo ""
Expand Down
6 changes: 6 additions & 0 deletions scripts/sync/tests/test-validate-sources.sh
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,12 @@ run "Rule 5: nonexistent intent_dir fails" \
run "Rule 6: v1 fields (updates[], sparse_paths) rejected" \
"$FIXTURES/rule6-v1-fields.json" 1 "Rule 6"

# Rule 7: npm_package format
run "Rule 7: valid scoped npm_package passes" \
"$FIXTURES/rule7-valid-npm-package.json" 0 "validation passed"
run "Rule 7: invalid npm_package name fails" \
"$FIXTURES/rule7-bad-npm-package.json" 1 "Rule 7"

# Missing argument
label="no argument: exits with usage error"
no_arg_output=$(bash "$VALIDATE" 2>&1) && no_arg_exit=0 || no_arg_exit=$?
Expand Down
Loading
Loading