Skip to content

Replace hardcoded workflow blocks with config-driven products.yml#19

Merged
mosheabr merged 3 commits intoNVIDIA:mainfrom
sayalinvidia:config-driven-workflows
Apr 16, 2026
Merged

Replace hardcoded workflow blocks with config-driven products.yml#19
mosheabr merged 3 commits intoNVIDIA:mainfrom
sayalinvidia:config-driven-workflows

Conversation

@sayalinvidia
Copy link
Copy Markdown
Collaborator

@sayalinvidia sayalinvidia commented Apr 15, 2026

Teams onboard by adding an entry to products.yml instead of editing workflow files. Both sync-skills.yml and ci.yml now read product definitions from this single config using yq.

Summary

Adds products.yml as the single source of truth for the product catalog
Rewrites sync-skills.yml to loop over products.yml using yq instead of 20+ hardcoded checkout/copy blocks
Rewrites ci.yml to read product definitions from products.yml instead of a hardcoded bash array
Adds continue-on-error: true to the issue-creation step (prevents double-failure when Issues is disabled)

Onboarding experience for teams

Before: edit README + sync-skills.yml + ci.yml (3 files, workflow knowledge required)

After: add an entry to products.yml and open a PR:

  • name: RAG Blueprint
    repo: NVIDIA-AI-Blueprints/rag
    description: "RAG pipeline — deploy, configure, and manage retrieval augmented generation."
    skills:
    • path: skill-source/.agents/skills/
      catalog_dir: rag-blueprint

Test plan

  • Verify yq parsing works on GitHub-hosted runner
  • Manual workflow_dispatch of sync-skills with valid SKILLS_SYNC_PAT
  • Manual workflow_dispatch of ci.yml to verify count checks
  • Confirm draft PR — do not merge until PAT is configured

@sayalinvidia sayalinvidia self-assigned this Apr 15, 2026
@sayalinvidia
Copy link
Copy Markdown
Collaborator Author

Tested locally!
image

Teams onboard by adding an entry to products.yml instead of editing
workflow files. Both sync-skills.yml and ci.yml now read product
definitions from this single config using yq.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sayalinvidia sayalinvidia force-pushed the config-driven-workflows branch from 39e3d31 to 69ad839 Compare April 16, 2026 01:22
The sync-skills workflow pulls actual skill files from product repos,
making the separate count-verification workflow unnecessary. The count
check also fails on symlinked paths (Megatron-Core) and requires a
separate token (PRIVATE_REPO_ACCESS_TOKEN) to maintain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sayalinvidia sayalinvidia marked this pull request as ready for review April 16, 2026 01:27
@sayalinvidia sayalinvidia requested a review from mosheabr as a code owner April 16, 2026 01:27
@mosheabr
Copy link
Copy Markdown
Collaborator

Nice work @sayalinvidiaproducts.yml as single source of truth is a big improvement. The onboarding experience goes from editing 3 files to adding one YAML entry.

Makes sense on removing ci.yml — the count check is cosmetic since the sync workflow already auto-pulls new skills and creates issues on fetch failures. No objection there.

A couple things to check before merging:

1. yq not installed (will break the workflow)

GitHub-hosted runners don't ship with yq. The sync step will fail on first run. Need to add an install step, something like:

- name: Install yq
  run: |
    sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
    sudo chmod +x /usr/local/bin/yq

I see this passed on your test repo (sayali-skills-test) — did that runner have yq pre-installed, or did you test with a different setup?

2. PAT in git clone URL (minor)

x-access-token:${GH_TOKEN} in the clone URL could leak in logs if error output isn't fully suppressed. The 2>/dev/null helps but won't catch everything. Consider using git config credential helper instead.

What's good

1. Add yq version check step — yq is pre-installed on ubuntu-24.04
   (ubuntu-latest) but this fails fast with a clear error if missing.

2. Use git credential helper instead of embedding the PAT in clone
   URLs. Prevents token leaking in error output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sayalinvidia
Copy link
Copy Markdown
Collaborator Author

Thanks for the review @mosheabr
Great feedback!!
Both concerns addressed in the latest push:

  1. yq availability — It worked on the test repo because yq comes pre-installed on ubuntu-latest (ubuntu-24.04) — runner image docs list yq 4.52.5. The test repo used the same GitHub-hosted runner, no custom setup. Added a yq --version verification step anyway so it fails fast with a clear message if the runner image ever drops it.

  2. PAT in clone URL — Good catch. Switched to a git credential helper so the token never appears in URLs or error output. Clone URLs are now plain https://github.com/....

@mosheabr
Copy link
Copy Markdown
Collaborator

Both concerns addressed nicely @sayalinvidia — thanks for the quick turnaround.

The yq --version guard and credential helper fix both look good. LGTM from my side.

@pmcaughan-nv — can you take a look before we merge? You're closer to the workflow infra so would be good to get your sign-off too.

@pmcaughan-nv
Copy link
Copy Markdown
Collaborator

Looks good as an interim solution - I think this could be improved by being more transparent in surfacing errors (clone and checkout commands pipe to null) and it would be more beneficial long-term to have a per-product sync/commit flow, but this would be too much with the interim HITM approach.

I believe this is good enough to have as a starting solution to build off of

@mosheabr mosheabr merged commit 67e18ed into NVIDIA:main Apr 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants