Skip to content

Implement source field tracking and update command for workflow provenance#1175

Closed
Copilot wants to merge 6 commits intomainfrom
copilot/add-source-field-functionality
Closed

Implement source field tracking and update command for workflow provenance#1175
Copilot wants to merge 6 commits intomainfrom
copilot/add-source-field-functionality

Conversation

Copy link
Contributor

Copilot AI commented Oct 2, 2025

Overview

This PR implements the source field proposal for tracking workflow provenance and enabling seamless updates from source repositories. Workflows installed from packages now automatically track their origin, allowing users to update them when new versions become available.

Key Changes

1. Automatic Source Field Population

When adding workflows from packages, the system now automatically populates a source field in the frontmatter:

---
on: push
source: githubnext/agentics abc123def456 samples/workflows/researcher.md
---

The format is: org/repo <ref> path/to/workflow.md

This happens transparently when using:

gh aw add weekly-research -r githubnext/agentics

2. New gh aw update Command

The update command allows updating workflows that have source information:

# Update all workflows with source
gh aw update

# Update specific workflow
gh aw update weekly-research

# Create PR with updates
gh aw update --pr

Update Algorithm:

  • Fetches the latest version from the source repository
  • Merges frontmatter from source while preserving local markdown content
  • Preserves the source field itself for continued tracking
  • Optionally creates a pull request with the changes

3. Enhanced gh aw status Command

The status command now displays source tracking information:

Workflow Status
Name              | Installed | Up-to-date | Status  | Source                         | Update Available
----------------- | --------- | ---------- | ------- | ------------------------------ | ----------------
weekly-research   | Yes       | Yes        | active  | githubnext/agentics abc123...  | No

The command checks:

  • Whether workflows have source information
  • If updates are available by comparing with installed package versions
  • Package installation status

Implementation Details

Source Field Generation

The generateSourceField() function creates the source string by:

  1. Extracting org/repo from the package path
  2. Reading the commit SHA from package metadata
  3. Computing the relative path to the workflow file within the package

Update Merge Strategy

The update process uses a conservative merge strategy:

  1. Frontmatter: Updated from source (except the source field itself)
  2. Markdown content: Preserved from local file (user customizations kept)
  3. Compilation: Automatically recompiles after update

This ensures that configuration updates flow from upstream while preserving local documentation customizations.

Testing

Added comprehensive unit tests:

  • TestGenerateSourceField: Validates source field generation logic
  • TestAddSourceToWorkflow: Tests adding source to workflow frontmatter
  • TestFindWorkflowsWithSource: Validates finding workflows with source
  • TestCheckUpdateAvailable: Tests update availability detection

Example Workflow

# Install a package with workflows
gh aw install githubnext/agentics

# Add a workflow (source field automatically added)
gh aw add weekly-research

# Check status to see source and update availability
gh aw status

# Later, when updates are available
gh aw update weekly-research

Files Changed

  • cmd/gh-aw/main.go: Added update command to CLI
  • pkg/cli/commands.go: Source field helpers and status enhancements
  • pkg/cli/update_command.go: Update command implementation (new)
  • pkg/cli/source_field_test.go: Unit tests (new)

Breaking Changes

None. The source field is optional and only populated for workflows added from packages. Existing workflows continue to work without modification.

Related Issue

Closes #[issue-number]

Original prompt

This section details on the original issue you should resolve

<issue_title>proposal: source</issue_title>
<issue_description># source

Tracking the provenance of an installed workflow to allow seamless updates.

frontmatter

The frontmatter field source contains information about where the current workflow was installed from. It allows to perform updates and detects changes.

---
source: githubnext/agentics <ref> samples/workflows/researcher.md
---

The source is a single string formatted as follows (similar to go.mod in golang)

<repository>@<ref>/<path>

gh aw add

The add command should populate the source field when executing.

gh aw update

The update command updates all or some agentic workflows. It only applies
to workflows with the source information.

# updates all workflows
gh aw update
# updates a single workflows
gh aw update <workflowid>
# gives a preview of the update
gh aw update --dry-run
# create pull request with changes
gh aw update --pr

When updating a script, the command follows this algorithm:

resolve the updated sha (either latest, or highest release)
download latest version source text
parse current front matter and update source reference in it
perform a 3-way merge of the markdown body using git. 
let user finish merge as needed.

If the merge is successful a "compile" should be done and the changes/additions staged (like "gh aw add"). That is it should match "gh aw add".

gh aw status

The status command should check if workflows with source have been modified or if updates are available.

includes

Include local paths are converted into global routes (same syntax as source) and resolved at compile time (local cache in .aw/includes/...)

Previius tracking strategy

The previous tracking strategy is deprecated and removed with no backward compatibility. </issue_description>

Comments on the Issue (you are @copilot in this section)

@dsyme > "gh aw update: When updating a script, the command follows this algorithm"

If the merge is successful a "compile" should be done and the changes/additions staged (like "gh aw add"). That is it should match "gh aw add" - if we decide not to do automatic staging we should remove it for "gh aw add" too

This means a "--pr" flag should also be supported on "gh aw update" </comment_new>
<comment_new>@pelikhan
The current plan is that it would update the entire file, frontmatter and markdown as a merge to retain modifications.

This would replace how we track the origin of script installed today.</comment_new>

Fixes #1005

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 4 commits October 2, 2025 21:42
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add source field handling for workflows Implement source field tracking and update command for workflow provenance Oct 2, 2025
Copilot AI requested a review from pelikhan October 2, 2025 21:54
@pelikhan
Copy link
Contributor

pelikhan commented Oct 2, 2025

@copilot

Update the reference format to <owner>/<repo>@<ref>/<path>

@pelikhan
Copy link
Contributor

pelikhan commented Oct 2, 2025

let's just restart it from the epic.

@pelikhan pelikhan deleted the copilot/add-source-field-functionality branch October 2, 2025 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

proposal: source

2 participants