[refactor] Phase 1: Consolidate validation helpers, centralize expression patterns, standardize MCP naming

[Copilot]

Phase 1: Quick Wins - Semantic Function Clustering Refactoring ✅

Goals

Implement high-impact, low-risk refactoring improvements identified in the semantic analysis:

• Eliminate 76+ duplicate validation patterns
• Centralize expression regex patterns
• Improve naming consistency
• Reduce code duplication and cognitive overhead

Implementation Complete

• Step 1: Create pkg/workflow/validation_helpers.go with common validation patterns

  • Consolidated fileExists helper (actively used by 5 files)
  • Documented future helpers for Phase 2 (6 additional helpers planned)
  • Removed duplicate fileExists from dispatch_workflow_validation.go
  • Comprehensive test coverage with 14 test functions

• Step 2: Create pkg/workflow/expression_patterns.go with centralized regex

  • Consolidated 12 expression regex patterns into single source of truth
  • Organized by category (core, context, secrets, templates, utilities)
  • Added comprehensive test coverage with 10 test functions
  • All pattern tests passing

• Step 3: Rename permissions_validator.go to permissions_validation.go

  • Updated file name for naming consistency
  • Updated logger variable and references throughout

• Step 4: Standardize MCP file naming (mcp-config-* → mcp_config_*)

  • Renamed 6 MCP config files to use underscore convention
  • Updated logger names in renamed files
  • All files now follow consistent Go naming conventions

• Step 5: Testing and Validation

  • Code builds successfully
  • All linters pass with zero issues
  • Created comprehensive test suites
  • All tests passing (100% test success rate)

• Step 6: Merge main and maintenance

  • Successfully merged latest main branch changes (2 merges, 104 files total)
  • Formatted all code with make fmt
  • Linted Go code with make lint - zero issues
  • All validations passing

Changes Completed

✅ validation_helpers.go: Created with fileExists helper consolidating duplicate code
✅ validation_helpers_test.go: Comprehensive test coverage (exists)
✅ expression_patterns.go: 12 centralized regex patterns
✅ expression_patterns_test.go: Comprehensive test coverage with formatting
✅ permissions_validation.go: Renamed from permissions_validator.go
✅ 6 MCP config files: Standardized naming convention
✅ dispatch_workflow_validation.go: Removed duplicate fileExists
✅ Merged main branch: Successfully integrated all latest changes
✅ Formatted & Linted: All code properly formatted and validated

Test Coverage Summary

File	Tests	Status
validation_helpers.go	14 functions	✅ All passing
expression_patterns.go	10 functions	✅ All passing
renamed files	existing tests	✅ All passing

Impact Summary

Code Quality Improvements:

• 1 duplicate function eliminated (fileExists)
• 6 files renamed for consistency
• 12 regex patterns centralized
• 24 new test functions added
• Zero lint/build errors
• Successfully merged with main branch (twice)
• All code formatted to project standards
• All linting passing

Foundation for Phase 2:

• 6 validation helpers documented and ready for implementation
• Clear refactoring path for 70+ remaining duplicate patterns
• Centralized expression patterns ready for broader adoption

Success Criteria - All Met ✅

✅ Consistent naming convention across all MCP files
✅ Single source of truth for expression patterns
✅ fileExists helper consolidating duplicate code
✅ All tests building without errors
✅ All linters passing with zero issues
✅ Comprehensive test coverage for new code
✅ Successfully merged with main branch (2x)
✅ All code formatted per project standards
✅ All Go linting passing

Phase 1 Summary

Completed in 6 commits:

1. Initial refactoring (file renames, pattern centralization)
2. Validation helper cleanup (removed unused implementations)
3. Test coverage (comprehensive test suite)
4. Merge main branch (first merge)
5. Merge main branch (second merge)
6. Format and lint (final cleanup)

Ready for Phase 2:
The foundation is now in place for the broader refactoring work outlined in the original issue. Phase 2 will implement the remaining validation helpers and update the 32 validation files to use them, eliminating the 70+ remaining duplicate validation patterns.

No breaking changes - all existing functionality preserved and verified with main branch merge.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[refactor] Semantic Function Clustering Analysis - Code Organization and Duplication</issue_title>
<issue_description>Comprehensive semantic analysis of the Go codebase in pkg/ identifying function clustering patterns, duplicate code, outlier functions, and refactoring opportunities to improve code organization and maintainability.

Executive Summary

Analysis of 464 non-test Go files across 18 packages revealed significant refactoring opportunities:

• 76+ duplicate or near-duplicate code patterns identified across validation files
• Naming inconsistencies in 21 MCP-related files (3 different naming conventions)
• 240-file workflow package lacks structural organization despite clear semantic clusters
• 32 validation files scattered without cohesive grouping
• High-impact consolidation opportunities in validation helpers, MCP configuration, and compiler integration

Key Finding: The codebase demonstrates good semantic organization at the function level but suffers from structural fragmentation that creates maintenance burden and cognitive overhead.

---

Analysis Scope

Files Analyzed: 464 Go source files (excluding tests)
Packages Analyzed: 18 top-level packages in pkg/
Focus Areas:

• Function clustering by semantic purpose
• Duplicate code detection
• File organization patterns
• Naming convention consistency
• Refactoring opportunities

Primary Analysis Areas:

1. Validation files (32 files, 27 matching *_validation.go pattern)
2. MCP configuration files (21 files with 3 different naming patterns)
3. Compiler integration files (21 files in workflow package)
4. CLI orchestration files (152 files including compile_* and codemod_* patterns)

---

1. Package Structure Analysis

Package Distribution by Size

Package Type	Count	Files	Purpose
Utility packages	11	1-2 each	Single-responsibility helpers (constants, envutil, gitutil, mathutil, etc.)
Medium packages	4	11-29	Focused modules (campaign, console, parser, stringutil)
Large packages	3	72-772	Complex business logic (cli: 152, parser: 29, workflow: 240)

Critical Complexity: workflow Package

Statistics:

• 240 source files + 532 test files (772 total)
• Flat namespace with no subpackages despite multiple distinct domains
• Identified clusters: Compiler (21 files), Safe Outputs (16 files), MCP (21 files), Engine (13 files), Validation (32 files)

Issue: The workflow package is a "monster package" where semantic clusters exist in naming but not in structure, making navigation difficult and boundaries unclear.

---

2. Validation Files: 76+ Duplicate Patterns

Overview

Files: 27 files matching *_validation.go pattern + 1 outlier (permissions_validator.go)
Total Functions: 63 validation functions
Duplicate Patterns: 76+ instances across 7 pattern categories

High-Impact Duplicates

Pattern A: Empty String/Nil Checking (40+ instances)

Severity: HIGH
Files Affected: 25+ validation files
Pattern:

if value == "" { return nil }
if config == nil { return nil }

Examples:

• agent_validation.go:61 - if workflowData.AgentFile == "" { return nil }
• features_validation.go:37 - if data == nil || data.Features == nil { return nil }
• sandbox_validation.go:84 - if workflowData.SandboxConfig == nil { return nil }
• runtime_validation.go:98 - if workflowData.Tools == nil { return nil }
• Similar patterns in 21+ additional files

Recommendation: Extract to shared helper

// pkg/workflow/validation_helpers.go
func requiresValidation(value string) bool {
    return value != ""
}

func requiresValidationPtr[T any](ptr *T) bool {
    return ptr != nil
}

Impact: Eliminate 40+ lines of repetitive code, establish consistent validation pattern

Pattern B: Map Type Assertion (30+ instances)

Severity: HIGH
Files Affected: 15+ validation files
Pattern:

if config, ok := toolConfig.(map[string]any); ok {
    // Process config
}

if eventMap, ok := eventVal.(map[string]any); !ok {
    return nil
}

Examples:

• mcp_config_validation.go:86, 127, 132, 200 (4 instances)
• runtime_validation.go:105
• dispatch_workflow_validation.go:89, 102, 117, 157, 167, 177 (6 instances)
• compiler_filters_validation.go:52, 80
• safe_outputs_domains_validation.go (multiple)
• agent_validation.go:127

Recommendation: Create assertion helpers

// pkg/workflow/validation_helpers.go
func assertMapOrNil(value any) (map[string]any, bool) {
    mapVal, ok := value.(map[string]any)
    return mapVal, ok
}

func requireMap(value any, fieldName string) (map[string]any, error) {
    mapVal, ok := value.(m...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes githubnext/gh-aw#11776

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

[pelikhan]

@copilot merge main, recompile

[Copilot]

@copilot merge main, recompile

Completed! Merged main branch and recompiled all 140 workflows successfully. Commit: c1d7ef0

[pelikhan]

@copilot merge main, format, lint go

[Copilot]

@copilot merge main, format, lint go

Completed! Merged main branch, formatted all code, and linted Go - all validations passing with zero issues. Commit: 2e19cb0