Skip to content

Improve Planner estimation using GitHub history and complexity #34

@prosdev

Description

@prosdev

Description

The Planner currently uses simple keyword-based heuristics for effort estimation (e.g., 'implement' = 6 hours). We can make this much smarter by leveraging:

  1. GitHub history from our GitHub Context subagent
  2. Task complexity analysis
  3. AI-assisted development factors

Current Behavior

// Simple keyword matching
if (lowerDesc.includes('implement')) {
  return 6;  // Always 6 hours
}

Example: Issue #28 estimated at 4 days, but with AI assistance took ~4-6 hours.

Proposed Solution: Use GitHub History! 🎯

We already have the GitHub Context subagent (#10) that indexes issues, PRs, and their metadata. Use this data!

Historical Data Available

// From GitHub Context subagent
interface GitHubDocument {
  number: number;
  title: string;
  state: 'open' | 'closed';
  created_at: string;
  closed_at?: string;     // ← Calculate duration!
  labels: string[];        // ← Complexity indicators
  comments: number;        // ← Complexity/difficulty indicator
  body: string;
  linkedPRs?: number[];    // ← See actual code changes
}

1. Learn from Similar Issues

// Use GitHub agent to find similar closed issues
async function estimateFromHistory(task: PlanTask): Promise<number> {
  // Search for similar closed issues
  const similar = await githubAgent.search(task.description, {
    type: 'issue',
    state: 'closed',
    limit: 5
  });
  
  // Calculate their durations
  const durations = similar.map(issue => {
    const created = new Date(issue.created_at);
    const closed = new Date(issue.closed_at!);
    return (closed - created) / (1000 * 60 * 60); // hours
  });
  
  // Use median duration as estimate
  return median(durations);
}

2. Complexity Indicators from History

interface ComplexityFactors {
  commentCount: number;      // More comments = more complex
  linkedPRCount: number;     // Multiple PRs = larger scope
  labelComplexity: string[]; // 'bug' < 'feature' < 'epic'
  descriptionLength: number; // Detailed = complex
  acceptanceCriteria: number; // More criteria = more work
}

function calculateComplexity(issue: GitHubDocument): number {
  let score = 1.0;
  
  if (issue.comments > 10) score *= 1.5;  // Lots of discussion
  if (issue.linkedPRs.length > 2) score *= 1.3;  // Multiple PRs
  if (issue.labels.includes('epic')) score *= 2.0;
  if (issue.labels.includes('enhancement')) score *= 1.2;
  
  return score;
}

3. AI Assistance Detection

// Detect if repo uses dev-agent
function detectAIAssistance(): boolean {
  return fs.existsSync('.dev-agent.json') || 
         fs.existsSync('.cursor/') ||
         // Check commit messages for AI signatures
         recentCommits.some(c => c.message.includes('feat('));
}

const AI_MULTIPLIER = 0.3;  // AI is 3x faster

4. Full Estimation Flow

async function smartEstimate(task: PlanTask): Promise<number> {
  // 1. Find similar historical issues
  const historical = await estimateFromHistory(task);
  
  // 2. Analyze current task complexity
  const complexity = analyzeComplexity(task);
  
  // 3. Apply AI multiplier if detected
  const isAI = detectAIAssistance();
  const multiplier = isAI ? AI_MULTIPLIER : 1.0;
  
  // 4. Combine signals
  let estimate = historical * complexity * multiplier;
  
  // 5. Fallback to heuristics if no history
  if (!historical) {
    estimate = keywordEstimate(task) * complexity * multiplier;
  }
  
  return Math.ceil(estimate);
}

Integration with Existing System

Planner already coordinates with GitHub agent:

// In PlannerAgent.createPlan()
const githubContext = await this.coordinator.sendMessage({
  recipient: 'github',
  payload: {
    action: 'search',
    query: task.description,
    searchOptions: { type: 'issue', state: 'closed' }
  }
});

Acceptance Criteria

  • Use GitHub agent to find similar closed issues
  • Calculate duration from created_at to closed_at
  • Analyze complexity from comments, PRs, labels
  • Detect AI assistance (dev-agent config, commit patterns)
  • Combine historical + complexity + AI factors
  • Fallback to keyword heuristics if no history
  • Tests with mock GitHub data
  • Estimates within 50% of actual time

Benefits

  1. Learn from past work - Similar issues took X hours historically
  2. Complexity-aware - More comments/PRs = harder task
  3. AI-realistic - 3x faster estimates when AI-assisted
  4. Self-improving - Gets better as more issues are closed
  5. Already integrated - Uses existing GitHub Context subagent!

Example

# Planner checks GitHub history
dev plan 34

# Finds: Issue #28 (similar, took 6 hours, had 3 comments)
# Calculates: 6h * 1.2 (more complex) * 0.3 (AI) = 2.2h
# → Estimates: 3 hours

Priority: Medium (improves dogfooding accuracy)
Estimate: 1-2 days
Dependencies: GitHub Context subagent (#10) ✅ (already done!)
Related: Issue #28 (discovered during dogfooding)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions