Add a skill for evolving skills or adding new skills based on interaction#952
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
rgsl888prabhu
added
non-breaking
…public into add_skill_evolution
📝 WalkthroughWalkthroughAdds a Skill Evolution framework: registers a new plugin and cursor rule, introduces comprehensive skill-evolution docs and workflows, updates agent documentation and CI validation docs, and inserts post-correction hooks and modeling guidance into existing skills. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Comment Tip CodeRabbit can use Trivy to scan for security misconfigurations and secrets in Infrastructure as Code files.Add a .trivyignore file to your project to customize which findings Trivy reports. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@skills/skill-evolution/SKILL.md`:
- Line 84: Add language identifiers to the two fenced code blocks in SKILL.md
that currently lack them: locate the fence under the "Skill update proposal:"
block and the fence under the "Skill insight (unscored):" block and change the
opening triple backticks from ``` to ```text so both code blocks are labeled
(for example, use the "text" tag) to satisfy markdownlint MD040.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ddc77017-3ae3-4df1-827c-9bbec706998f
📒 Files selected for processing (10)
.claude-plugin/marketplace.json.claude/CLAUDE.md.cursor/rules/skill-evolution.mdcAGENTS.mdagents/AGENTS.mdagents/AGENTS.mdci/README.mdskills/cuopt-user-rules/SKILL.mdskills/lp-milp-formulation/SKILL.mdskills/skill-evolution/SKILL.md
|
|
||
| The user may approve, decline, or defer for offline reflection. | ||
|
|
||
| ## Phase 3: Offline reflection |
There was a problem hiding this comment.
Few questions:
- For training the agent should have a clear context for each new prompt.
- The agent has seen the ground truths and would be jumping to inference without clearing context. I think we need state it explicitly and check that it works otherwise it may not be possible to do in the same active session.
- I think we need an evaluate.py that sets up the train pipeline. I don't think it can go over a dataset and understand what is train and test with the skill only. You can take a look at: https://github.com/karpathy/autoresearch/blob/master/prepare.py
- Should we provide a parser.py? The data is either in json or csv for outbound skill refinement: https://github.com/NVIDIA/cuopt-examples/tree/main/cuopt-agent/cuopt_agent/data/max_supply_what_ifs/eval
There was a problem hiding this comment.
I am currently working on eval part where I will try to evolve skill based on industry or dataset and then run again with updated ones. And also run this against the optmath and other dataset to see if it provides better performance.
There was a problem hiding this comment.
@hlinsen I agree, evaluation and evolution script needs to be separate. There is a on going discussion on how to tackle this, may be we can work on this as follow-up.
|
@rgsl888prabhu Is there a way to ensure that the new edits being added is not extensive and also not specific. I remember when I was trying to do the same with industryor it updated skills to be very specific wrt the currency and terminology used to perform better to that specific benchmark |
I have added specific rules so that it will not add very specific details, but a human needs to be in the loop to make sure these edits are not of that nature. That's why in normal circumstances, it proposes edits to the user and user needs to verify before adding those. Currently we are working on evaluation methods that should help us mitigate these issue more robustly. |
There was a problem hiding this comment.
♻️ Duplicate comments (1)
skills/lp-milp-formulation/SKILL.md (1)
176-192:⚠️ Potential issue | 🔴 CriticalMissing
<!-- skill-evolution:end -->marker.The goal programming section has a start marker at line 177 but no corresponding end marker before the separator at line 193. All other skill-evolution sections in this file have matching start/end markers. This inconsistency will likely break any automated processing that relies on these markers.
🔧 Proposed fix
**Default:** If the problem does not specify timing within a period, use **only** end-of-period capacity (`stock[t] <= capacity`). Add the after-purchase constraint only if the problem explicitly requires it. +<!-- skill-evolution:end --> ---Note: This issue was previously flagged in past reviews and remains unresolved.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@skills/lp-milp-formulation/SKILL.md` around lines 176 - 192, The goal programming section has a start marker "<!-- skill-evolution:start — goal programming section -->" but is missing the matching end marker; add the corresponding "<!-- skill-evolution:end -->" immediately after the goal programming block (before the following separator/next section) so the start/end marker pair is balanced and automated processing can detect the section correctly.
🧹 Nitpick comments (1)
skills/lp-milp-formulation/SKILL.md (1)
141-148: Consider adding language specifiers to fenced code blocks.The fenced code blocks at lines 141, 160, and 166 lack language specifiers. While the content is clear, adding specifiers (e.g.,
textorpseudocode) would satisfy markdown linting rules and improve consistency.✨ Example fix
For line 141:
-``` +```text x_total — INTEGER (total production of a product)Similarly for lines 160 and 166.
Also applies to: 160-170
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@skills/lp-milp-formulation/SKILL.md` around lines 141 - 148, Update the three fenced code blocks that document segment variables so they include a markdown language specifier (e.g., use ```text or ```pseudocode) to satisfy linting; specifically edit the block containing "x_total — INTEGER" and the blocks referencing "s1, s2, …" (the blocks around the symbols x_total, s1, s2 and the other two fenced examples noted) to prepend a language token after the opening backticks so the blocks render consistently and pass markdown lints.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@skills/lp-milp-formulation/SKILL.md`:
- Around line 176-192: The goal programming section has a start marker "<!--
skill-evolution:start — goal programming section -->" but is missing the
matching end marker; add the corresponding "<!-- skill-evolution:end -->"
immediately after the goal programming block (before the following
separator/next section) so the start/end marker pair is balanced and automated
processing can detect the section correctly.
---
Nitpick comments:
In `@skills/lp-milp-formulation/SKILL.md`:
- Around line 141-148: Update the three fenced code blocks that document segment
variables so they include a markdown language specifier (e.g., use ```text or
```pseudocode) to satisfy linting; specifically edit the block containing
"x_total — INTEGER" and the blocks referencing "s1, s2, …" (the blocks around
the symbols x_total, s1, s2 and the other two fenced examples noted) to prepend
a language token after the opening backticks so the blocks render consistently
and pass markdown lints.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 7d05ebfc-31cd-4fc4-a9ff-2d0c083e8154
📒 Files selected for processing (1)
skills/lp-milp-formulation/SKILL.md
hlinsen
left a comment
hlinsen
left a comment
There was a problem hiding this comment.
Thanks @rgsl888prabhu!
As discussed in comments, the skill could be further tuned but we already see performance benefits with this version.
|
Created an issue to follow-up on the discussions #965 |
|
/merge |
5 participants
Description
The new skill will try to propose a change or new skills so that it can capture generic patterns to help next developer
Checklist