Prompt eval changes

.cursor/rules/code-generation.mdc

@@ -0,0 +1,102 @@
+---
+description: Code Generation Development Rules for CLI
+globs: internal/bundler/*.go, cmd/*.go
+alwaysApply: true
+---
+
+# Code Generation Development Rules
+
+> **⚠️ IMPORTANT**: These rules work in conjunction with the SDK rules. When updating these CLI rules, also update `sdk-js/.cursor/rules/code-generation.mdc` to keep them in sync.
+
+## Core Principles
+
+### Never Modify Generated Content in Source Files
+- ❌ NEVER edit files in `sdk-js/src/` that contain generated content
+- ❌ NEVER hardcode generated content like `copyWriter` prompts in source files
+- ✅ ALWAYS generate content into `node_modules/@agentuity/sdk/dist/` or `src/` directories
+- ✅ Use dynamic loading patterns for generated content
+
+### Code Generation Workflow
+1. **Modify CLI generation logic** (e.g., `internal/bundler/prompts.go`)
+2. **Update SDK to handle generated content dynamically** (e.g., `src/apis/prompt/index.ts`)
+3. **Build and test the full pipeline**: CLI generation → SDK loading → Agent usage
+
+### Optional Field Handling
+- ✅ Generated code should NEVER require optional chaining (`?.`)
+- ✅ Always generate both `system` and `prompt` fields, even if empty
+- ✅ Empty fields should return empty strings, not undefined
+- ❌ Never generate partial objects that require optional chaining
+
+## CLI-Specific Rules
+
+### Generation Target Locations
+```go
+// ✅ Correct: Generate into installed SDK
+sdkPath := filepath.Join(root, "node_modules", "@agentuity", "sdk", "dist", "generated")
+
+// ❌ Wrong: Generate into source SDK
+sdkPath := filepath.Join(root, "src", "generated")
+```
+
+### Path Resolution Priority
+1. Try `dist/` directory first (production)
+2. Fallback to `src/` directory (development)
+3. Always check if SDK exists before generating
+
+### File Generation Pattern
+```go
+func FindSDKGeneratedDir(ctx BundleContext, projectDir string) (string, error) {
+    possibleRoots := []string{
+        findWorkspaceInstallDir(ctx.Logger, projectDir),
+        projectDir,
+    }
+
+    for _, root := range possibleRoots {
+        // Try dist directory first (production)
+        sdkPath := filepath.Join(root, "node_modules", "@agentuity", "sdk", "dist", "generated")
+        if _, err := os.Stat(filepath.Join(root, "node_modules", "@agentuity", "sdk")); err == nil {
+            if err := os.MkdirAll(sdkPath, 0755); err == nil {
+                return sdkPath, nil
+            }
+        }
+        // Fallback to src directory (development)
+        sdkPath = filepath.Join(root, "node_modules", "@agentuity", "sdk", "src", "generated")
+        if _, err := os.Stat(filepath.Join(root, "node_modules", "@agentuity", "sdk", "src")); err == nil {
+            if err := os.MkdirAll(sdkPath, 0755); err == nil {
+                return sdkPath, nil
+            }
+        }
+    }
+    return "", fmt.Errorf("could not find @agentuity/sdk in node_modules")
+}
+```
+
+## Common Pitfalls to Avoid
+
+### ❌ Don't Do This
+```go
+// Hardcoding generated content in source files
+const prompts = `export const prompts = { copyWriter: { ... } };`
+
+// Generating to source SDK files
+sdkPath := filepath.Join(root, "src", "generated")
+
+// Not checking if SDK exists
+os.WriteFile(path, content, 0644) // Without checking if path exists
+```
+
+### ✅ Do This Instead
+```go
+// Generate dynamic content from YAML/data
+content := GenerateTypeScriptTypes(prompts)
+
+// Generate to installed SDK
+sdkPath := filepath.Join(root, "node_modules", "@agentuity", "sdk", "dist", "generated")
+
+// Check and create directories
+if err := os.MkdirAll(sdkPath, 0755); err != nil {
+    return fmt.Errorf("failed to create directory: %w", err)
+}
+```
+
+Remember: The CLI's job is to generate content into the installed SDK, not modify source files.

.cursor/rules/prompt-docstrings.mdc

@@ -0,0 +1,95 @@
+# Prompt JSDoc Docstrings
+
+## Goal
+Add JSDoc-style docstrings to generated prompt objects that provide better IDE support and make the generated code self-documenting.
+
+## Requirements
+
+### 1. JSDoc Format for PromptsCollection
+Generate JSDoc comments on the `PromptsCollection` type properties with the following structure:
+```typescript
+export type PromptsCollection = {
+  /**
+   * [Prompt Name] - [Prompt Description]
+   *
+   * @prompt
+   * [Original prompt template with variables]
+   */
+  promptName: PromptName;
+};
+```
+
+### 2. Content Inclusion
+- **Name and Description**: Include both from YAML in format "Name - Description"
+- **@prompt**: Include only the original prompt template (not system template)
+- **Template Preservation**: Show original templates exactly as written in YAML
+
+### 3. Template Preservation
+- Show original templates exactly as written in YAML
+- Preserve variable syntax: `{variable:default}`, `{!variable}`, `{{variable}}`
+- Maintain line breaks and formatting
+- Escape JSDoc comment characters (`*/` → `* /`)
+
+### 4. IDE Integration
+- Docstrings should be visible in IDE hover tooltips when accessing `prompts.promptName`
+- Should work with "Go to Definition" functionality
+- Provide IntelliSense documentation
+- Show original prompt template for reference
+
+## Implementation
+
+### Code Generator Updates
+1. **PromptsCollection JSDoc**: Add JSDoc comments to each property in the `PromptsCollection` type
+2. **Template Escaping**: Handle JSDoc comment characters in templates
+3. **Line Break Handling**: Split templates by newlines and add proper JSDoc formatting
+4. **Prompt-Only Focus**: Only include `@prompt` section, not `@system`
+
+### File Structure
+- **`_index.js`**: Contains actual prompt objects (no JSDoc on individual objects)
+- **`index.d.ts`**: Contains TypeScript types with JSDoc comments on `PromptsCollection` properties
+
+## Example Output
+
+```typescript
+export type PromptsCollection = {
+  /**
+   * Optional Variables with Defaults - Test optional variables that have default values
+   *
+   * @prompt
+   * Help the user with: {task:their question}
+   * Use a {tone:friendly} approach.
+   */
+  optionalWithDefaults: OptionalWithDefaults;
+  /**
+   * Required Variables Test - Test required variables that must be provided
+   *
+   * @prompt
+   * Complete this {!task} for the user.
+   * The task must be specified.
+   */
+  requiredVariables: RequiredVariables;
+};
+```
+
+## Key Decisions
+
+### Why PromptsCollection Only
+- Individual prompt objects are not directly accessible to users
+- IDE hover works on `prompts.promptName` which maps to `PromptsCollection` properties
+- Avoids redundant JSDoc comments that don't provide value
+
+### Why Prompt Template Only
+- Users primarily care about what the prompt does, not the system instructions
+- Keeps JSDoc comments focused and concise
+- System templates are implementation details
+
+### Why No Individual Type JSDoc
+- Individual prompt types are not directly used by developers
+- JSDoc on `PromptsCollection` properties provides the IDE support needed
+- Keeps generated code clean and focused
+
+## Benefits
+- **IDE Support**: Better IntelliSense and hover information on `prompts.promptName`
+- **Focused Documentation**: Shows only the prompt template that users will see
+- **Clean Code**: No redundant JSDoc comments on unused objects
+- **Self-Documenting**: Developers can understand prompts without looking at YAML

.cursor/rules/testing.mdc

@@ -0,0 +1,30 @@
+# Testing Rules
+
+## Use `go run .` for Testing
+- ✅ ALWAYS use `go run .` instead of building the binary for testing
+- ✅ This is faster and avoids unnecessary binary generation
+- ❌ Don't use `go build -o agentuity` unless you need a persistent binary
+
+## Testing Workflow
+```bash
+# ✅ Good: Test with go run
+go run . bundle
+go run . dev
+go run . project create
+
+# ❌ Avoid: Building every time for testing
+go build -o agentuity && ./agentuity bundle
+```
+
+## When to Build
+- Only build when you need a persistent binary for:
+  - Installation scripts
+  - Distribution
+  - CI/CD pipelines
+  - Manual testing outside the project directory
+
+## Development Benefits
+- Faster iteration cycle
+- No need to manage binary files
+- Cleaner development workflow
+- Less disk space usage

.cursor/rules/unit-testing.mdc

@@ -0,0 +1,106 @@
+---
+description: Unit Testing Rules for CLI
+globs: *_test.go
+alwaysApply: true
+---
+
+# Unit Testing Rules
+
+## Use Testify for All Tests
+- ✅ ALWAYS use `github.com/stretchr/testify/assert` and `github.com/stretchr/testify/require` for testing
+- ✅ Use `assert.Equal()`, `assert.True()`, `assert.False()`, `assert.NoError()`, `assert.Error()` for assertions
+- ✅ Use `require.NoError()`, `require.True()`, `require.False()` for fatal assertions that should stop test execution
+- ✅ Use table-driven tests with `t.Run()` for multiple test cases
+
+## Test Structure Pattern
+```go
+func TestFunctionName(t *testing.T) {
+    tests := []struct {
+        name     string
+        input    string
+        expected string
+    }{
+        {"test case 1", "input1", "expected1"},
+        {"test case 2", "input2", "expected2"},
+    }
+
+    for _, test := range tests {
+        t.Run(test.name, func(t *testing.T) {
+            result := FunctionName(test.input)
+            assert.Equal(t, test.expected, result)
+        })
+    }
+}
+```
+
+## Reference Examples
+- See `internal/util/strings_test.go` for basic assertion patterns
+- See `internal/util/api_test.go` for complex test scenarios with `require` and `assert`
+- See `internal/bundler/bundler_test.go` for table-driven test patterns
+
+## Common Testify Patterns
+
+### Basic Assertions
+```go
+import (
+    "testing"
+    "github.com/stretchr/testify/assert"
+    "github.com/stretchr/testify/require"
+)
+
+func TestBasicAssertions(t *testing.T) {
+    // Equality
+    assert.Equal(t, expected, actual)
+    assert.NotEqual(t, expected, actual)
+
+    // Boolean checks
+    assert.True(t, condition)
+    assert.False(t, condition)
+
+    // Error handling
+    assert.NoError(t, err)
+    assert.Error(t, err)
+
+    // Fatal assertions (stop test on failure)
+    require.NoError(t, err)
+    require.True(t, condition)
+}
+```
+
+### Table-Driven Tests
+```go
+func TestMultipleCases(t *testing.T) {
+    tests := []struct {
+        name     string
+        input    string
+        expected string
+    }{
+        {"empty string", "", ""},
+        {"simple case", "input", "output"},
+        {"special chars", "input@#$", "output___"},
+    }
+
+    for _, test := range tests {
+        t.Run(test.name, func(t *testing.T) {
+            result := ProcessInput(test.input)
+            assert.Equal(t, test.expected, result)
+        })
+    }
+}
+```
+
+### Error Testing
+```go
+func TestErrorHandling(t *testing.T) {
+    t.Run("should return error for invalid input", func(t *testing.T) {
+        err := ProcessInvalidInput("invalid")
+        require.Error(t, err)
+        assert.Contains(t, err.Error(), "invalid input")
+    })
+
+    t.Run("should not return error for valid input", func(t *testing.T) {
+        err := ProcessValidInput("valid")
+        assert.NoError(t, err)
+    })
+}
+```

cmd/bundle.go

@@ -40,9 +40,7 @@ Examples:
 		projectContext := project.EnsureProject(ctx, cmd)
 
 		// Check for prompts evals feature flag
-		if CheckFeatureFlag(cmd, FeaturePromptsEvals, "enable-prompts-evals") {
-			projectContext.Logger.Info("Prompts evaluations feature is enabled")
-		}
+		promptsEvalsFF := CheckFeatureFlag(cmd, FeaturePromptsEvals, "enable-prompts-evals")
 
 		production, _ := cmd.Flags().GetBool("production")
 		install, _ := cmd.Flags().GetBool("install")
@@ -52,14 +50,15 @@ Examples:
 		description, _ := cmd.Flags().GetString("description")
 
 		if err := bundler.Bundle(bundler.BundleContext{
-			Context:    ctx,
-			Logger:     projectContext.Logger,
-			Project:    projectContext.Project,
-			ProjectDir: projectContext.Dir,
-			Production: production,
-			Install:    install,
-			CI:         ci,
-			Writer:     os.Stderr,
+			Context:        ctx,
+			Logger:         projectContext.Logger,
+			Project:        projectContext.Project,
+			ProjectDir:     projectContext.Dir,
+			Production:     production,
+			PromptsEvalsFF: promptsEvalsFF,
+			Install:        install,
+			CI:             ci,
+			Writer:         os.Stderr,
 		}); err != nil {
 			errsystem.New(errsystem.ErrInvalidConfiguration, err, errsystem.WithContextMessage("Failed to bundle project")).ShowErrorAndExit()
 		}