Refactor generators to use NodeSpec's own files as templates (dogfooding approach)

[deepracticexs]

Problem

Currently, our generators (PackageGenerator, AppGenerator, ServiceGenerator) use hardcoded strings to generate configuration files like package.json, tsconfig.json, tsup.config.ts, etc. This creates several issues:

1. Duplication: We maintain the same configuration in multiple places:

   • NodeSpec's own configuration files
   • Hardcoded strings in generator classes

2. Drift: When we update NodeSpec's configuration (e.g., add a new script, change tsup settings), we must remember to update the generators too. This is error-prone.

3. Inconsistency: Generated projects might use different configurations than NodeSpec itself.

Proposed Solution

"Eat our own dog food" - Use NodeSpec's actual configuration files as templates instead of hardcoding them.

Architecture

// Base class for all generators
class BaseGenerator {
  /**
   * Get the root directory of NodeSpec project
   * This is the source of truth for all templates
   */
  protected getNodeSpecRoot(): string {
    // When running from published CLI: point to embedded templates
    // When running from source: point to NodeSpec repo root
  }
  
  /**
   * Read a template file from NodeSpec's own files
   */
  protected async readTemplate(relativePath: string): Promise<string> {
    const nodespecRoot = this.getNodeSpecRoot();
    return fs.readFile(path.join(nodespecRoot, relativePath), 'utf-8');
  }
  
  /**
   * Read and parse a JSON template
   */
  protected async readJsonTemplate(relativePath: string): Promise<any> {
    const content = await this.readTemplate(relativePath);
    return JSON.parse(content);
  }
}

Template Source Mapping

Generator	Template Source	Files to Use
PackageGenerator	packages/logger/ or packages/template/	package.json, tsconfig.json, tsup.config.ts
AppGenerator	apps/cli/	package.json, tsconfig.json, tsup.config.ts, src/ structure
ServiceGenerator	services/example/ (to be created)	package.json, tsconfig.json, tsup.config.ts, src/ structure
ProjectGenerator	Root directory	package.json, tsconfig.json, turbo.json, .gitignore, etc.

Implementation Plan

Phase 1: Create BaseGenerator

• Create src/core/template/BaseGenerator.ts with template reading utilities
• Implement getNodeSpecRoot() logic:
  • For development: find NodeSpec repo root
  • For published package: use embedded template directory
• Add helper methods: readTemplate(), readJsonTemplate(), copyTemplate()

Phase 2: Refactor PackageGenerator

• Extend BaseGenerator
• Use packages/template/package.json as template source
• Only modify necessary fields (name, version, description)
• Preserve all other fields from template
• Do the same for tsconfig.json, tsup.config.ts

Phase 3: Refactor AppGenerator

• Extend BaseGenerator
• Use apps/cli/ as template source
• Copy structure and configuration
• Only modify app-specific fields

Phase 4: Refactor ServiceGenerator

• Create a reference service in services/example/
• Extend BaseGenerator
• Use the reference service as template

Phase 5: Refactor ProjectGenerator

• Use NodeSpec's root files as templates
• Simplify the massive hardcoded configuration strings

Phase 6: Template Embedding for Distribution

When publishing the CLI package, we need to embed the template files:

• Copy template source files to dist/templates/ during build
• Update getNodeSpecRoot() to detect runtime environment
• Update package.json files field to include templates

Benefits

1. ✅ Single Source of Truth: NodeSpec's configuration is the only source
2. ✅ Automatic Synchronization: Update NodeSpec config → generated projects automatically use new config
3. ✅ No Duplication: Eliminate hardcoded configuration strings
4. ✅ Consistency: Generated projects are guaranteed to match NodeSpec's structure
5. ✅ Easier Maintenance: Only update in one place
6. ✅ Better Testing: We test the exact same configuration we use

Example: Before & After

Before (PackageGenerator.ts):

private async generatePackageJson(targetDir: string, packageName: string) {
  const packageJson = {
    name: packageName,
    version: "0.0.1",
    description: `${packageName} package`,
    type: "module",
    main: "./dist/index.js",
    // ... 50+ lines of hardcoded config
  };
  await fs.writeJson(path.join(targetDir, "package.json"), packageJson);
}

After (PackageGenerator.ts):

private async generatePackageJson(targetDir: string, packageName: string) {
  // Read NodeSpec's own template
  const template = await this.readJsonTemplate('packages/template/package.json');
  
  // Only modify necessary fields
  template.name = packageName;
  template.version = "0.0.1";
  template.description = `${packageName} package`;
  
  await fs.writeJson(path.join(targetDir, "package.json"), template);
}

Migration Strategy

1. Create packages/template/ as the canonical template package in NodeSpec
2. Implement BaseGenerator with template reading
3. Migrate generators one by one
4. Update tests to verify templates are used correctly
5. Deprecate old hardcoded approach

Open Questions

1. Template Location: Should we create dedicated template directories (e.g., packages/template/) or use existing packages (e.g., packages/logger/)?

2. Template Customization: How do we handle fields that need computation (e.g., bin names based on package name)?

3. Template Versioning: Should templates be versioned separately from the CLI?

4. Template Distribution: Best way to bundle templates with the published CLI package?

Related Issues

• Improves consistency with #[related-issue-if-any]
• Addresses maintenance burden mentioned in #[related-issue-if-any]

Priority

Medium-High - This is a code quality and maintainability improvement that will pay dividends over time, especially as we add more features and templates.

[deepracticexs]

Architecture Design for Generator Refactoring

After discussion, we've agreed on a file-level composition approach using the Responsibility Chain pattern. This design avoids over-engineering while maintaining flexibility.

Core Philosophy

File-level building blocks, bottom-up composition

• Generators explicitly declare which files they need
• Each file is matched to a corresponding Processor for handling
• Processors are responsible for "how to process this file", not prioritization
• No directory scanning, no exclude lists - just explicit file declarations (whitelist approach)

Architecture Design

interface FileProcessor {
  // Determine if this processor can handle the file (based on filename/extension)
  canProcess(fileName: string): boolean;
  
  // Process file: read source, apply transformation, write to target
  process(sourcePath: string, targetPath: string, context: ProcessContext): Promise<void>;
}

interface ProcessContext {
  packageName: string;
  dirName: string;
  [key: string]: any; // extensible
}

class BaseGenerator {
  protected processors: FileProcessor[] = [];
  
  constructor() {
    // Register all processors (order doesn't matter - atomic matching)
    this.processors = [
      new PackageJsonProcessor(),
      new TsConfigProcessor(),
      new TsupConfigProcessor(),
      new TypeScriptProcessor(),
      new MarkdownProcessor(),
    ];
  }
  
  protected async processFile(
    fileName: string,
    sourcePath: string, 
    targetPath: string, 
    context: ProcessContext
  ): Promise<void> {
    const processor = this.processors.find(p => p.canProcess(fileName));
    
    if (!processor) {
      throw new Error(`No processor found for file: ${fileName}`);
    }
    
    await processor.process(sourcePath, targetPath, context);
  }
  
  protected getNodeSpecRoot(): string {
    // When running from source: NodeSpec repo root
    // When running from published CLI: embedded template directory
    // TODO: implement runtime detection
  }
}

Example: PackageGenerator

class PackageGenerator extends BaseGenerator {
  async generate(monorepoRoot: string, options: PackageOptions): Promise<void> {
    const context = {
      packageName: options.name,
      dirName: this.extractDirectoryName(options.name),
    };
    
    const templateRoot = this.getNodeSpecRoot();
    const targetDir = path.join(monorepoRoot, 'packages', context.dirName);
    
    // Explicitly declare needed files
    const files = [
      { source: 'packages/template/package.json', target: 'package.json' },
      { source: 'packages/template/tsconfig.json', target: 'tsconfig.json' },
      { source: 'packages/template/tsup.config.ts', target: 'tsup.config.ts' },
      { source: 'packages/template/src/index.ts', target: 'src/index.ts' },
      { source: 'packages/template/README.md', target: 'README.md' },
    ];
    
    for (const file of files) {
      const sourcePath = path.join(templateRoot, file.source);
      const targetPath = path.join(targetDir, file.target);
      
      await fs.ensureDir(path.dirname(targetPath));
      await this.processFile(file.target, sourcePath, targetPath, context);
    }
  }
}

File Processors

Each processor handles one file type:

class PackageJsonProcessor implements FileProcessor {
  canProcess(fileName: string): boolean {
    return fileName.endsWith('package.json');
  }
  
  async process(sourcePath: string, targetPath: string, context: ProcessContext): Promise<void> {
    const template = await fs.readJson(sourcePath);
    
    // Only modify necessary fields, preserve everything else
    const result = {
      ...template,
      name: context.packageName,
      version: '0.0.1',
      description: `${context.packageName} package`,
    };
    
    await fs.writeJson(targetPath, result, { spaces: 2 });
  }
}

class TsConfigProcessor implements FileProcessor {
  canProcess(fileName: string): boolean {
    return fileName.endsWith('tsconfig.json');
  }
  
  async process(sourcePath: string, targetPath: string, _context: ProcessContext): Promise<void> {
    // tsconfig mostly doesn't need modification, just copy
    const content = await fs.readJson(sourcePath);
    await fs.writeJson(targetPath, content, { spaces: 2 });
  }
}

class TypeScriptProcessor implements FileProcessor {
  canProcess(fileName: string): boolean {
    return fileName.endsWith('.ts');
  }
  
  async process(sourcePath: string, targetPath: string, _context: ProcessContext): Promise<void> {
    // TypeScript files: direct copy
    // Add variable replacement here if needed in the future
    const content = await fs.readFile(sourcePath, 'utf-8');
    await fs.writeFile(targetPath, content);
  }
}

Advantages

1. Atomic: Each file is processed independently, clear responsibilities
2. Explicit Declaration: Generators explicitly list which files they need
3. No Priority Issues: canProcess() is based on filename, no conflicts
4. No Exclude Lists Needed: Whitelist approach, only process declared files
5. Avoids Over-engineering: No directory scanning, no priority, no complex filtering

Template Source Mapping

Generator	Template Source	Files to Use
PackageGenerator	packages/template/	package.json, tsconfig.json, tsup.config.ts, src/index.ts
AppGenerator	apps/cli/	package.json, tsconfig.json, tsup.config.ts, src/ structure
ServiceGenerator	services/example/ (to be created)	package.json, tsconfig.json, tsup.config.ts, src/ structure
ProjectGenerator	Root directory	package.json, tsconfig.json, turbo.json, etc.

Implementation Plan

1. Create BaseGenerator class with file processor chain
2. Implement file processors: PackageJsonProcessor, TsConfigProcessor, TsupConfigProcessor, TypeScriptProcessor, MarkdownProcessor
3. Refactor PackageGenerator to extend BaseGenerator
4. Refactor AppGenerator to extend BaseGenerator
5. Test the refactored generators
6. Handle template embedding for CLI distribution

Open Questions

1. Runtime Environment Detection: How to implement getNodeSpecRoot() to detect dev vs published environment?
2. Template Variable Syntax: Do we need variable replacement in TypeScript files? If yes, what syntax?

---

Ready to start implementation! 🚀