From d214486f2edc3cbf46107c19617cf7c5c7bb7f87 Mon Sep 17 00:00:00 2001 From: cmjaolan <69292391+cmjaolan@users.noreply.github.com> Date: Wed, 28 Jan 2026 09:07:31 +0800 Subject: [PATCH] Create copilot-instructions.md --- .github/copilot-instructions.md | 117 ++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 .github/copilot-instructions.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..b123141 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,117 @@ +# TensorStack Copilot Instructions + +TensorStack is a modular .NET framework for high-performance ONNX-based AI model inference across multiple hardware providers and media types (image, video, audio). + +## Architecture Overview + +**Layered Design**: The codebase follows a strict layering pattern: +- **TensorStack.Common**: Core abstractions - `IPipeline`, tensor types, model session management, device/execution provider registry +- **Media Layer** (`TensorStack.Image*`, `TensorStack.Audio*`, `TensorStack.Video*`): Platform-specific input handling (WinForms Bitmap, BitmapImage, ImageSharp, Windows audio/video APIs) +- **Provider Layer** (`TensorStack.Providers.*`): Execution providers wrap ONNX Runtime's `SessionOptions` - CPU, CUDA, DML each expose a `Provider.GetProvider()` static method +- **Feature Pipelines** (`TensorStack.Extractors`, `TensorStack.Upscaler`, `TensorStack.TextGeneration`, `TensorStack.StableDiffusion`): Concrete `IPipeline` implementations + +**Key Integration Points**: +- Pipelines consume `ModelConfig` (path + ExecutionProvider) and produce typed outputs via `RunAsync()` +- `DeviceManager` is a static registry initialized with an ExecutionProvider and optional native library path +- `ModelSession` manages ONNX Runtime's `InferenceSession` lifecycle and `SessionOptions` configuration + +## Essential Patterns + +### Pipeline Implementation +Every feature pipeline implements `IPipeline`: +```csharp +public class MyPipeline : IPipeline +{ + public async Task LoadAsync(CancellationToken ct) => await _model.LoadAsync(ct); + public async Task RunAsync(MyOptions opts, IProgress progress = null, CancellationToken ct = default) + { + var result = await _model.InferAsync(opts.Input); + progress?.Report(new RunProgress(RunProgress.GetTimestamp())); + return result; + } +} +``` +- Always implement `IDisposable` to release ONNX sessions +- Use `IPipelineStream` for streaming video frame processing with `IAsyncEnumerable` +- Progress callbacks report elapsed time via `RunProgress.GetTimestamp()` + +### Model Configuration Pattern +Inherit from `ModelConfig` to define model-specific settings: +```csharp +public record ExtractorConfig : ModelConfig +{ + public Normalization Normalization { get; set; } + public Normalization OutputNormalization { get; set; } + public bool IsDynamicOutput { get; set; } +} +``` +- Always use `record` for immutability +- Set execution provider via `config.SetProvider(executionProvider)` or in initialization +- Path must point to a valid ONNX file + +### Tensor Type Conversions +`ImageTensor` and `VideoTensor` inherit from `Tensor` with automatic shape handling: +- `ImageTensor`: Always dims `[1, channels, height, width]`; auto-validates 1, 3, or 4 channel images +- `VideoTensor`: Dims `[batch, frames, channels, height, width]` +- Use `tensor.AsSpan()` to access underlying data; dimensions available via `tensor.Dimensions` + +## Build & Release Workflow + +**Individual Package Build**: +```powershell +dotnet build TensorStack.Extractors/TensorStack.Extractors.csproj -c Release +dotnet pack TensorStack.Extractors/TensorStack.Extractors.csproj -c Release +``` + +**Batch Release** (automated via `BuildRelease.bat`): +- Builds all projects in dependency order: Common → Media (Bitmap, Audio, Video) → Providers → Features +- Packs each as a NuGet package to `./Nuget/` directory +- Version centralized in `Directory.Build.props` (currently 0.4.0) + +## Platform-Specific Conventions + +**Windows Media APIs**: `TensorStack.Image.Bitmap`, `TensorStack.Audio.Windows`, `TensorStack.Video.Windows` wrap WinForms/Windows Media Foundation: +- Abstract base classes (`ImageInputBase`, `AudioInputBase`) in platform-agnostic projects +- Concrete implementations (e.g., `ImageInput`) in Windows-specific projects +- Use `.targets` files (see `TensorStack.Audio.Windows.targets`) to define native dependencies + +**Provider Selection**: Each provider has a static `Provider` class: +```csharp +ExecutionProvider provider = TensorStack.Providers.CPU.Provider.GetProvider(); +DeviceManager.Initialize(provider, "TensorStack"); +``` + +## Critical Developer Tasks + +**Adding a New Pipeline**: +1. Create `TensorStack.MyFeature/` folder with `.csproj` +2. Define `MyConfig : ModelConfig` and `MyOptions : IRunOptions` +3. Implement concrete pipeline class inheriting `IPipeline` +4. Add model loading via `ModelSession` in the pipeline +5. Reference `TensorStack.Common` and required media packages +6. Add build+pack steps to `BuildRelease.bat` + +**Supporting a New Execution Provider**: +1. Create `TensorStack.Providers.NewProvider/` with `Provider.cs` +2. Implement static `GetProvider()` returning `ExecutionProvider` with proper `SessionOptions` +3. Register native library path in `DeviceManager.Initialize()` +4. Each provider's `SessionOptions` factory is stored in the `ExecutionProvider` constructor + +**Cross-Cutting Concerns**: Use extensions in each module's `Common/` folder (e.g., `TensorStack.Extractors/Common/`) for normalization, model-specific tensor conversions, and pipeline-specific utilities. + +## NuGet & Dependencies + +- **Core Dependency**: `Microsoft.ML.OnnxRuntime` (abstracts ONNX inference) +- **Media**: Individual packages for Bitmap, BitmapImage, ImageSharp, Windows audio/video +- **Platform**: .NET 8+ target framework; Windows-specific projects use Windows SDK APIs +- **No External Model Loading**: Models are referenced by file path; users supply ONNX files + +## Nullable & Implicit Usings Policy + +- `Directory.Build.props` disables `ImplicitUsings` and `Nullable` globally +- Always include full namespace imports explicitly +- Null checks use `ArgumentNullException.ThrowIfNull()` pattern + +--- + +**Reference Examples**: See [TensorStack.Extractors/Pipelines/ExtractorPipeline.cs](../../TensorStack.Extractors/Pipelines/ExtractorPipeline.cs), [TensorStack.Common/Pipeline/IPipeline.cs](../../TensorStack.Common/Pipeline/IPipeline.cs), [TensorStack.Providers.CPU/Provider.cs](../../TensorStack.Providers.CPU/Provider.cs)