-
Notifications
You must be signed in to change notification settings - Fork 5
Move llmAsJury folder to patterns directory #21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| --- | ||
| description: Guidelines for writing Agentuity AI Agents in TypeScript | ||
| globs: "**/src/agents/**/index.ts" | ||
| alwaysApply: true | ||
| --- | ||
|
|
||
| # AI Agent File | ||
|
|
||
| - Prefer using the `agentuity agent create` command to create a new Agent | ||
| - Prefer loading types from the node modules package `@agentuity/sdk` in the node_modules folder | ||
| - The file should export a default function | ||
| - Prefer naming the default function Agent or the name of the Agent based on the context of the Agent description | ||
| - All code should be in Typescript format | ||
| - Use the provided logger from the `AgentContext` interface such as `ctx.logger.info("my message: %s", "hello")` | ||
|
|
||
| ## Example Agent File | ||
|
|
||
| ```typescript | ||
| import type { AgentRequest, AgentResponse, AgentContext } from "@agentuity/sdk"; | ||
|
|
||
| export default async function Agent(req: AgentRequest, resp: AgentResponse, ctx: AgentContext) { | ||
| return resp.json({"hello":"world"}); | ||
| } | ||
| ``` | ||
|
|
||
| ### AgentRequest | ||
|
|
||
| The AgentRequest interface provides a set of helper methods and public variables which can be used for working with data has been passed to the Agent. | ||
|
|
||
| ### AgentResponse | ||
|
|
||
| The AgentResponse interface provides a set of helper methods for responding with different data formats from the Agent. | ||
|
|
||
| ### AgentContext | ||
|
|
||
| The AgentContext has information specific to the incoming Agent request and a set of helper methods for accessing AI services like KeyValue storage and Vector storage. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| --- | ||
| description: Guidelines for the Agentuity AI Configuration file | ||
| globs: "agentuity.yaml" | ||
| alwaysApply: true | ||
| --- | ||
|
|
||
| # Agentuity Configuration File | ||
|
|
||
| This file is used by agentuity to configure the AI Agent project. You should NOT suggest edits to this file. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| --- | ||
| description: Agentuity JavaScript SDK API Reference | ||
| globs: "src/agents/**/*.ts" | ||
| alwaysApply: false | ||
| --- | ||
|
|
||
| # Agentuity JavaScript SDK | ||
|
|
||
| The Agentuity JavaScript SDK provides a powerful framework for building AI agents in JavaScript and TypeScript. This cursor rules file helps you navigate the SDK's core interfaces and methods. | ||
|
|
||
| ## Core Interfaces | ||
|
|
||
| ### AgentHandler | ||
|
|
||
| The main handler function type for an agent: | ||
|
|
||
| ```typescript | ||
| type AgentHandler = ( | ||
| request: AgentRequest, | ||
| response: AgentResponse, | ||
| context: AgentContext | ||
| ) => Promise<AgentResponseType>; | ||
| ``` | ||
|
|
||
| ### AgentRequest | ||
|
|
||
| The `AgentRequest` interface provides methods for accessing request data: | ||
|
|
||
| - `request.trigger`: Gets the trigger type of the request | ||
| - `request.metadata(key, defaultValue)`: Gets metadata associated with the request | ||
| - `request.get(key, defaultValue)`: Gets the metadata value of the request | ||
| - `request.data.contentType`: Gets the content type of the request payload | ||
| - `request.data.json(): Promise<Json>`: Gets the payload as a JSON object | ||
| - `request.data.text(): Promise<string>`: Gets the payload as a string | ||
| - `request.data.buffer(): Promise<ArrayBuffer>`: Gets the payload as a ArrayBuffer | ||
| - `request.data.binary(): Promise<ArrayBuffer>`: Gets the payload as a ArrayBuffer | ||
| - `request.data.object<T>: Promise<T>`: Gets the payload as a typed object | ||
|
|
||
| ### AgentResponse | ||
|
|
||
| The `AgentResponse` interface provides methods for creating responses: | ||
|
|
||
| - `response.json(data, metadata)`: Creates a JSON response | ||
| - `response.text(data, metadata)`: Creates a text response | ||
| - `response.binary(data, metadata)`: Creates a binary response | ||
| - `response.html(data, metadata)`: Creates an HTML response | ||
| - `response.empty(metadata)`: Creates an empty response | ||
| - `response.handoff(agent, args?)`: Redirects to another agent within the same project | ||
|
|
||
| ### AgentContext | ||
|
|
||
| The `AgentContext` interface provides access to various capabilities: | ||
|
|
||
| - `context.logger`: Logging functionality | ||
| - `context.kv`: Key-Value storage | ||
| - `context.vector`: Vector storage | ||
| - `context.getAgent(params)`: Gets a handle to a remote agent | ||
| - `context.tracer`: OpenTelemetry tracing | ||
|
|
||
| ## Storage APIs | ||
|
|
||
| ### Key-Value Storage | ||
|
|
||
| Access through `context.kv`: | ||
|
|
||
| - `context.kv.get(name, key)`: Retrieves a value | ||
| - `context.kv.set(name, key, value, params)`: Stores a value with optional params (KeyValueStorageSetParams) | ||
| - `context.kv.delete(name, key)`: Deletes a value | ||
|
|
||
| ### Vector Storage | ||
|
|
||
| Access through `context.vector`: | ||
|
|
||
| - `context.vector.upsert(name, ...documents)`: Inserts or updates vectors | ||
| - `context.vector.search(name, params)`: Searches for vectors | ||
| - `context.vector.delete(name, ...ids)`: Deletes vectors | ||
|
|
||
| ## Logging | ||
|
|
||
| Access through `context.logger`: | ||
|
|
||
| - `context.logger.debug(message, ...args)`: Logs a debug message | ||
| - `context.logger.info(message, ...args)`: Logs an informational message | ||
| - `context.logger.warn(message, ...args)`: Logs a warning message | ||
| - `context.logger.error(message, ...args)`: Logs an error message | ||
| - `context.logger.child(opts)`: Creates a child logger with additional context | ||
|
|
||
| ## Best Practices | ||
|
|
||
| - Use TypeScript for better type safety and IDE support | ||
| - Import types from `@agentuity/sdk` | ||
| - Use structured error handling with try/catch blocks | ||
| - Leverage the provided logger for consistent logging | ||
| - Use the storage APIs for persisting data | ||
| - Consider agent communication for complex workflows | ||
|
|
||
| For complete documentation, visit: https://agentuity.dev/SDKs/javascript/api-reference | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| # EditorConfig is awesome: https://EditorConfig.org | ||
|
|
||
| # top-most EditorConfig file | ||
| root = true | ||
|
|
||
| [*] | ||
| indent_style = tab | ||
| indent_size = 2 | ||
| end_of_line = lf | ||
| charset = utf-8 | ||
| trim_trailing_whitespace = false | ||
| insert_final_newline = false |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # dependencies (bun install) | ||
| node_modules | ||
|
|
||
| # output | ||
| out | ||
| dist | ||
| *.tgz | ||
|
|
||
| # code coverage | ||
| coverage | ||
| *.lcov | ||
|
|
||
| # logs | ||
| logs | ||
| _.log | ||
joel13samuel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json | ||
|
|
||
| # dotenv environment variable files | ||
| .env | ||
| .env.development.local | ||
| .env.test.local | ||
| .env.production.local | ||
| .env.local | ||
|
|
||
| # caches | ||
| .eslintcache | ||
| .cache | ||
| *.tsbuildinfo | ||
|
|
||
| # IntelliJ based IDEs | ||
| .idea | ||
|
|
||
| # Finder (MacOS) folder config | ||
| .DS_Store | ||
| # don't commit the agentuity build folder | ||
| .agentuity | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| <div align="center"> | ||
| <img src="https://raw.githubusercontent.com/agentuity/cli/refs/heads/main/.github/Agentuity.png" alt="Agentuity" width="100"/> <br/> | ||
| <strong>Build Agents, Not Infrastructure</strong> <br/> | ||
| <br /> | ||
| </div> | ||
|
|
||
|
|
||
| # 🤖 LLM as Jury System | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Needs deploy button. |
||
|
|
||
| [](https://app.agentuity.com/deploy) | ||
|
|
||
| This project uses Agentuity to create a multi-agent system where one AI agent (ContentWriter) creates blog posts, and another AI agent (Jury) evaluates them on multiple criteria using different AI models. | ||
|
|
||
| ## Overview | ||
|
|
||
| - **ContentWriter Agent**: Takes a topic as input and generates a well-structured blog post about that topic | ||
| - **Jury Agent**: Evaluates blog posts across multiple criteria (readability, technical accuracy, engagement) using multiple AI models for balanced assessment | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. The ContentWriter agent receives a topic and uses OpenAI to generate a high-quality blog post | ||
| 2. The Jury agent receives the blog post and evaluates it using multiple specialized "judge" LLMs | ||
| 3. Each judge evaluates the blog post on specific criteria and provides scores out of 10 | ||
| 4. The Jury agent combines all evaluations and returns a comprehensive assessment | ||
|
|
||
| ## 🚀 Quick Start | ||
|
|
||
| ### Prerequisites | ||
| - **Bun**: Version 1.2.4 or higher | ||
|
|
||
| ### Setup | ||
| 1. Install dependencies: `bun install` | ||
| 2. Authenticate with Agentuity: `agentuity login` | ||
| 3. Start development mode: `agentuity dev` | ||
|
|
||
| ## 🎯 Using the Agents | ||
|
|
||
| ### Via DevMode UI | ||
| 1. Open the DevMode URL provided when you start `agentuity dev` | ||
| 2. **Generate Content**: Select ContentWriter agent → Enter a topic → Get blog post | ||
| 3. **Evaluate Content**: Select Jury agent → Paste blog post → Get detailed evaluation | ||
|
|
||
| ### Via CLI Test Client | ||
| ```bash | ||
| # Generate a blog post on a topic | ||
| bun run index.ts ContentWriter "artificial intelligence" | ||
|
|
||
| # Evaluate a blog post | ||
| bun run index.ts Jury "Your blog post content here..." | ||
|
|
||
| # Run the full workflow (ContentWriter -> Jury) | ||
| bun run index.ts workflow "technology trends" | ||
| ``` | ||
|
|
||
| ## 🔧 Agent Details | ||
|
|
||
| ### ContentWriter | ||
| Uses the Mastra framework with OpenAI's gpt-4o-mini model to generate blog posts with: | ||
| - Engaging titles | ||
| - Clear introductions | ||
| - Well-organized body paragraphs with subheadings | ||
| - Strong conclusions | ||
|
|
||
| ### Jury | ||
| A multi-model evaluation system that provides balanced assessment using: | ||
|
|
||
| **Default Models:** | ||
| - **GPT-4o Mini**: Precise and thorough evaluator | ||
| - **GPT-4o**: Critical and detailed evaluator focused on technical merits | ||
| - **Claude**: Pretty cool model I can't lie | ||
|
|
||
| **Evaluation Criteria:** | ||
| - Clarity | ||
| - Structure | ||
| - Engagement | ||
| - Technical accuracy | ||
|
|
||
|
|
||
| To add other models (Grok, Llama, Mistral), install the appropriate SDK and update the Jury agent code. | ||
|
|
||
| ## 📁 Project Structure | ||
|
|
||
| ``` | ||
| ├── agents/ # Agent implementations | ||
| ├── node_modules/ # Dependencies | ||
| ├── package.json # Project dependencies | ||
| ├── agentuity.yaml # Agentuity configuration | ||
| └── index.ts # CLI test client | ||
| ``` | ||
|
|
||
| ## 🌐 Development & Deployment | ||
|
|
||
| ### Development Mode | ||
| ```bash | ||
| agentuity dev | ||
| ``` | ||
| Opens browser with Agentuity Console in Live Mode for real-time testing. | ||
|
|
||
| ### Creating New Agents | ||
| ```bash | ||
| agentuity agent new | ||
| ``` | ||
|
|
||
| ### Deployment | ||
| ```bash | ||
| agentuity deploy | ||
| ``` | ||
|
|
||
| ### Environment Variables | ||
| ```bash | ||
| agentuity env set KEY=VALUE | ||
| agentuity env set --secret KEY=VALUE # for sensitive data | ||
| ``` | ||
|
|
||
| ## 📖 Documentation | ||
|
|
||
| For comprehensive Agentuity SDK documentation: [https://agentuity.dev/SDKs/javascript](https://agentuity.dev/SDKs/javascript) | ||
|
|
||
| ## 🆘 Support | ||
|
|
||
| - [Documentation](https://agentuity.dev/SDKs/javascript) | ||
| - [Discord Community](https://discord.com/invite/vtn3hgUfuc) | ||
| - Agentuity Support Team | ||
|
|
||
| ## 📝 License | ||
|
|
||
| This project is licensed under the terms specified in the LICENSE file. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # yaml-language-server: $schema=https://raw.githubusercontent.com/agentuity/cli/refs/heads/main/agentuity.schema.json | ||
|
|
||
| # ------------------------------------------------ | ||
| # This file is generated by Agentuity | ||
| # You should check this file into version control | ||
| # ------------------------------------------------ | ||
|
|
||
| # The version semver range required to run this project | ||
| version: '>=0.0.131' | ||
| # The ID of the project which is automatically generated | ||
| project_id: proj_d9627b411dba11cfaf9499f8c869e71a | ||
| # The name of the project which is editable | ||
| name: llmAsJury | ||
| # The description of the project which is editable | ||
| description: "" | ||
| # The development configuration for the project | ||
| development: | ||
| # The port to run the development server on which can be overridden by setting the PORT environment variable | ||
| port: 3500 | ||
| watch: | ||
| # Whether to watch for changes and automatically restart the server | ||
| enabled: true | ||
| # Rules for files to watch for changes | ||
| files: | ||
| - src/** | ||
| # The command to run the development server | ||
| command: bun | ||
| # The arguments to pass to the development server | ||
| args: | ||
| - run | ||
| - --silent | ||
| - .agentuity/index.js | ||
| deployment: | ||
| command: bun | ||
| args: | ||
| - run | ||
| - --no-install | ||
| - --prefer-offline | ||
| - --silent | ||
| - --no-macros | ||
| - --no-global-search-paths | ||
| - --report-uncaught-exception | ||
| - --disable-sigusr1 | ||
| - --disallow-code-generation-from-strings | ||
| - --no-addons | ||
| - --no-deprecation | ||
| - .agentuity/index.js | ||
| # You should tune the resources for the deployment | ||
| resources: | ||
| # The memory requirements | ||
| memory: 250Mi | ||
| # The CPU requirements | ||
| cpu: 500M | ||
| # The disk size requirements | ||
| disk: 100Mi | ||
| # You should not need to change these value | ||
| bundler: | ||
| enabled: true | ||
| identifier: bunjs | ||
| language: javascript | ||
| runtime: bunjs | ||
| agents: | ||
| dir: src/agents | ||
| ignore: | ||
| - node_modules/** | ||
| - dist/** | ||
| - src/** | ||
joel13samuel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # The agents that are part of this project | ||
| agents: | ||
| - # The ID of the Agent which is automatically generated | ||
| id: agent_a7e9d6109aa3c31a69544027e58354aa | ||
| # The name of the Agent which is editable | ||
| name: Jury | ||
| - id: agent_2543cab6ee778d150cf5ee1ca97f2518 | ||
| name: ContentWriter | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.