Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions patterns/llmAsJury/.cursor/rules/agent.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
description: Guidelines for writing Agentuity AI Agents in TypeScript
globs: "**/src/agents/**/index.ts"
alwaysApply: true
---

# AI Agent File

- Prefer using the `agentuity agent create` command to create a new Agent
- Prefer loading types from the node modules package `@agentuity/sdk` in the node_modules folder
- The file should export a default function
- Prefer naming the default function Agent or the name of the Agent based on the context of the Agent description
- All code should be in Typescript format
- Use the provided logger from the `AgentContext` interface such as `ctx.logger.info("my message: %s", "hello")`

## Example Agent File

```typescript
import type { AgentRequest, AgentResponse, AgentContext } from "@agentuity/sdk";

export default async function Agent(req: AgentRequest, resp: AgentResponse, ctx: AgentContext) {
return resp.json({"hello":"world"});
}
```

### AgentRequest

The AgentRequest interface provides a set of helper methods and public variables which can be used for working with data has been passed to the Agent.

### AgentResponse

The AgentResponse interface provides a set of helper methods for responding with different data formats from the Agent.

### AgentContext

The AgentContext has information specific to the incoming Agent request and a set of helper methods for accessing AI services like KeyValue storage and Vector storage.
9 changes: 9 additions & 0 deletions patterns/llmAsJury/.cursor/rules/agentuity.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
description: Guidelines for the Agentuity AI Configuration file
globs: "agentuity.yaml"
alwaysApply: true
---

# Agentuity Configuration File

This file is used by agentuity to configure the AI Agent project. You should NOT suggest edits to this file.
97 changes: 97 additions & 0 deletions patterns/llmAsJury/.cursor/rules/sdk.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
description: Agentuity JavaScript SDK API Reference
globs: "src/agents/**/*.ts"
alwaysApply: false
---

# Agentuity JavaScript SDK

The Agentuity JavaScript SDK provides a powerful framework for building AI agents in JavaScript and TypeScript. This cursor rules file helps you navigate the SDK's core interfaces and methods.

## Core Interfaces

### AgentHandler

The main handler function type for an agent:

```typescript
type AgentHandler = (
request: AgentRequest,
response: AgentResponse,
context: AgentContext
) => Promise<AgentResponseType>;
```

### AgentRequest

The `AgentRequest` interface provides methods for accessing request data:

- `request.trigger`: Gets the trigger type of the request
- `request.metadata(key, defaultValue)`: Gets metadata associated with the request
- `request.get(key, defaultValue)`: Gets the metadata value of the request
- `request.data.contentType`: Gets the content type of the request payload
- `request.data.json(): Promise<Json>`: Gets the payload as a JSON object
- `request.data.text(): Promise<string>`: Gets the payload as a string
- `request.data.buffer(): Promise<ArrayBuffer>`: Gets the payload as a ArrayBuffer
- `request.data.binary(): Promise<ArrayBuffer>`: Gets the payload as a ArrayBuffer
- `request.data.object<T>: Promise<T>`: Gets the payload as a typed object

### AgentResponse

The `AgentResponse` interface provides methods for creating responses:

- `response.json(data, metadata)`: Creates a JSON response
- `response.text(data, metadata)`: Creates a text response
- `response.binary(data, metadata)`: Creates a binary response
- `response.html(data, metadata)`: Creates an HTML response
- `response.empty(metadata)`: Creates an empty response
- `response.handoff(agent, args?)`: Redirects to another agent within the same project

### AgentContext

The `AgentContext` interface provides access to various capabilities:

- `context.logger`: Logging functionality
- `context.kv`: Key-Value storage
- `context.vector`: Vector storage
- `context.getAgent(params)`: Gets a handle to a remote agent
- `context.tracer`: OpenTelemetry tracing

## Storage APIs

### Key-Value Storage

Access through `context.kv`:

- `context.kv.get(name, key)`: Retrieves a value
- `context.kv.set(name, key, value, params)`: Stores a value with optional params (KeyValueStorageSetParams)
- `context.kv.delete(name, key)`: Deletes a value

### Vector Storage

Access through `context.vector`:

- `context.vector.upsert(name, ...documents)`: Inserts or updates vectors
- `context.vector.search(name, params)`: Searches for vectors
- `context.vector.delete(name, ...ids)`: Deletes vectors

## Logging

Access through `context.logger`:

- `context.logger.debug(message, ...args)`: Logs a debug message
- `context.logger.info(message, ...args)`: Logs an informational message
- `context.logger.warn(message, ...args)`: Logs a warning message
- `context.logger.error(message, ...args)`: Logs an error message
- `context.logger.child(opts)`: Creates a child logger with additional context

## Best Practices

- Use TypeScript for better type safety and IDE support
- Import types from `@agentuity/sdk`
- Use structured error handling with try/catch blocks
- Leverage the provided logger for consistent logging
- Use the storage APIs for persisting data
- Consider agent communication for complex workflows

For complete documentation, visit: https://agentuity.dev/SDKs/javascript/api-reference
12 changes: 12 additions & 0 deletions patterns/llmAsJury/.editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# EditorConfig is awesome: https://EditorConfig.org

# top-most EditorConfig file
root = true

[*]
indent_style = tab
indent_size = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = false
insert_final_newline = false
36 changes: 36 additions & 0 deletions patterns/llmAsJury/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# dependencies (bun install)
node_modules

# output
out
dist
*.tgz

# code coverage
coverage
*.lcov

# logs
logs
_.log
report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json

# dotenv environment variable files
.env
.env.development.local
.env.test.local
.env.production.local
.env.local

# caches
.eslintcache
.cache
*.tsbuildinfo

# IntelliJ based IDEs
.idea

# Finder (MacOS) folder config
.DS_Store
# don't commit the agentuity build folder
.agentuity
127 changes: 127 additions & 0 deletions patterns/llmAsJury/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
<div align="center">
<img src="https://raw.githubusercontent.com/agentuity/cli/refs/heads/main/.github/Agentuity.png" alt="Agentuity" width="100"/> <br/>
<strong>Build Agents, Not Infrastructure</strong> <br/>
<br />
</div>


# 🤖 LLM as Jury System
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs deploy button.


[![Deploy with Agentuity](https://app.agentuity.com/img/deploy.svg)](https://app.agentuity.com/deploy)

This project uses Agentuity to create a multi-agent system where one AI agent (ContentWriter) creates blog posts, and another AI agent (Jury) evaluates them on multiple criteria using different AI models.

## Overview

- **ContentWriter Agent**: Takes a topic as input and generates a well-structured blog post about that topic
- **Jury Agent**: Evaluates blog posts across multiple criteria (readability, technical accuracy, engagement) using multiple AI models for balanced assessment

## How It Works

1. The ContentWriter agent receives a topic and uses OpenAI to generate a high-quality blog post
2. The Jury agent receives the blog post and evaluates it using multiple specialized "judge" LLMs
3. Each judge evaluates the blog post on specific criteria and provides scores out of 10
4. The Jury agent combines all evaluations and returns a comprehensive assessment

## 🚀 Quick Start

### Prerequisites
- **Bun**: Version 1.2.4 or higher

### Setup
1. Install dependencies: `bun install`
2. Authenticate with Agentuity: `agentuity login`
3. Start development mode: `agentuity dev`

## 🎯 Using the Agents

### Via DevMode UI
1. Open the DevMode URL provided when you start `agentuity dev`
2. **Generate Content**: Select ContentWriter agent → Enter a topic → Get blog post
3. **Evaluate Content**: Select Jury agent → Paste blog post → Get detailed evaluation

### Via CLI Test Client
```bash
# Generate a blog post on a topic
bun run index.ts ContentWriter "artificial intelligence"

# Evaluate a blog post
bun run index.ts Jury "Your blog post content here..."

# Run the full workflow (ContentWriter -> Jury)
bun run index.ts workflow "technology trends"
```

## 🔧 Agent Details

### ContentWriter
Uses the Mastra framework with OpenAI's gpt-4o-mini model to generate blog posts with:
- Engaging titles
- Clear introductions
- Well-organized body paragraphs with subheadings
- Strong conclusions

### Jury
A multi-model evaluation system that provides balanced assessment using:

**Default Models:**
- **GPT-4o Mini**: Precise and thorough evaluator
- **GPT-4o**: Critical and detailed evaluator focused on technical merits
- **Claude**: Pretty cool model I can't lie

**Evaluation Criteria:**
- Clarity
- Structure
- Engagement
- Technical accuracy


To add other models (Grok, Llama, Mistral), install the appropriate SDK and update the Jury agent code.

## 📁 Project Structure

```
├── agents/ # Agent implementations
├── node_modules/ # Dependencies
├── package.json # Project dependencies
├── agentuity.yaml # Agentuity configuration
└── index.ts # CLI test client
```

## 🌐 Development & Deployment

### Development Mode
```bash
agentuity dev
```
Opens browser with Agentuity Console in Live Mode for real-time testing.

### Creating New Agents
```bash
agentuity agent new
```

### Deployment
```bash
agentuity deploy
```

### Environment Variables
```bash
agentuity env set KEY=VALUE
agentuity env set --secret KEY=VALUE # for sensitive data
```

## 📖 Documentation

For comprehensive Agentuity SDK documentation: [https://agentuity.dev/SDKs/javascript](https://agentuity.dev/SDKs/javascript)

## 🆘 Support

- [Documentation](https://agentuity.dev/SDKs/javascript)
- [Discord Community](https://discord.com/invite/vtn3hgUfuc)
- Agentuity Support Team

## 📝 License

This project is licensed under the terms specified in the LICENSE file.
75 changes: 75 additions & 0 deletions patterns/llmAsJury/agentuity.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/agentuity/cli/refs/heads/main/agentuity.schema.json

# ------------------------------------------------
# This file is generated by Agentuity
# You should check this file into version control
# ------------------------------------------------

# The version semver range required to run this project
version: '>=0.0.131'
# The ID of the project which is automatically generated
project_id: proj_d9627b411dba11cfaf9499f8c869e71a
# The name of the project which is editable
name: llmAsJury
# The description of the project which is editable
description: ""
# The development configuration for the project
development:
# The port to run the development server on which can be overridden by setting the PORT environment variable
port: 3500
watch:
# Whether to watch for changes and automatically restart the server
enabled: true
# Rules for files to watch for changes
files:
- src/**
# The command to run the development server
command: bun
# The arguments to pass to the development server
args:
- run
- --silent
- .agentuity/index.js
deployment:
command: bun
args:
- run
- --no-install
- --prefer-offline
- --silent
- --no-macros
- --no-global-search-paths
- --report-uncaught-exception
- --disable-sigusr1
- --disallow-code-generation-from-strings
- --no-addons
- --no-deprecation
- .agentuity/index.js
# You should tune the resources for the deployment
resources:
# The memory requirements
memory: 250Mi
# The CPU requirements
cpu: 500M
# The disk size requirements
disk: 100Mi
# You should not need to change these value
bundler:
enabled: true
identifier: bunjs
language: javascript
runtime: bunjs
agents:
dir: src/agents
ignore:
- node_modules/**
- dist/**
- src/**
# The agents that are part of this project
agents:
- # The ID of the Agent which is automatically generated
id: agent_a7e9d6109aa3c31a69544027e58354aa
# The name of the Agent which is editable
name: Jury
- id: agent_2543cab6ee778d150cf5ee1ca97f2518
name: ContentWriter
Loading