Skip to content

LLM snippet description #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Aditya-Mishra54 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

LLM snippet description #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
420a0dd
First commit
Aditya-Mishra54 Jun 4, 2025
1927ee3
Slight formatting issues
Aditya-Mishra54 Jun 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
251 changes: 251 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,257 @@ Update boilerplate_agent.py to create the following agents.

- Level 2: Refer to the CMD boilerplate agent, and update it (the sytem prompt and the json schema descriptions) so that it runs only aws cli commands.

## LLM Snippet

We use AWS Bedrock in order to communicate with an LLM.
You can try running these examples in services/llm.py

This is the outline to make a call to given LLM using Bedrock:
```python
raw = client.invoke_model(
modelId="anthropic.claude-3-5-sonnet-20240620-v1:0", # The given LLM ID
body=json.dumps(request_body), # Passing in the request body to the API
)
# Parsing the body response from Bedrock
response = json.loads(raw["body"].read())
```

Here are different request_bodys that we can send to the LLM:

### 1) One-shot prompt — minimal request / response:

The simplest call we can make to the API. In the input body to the API, we have to define three values
- messages, the list of message data from the user and assistant
- max_tokens, the "size" of the LLM response (4*max_tokens is approximately the max number of characters)
- anthropic_version, the API-schema date string

Input:
```
request_body = {
"messages": [
{ "role": "user", "content": "Hi" }
],
"max_tokens": 100,
"anthropic_version": "bedrock-2023-05-31"
}
```

Output from Bedrock API:
```
response = {
'id': 'msg_bdrk_01Lcgh98dzetRKiYzGEdkNBu',
'type': 'message',
'role': 'assistant',
'model': 'claude-3-5-sonnet-20240620',
'content': [{
'type': 'text',
'text': 'Hello! How can I assist you today? Feel free to ask any questions or let me know if you need help with anything.'
}],
'stop_reason': 'end_turn',
'stop_sequence': None,
'usage': {'input_tokens': 8, 'output_tokens': 29}
}
```

The response dictionary contains the metadata for the LLM reply. The 'content' field contains the response from the LLM.

### 2) Add conversation history through multiple messages

Messages can contain an entire conversation between the user and model, labeled respectively with "user" and "assistant" under 'role'.
- Messages must strictly alternate between user and alternate for Bedrock to accept the response
- Each message follows this json format with 'role' and 'content'.

Input:
```
request_body = {
"messages": [
{ "role": "user", "content": "Hi" },
{ "role": "assistant", "content": "Hello!" },
{ "role": "user", "content": "What is your name?" }
],
"max_tokens": 100,
"anthropic_version": "bedrock-2023-05-31"
}
```
Output from Bedrock API:
```
response = {
'id': 'msg_bdrk_01SLhsudGzqwCBcyLoo9qvwh',
'type': 'message',
'role': 'assistant',
'model': 'claude-3-5-sonnet-20240620',
'content': [{
'type': 'text',
'text': 'My name is Claude. It's nice to meet you!'
}],
'stop_reason': 'end_turn',
'stop_sequence': None,
'usage': {'input_tokens': 21, 'output_tokens': 15}
}
```


### 3) System prompt

We can use the system prompt to give specific instructions and any other information useful for the LLM.
- The system prompt is the first message sent to the LLM.
- It is given the highest priority of all messages.
- The user is unaware of any instruction outlined in the system prompt

Input:
```
request_body = {
"messages": [
{ "role": "user", "content": "Hi" },
{ "role": "assistant", "content": "Hello!" },
{ "role": "user", "content": "What is your name?" }
],
"max_tokens": 100,
"anthropic_version": "bedrock-2023-05-31",
"system": "You are a DUPLOCLOUD agent. Your name is DUPLO."
}
```
Output from Bedrock API:
```
response = {
'id': 'msg_bdrk_01CvBc4Vctw2iB56o2QW9jmE',
'type': 'message',
'role': 'assistant',
'model': 'claude-3-5-sonnet-20240620',
'content': [{
'type': 'text',
'text': "My name is Duplo. I'm an AI assistant created by DuploCloud to help with questions and tasks related to cloud infrastructure, DevOps, and software development. How can I assist you today?"
}],
'stop_reason': 'end_turn',
'stop_sequence': None,
'usage': {'input_tokens': 37, 'output_tokens': 47}
}
```

### 4) Structured JSON replies

Assume we need the response to be in the form of the following json format:
```
{
'content':"<response from LLM>",
'tone': "<Describes the tone of the message. Either angry, sad, happy, or funny.>"
}
```

There are two main ways of doing this.

- We can force a JSON response through system prompt.
Here, we use the system prompt's priority to enforce a restriction on all responses from the LLM. Since LLMs only deal in strings, the response will be a string as well but formatted as a JSON.

Input:
```
request_body = {
"messages": [
{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello!"},
{"role": "user", "content": "What is your name?"},
],
"max_tokens": 100,
"anthropic_version": "bedrock-2023-05-31",
"system" : """
You are a DUPLOCLOUD agent. Your name is DUPLO.
Every response you have, make it in a JSON format.
It should have two fields.
One is the 'content' where you output your normal response. The other is 'tone' where you have one of the following values: 'angry', 'sad', 'happy','funny'
"""
}
```
Output from Bedrock API:
```
response = {
'id': 'msg_bdrk_01XVLxJufNWbRgKKi4ujwFsE',
'type': 'message',
'role': 'assistant',
'model': 'claude-3-5-sonnet-20240620',
'content': [{'
type': 'text',
'text': "{ 'content': 'Hello there! My name is DUPLO. I\'m an AI agent created by DuploCloud to assist with various tasks and answer questions. How may I help you today?',\n 'tone': 'happy'\n}"
}],
'stop_reason':
'end_turn',
'stop_sequence': None,
'usage': {'input_tokens': 108, 'output_tokens': 58}}
```

Ideally, you would use the json library to convert the 'text' field inside of 'content' to strictly a dictionary.

- We can format the response with the use of tools.

Tools are built-in formatter. You can pass in several tools through the "tools" parameter. Each tool is formatting like the following:
```
example_tool = {
"name" : "Example tool name",
"description" : "<description of the tool>",
"input_schema":{
"type":"object",
"properties": {
...
}
}
}
```
You can use "input_schema" to dictate what the input for a tool call should look like.

"tool_choice" identifies to Bedrock and the LLM which tools are required or not required in its response.

Input:
```
request_body = {
"messages": [
{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello!"},
{"role": "user", "content": "What is your name?"},
],
"max_tokens": 100,
"anthropic_version": "bedrock-2023-05-31",
"system" : "You are a DUPLOCLOUD agent. Your name is DUPLO.",
"tools" : [{
"name":"tone",
"description":"Gives the tone of the message while also giving your regular response to the previous message",
"input_schema": {
"type":"object",
"properties": {
"content" : {
"type":"string",
"description" : "The normal response you would have for the user"
},
"tone" : {
"type":"string",
"enum": ["happy", "angry", "sad", "funny"],
"description" : "Should be the tone of the message"
}
}
}
}],
"tool_choice" : {"type": "tool", "name" : "tone"}
}
```

Output from Bedrock API:
```
response = {
'id': 'msg_bdrk_011w2xpjfyyf8hLCNaMJCNBF',
'type': 'message', 'role': 'assistant',
'model': 'claude-3-5-sonnet-20240620',
'content': [{
'type': 'tool_use',
'id': 'toolu_bdrk_01Ks3jE3VcoT4T8Wi1RLFLCv',
'name': 'tone',
'input': {
'content': "Hello! My name is DUPLO. I'm the DUPLOCLOUD agent, here to assist you with any questions or information you need about DUPLOCLOUD. How can I help you today?",
'tone': 'happy'
}
}],
'stop_reason': 'tool_use',
'stop_sequence': None,
'usage': {'input_tokens': 425, 'output_tokens': 91}}
```

## Coming Soon (Hopefully before tomorrow)

- A boilerplate for a tool calling agent using strands, so that you can create an agent by just adding some python functions with doc strings (tools) and by editing a system prompt
Expand Down
109 changes: 108 additions & 1 deletion services/llm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,4 +255,111 @@ def _extract_response(self, response_body: Dict[str, Any], model_id: str, tool_c
output = response_body["content"][0]["input"]
return output
else:
return response_body["content"][0]["text"]
return response_body["content"][0]["text"]


def call_bedrock(client, request_body: dict):
"""Invoke Claude and print the assistant text / tool input."""
raw = client.bedrock_runtime.invoke_model(
modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",
body=json.dumps(request_body),
)
response = json.loads(raw["body"].read())
return response

# tool_use replies store text under ['content'][0]['input'] instead of ['text']
leaf_key = "input" if "input" in response["content"][0] else "text"
LLM_content = response["content"][0][leaf_key]
return LLM_content

if __name__ == "__main__":
client = BedrockAnthropicLLM()

# 1) Simplest LLM call
basic_body_message = {
"messages": [{"role": "user", "content": "Hi"}],
"max_tokens": 1000,
"anthropic_version": "bedrock-2023-05-31",
}
basic_body_response = call_bedrock(client, basic_body_message)
print(basic_body_response)

# 2) LLM call with back-and-forth history
back_and_forth_message_body = {
"messages": [
{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello!"},
{"role": "user", "content": "What is your name?"},
],
"max_tokens": 1000,
"anthropic_version": "bedrock-2023-05-31"
}
back_and_forth_message_response = call_bedrock(client, back_and_forth_message_body)
print(back_and_forth_message_response)

# 3) Same with a system prompt
system_prompt_body = {
"messages": [
{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello!"},
{"role": "user", "content": "What is your name?"},
],
"max_tokens": 1000,
"anthropic_version": "bedrock-2023-05-31",
"system" : "You are a DUPLOCLOUD agent. Your name is DUPLO."
}
system_prompt_body_response = call_bedrock(client, system_prompt_body)
print(system_prompt_body_response)

# 4) Force JSON output via system-prompt *and* via tool-use
# 4a) Using the system prompt
JSON_system_prompt_body = {
"messages": [
{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello!"},
{"role": "user", "content": "What is your name?"},
],
"max_tokens": 1000,
"anthropic_version": "bedrock-2023-05-31",
"system" : """
You are a DUPLOCLOUD agent. Your name is DUPLO.
Every response you have, make it in a JSON format.
It should have two fields.
One is the 'content' where you output your normal response. The other is 'tone' where you have one of the following values: 'angry', 'sad', 'happy','funny'
"""
}
JSON_output_response = call_bedrock(client, JSON_system_prompt_body)
print(JSON_output_response)

tool_use_body = {
"messages": [
{"role": "user", "content": "Hi"},
{"role": "assistant", "content": "Hello!"},
{"role": "user", "content": "What is your name?"},
],
"max_tokens": 1000,
"anthropic_version": "bedrock-2023-05-31",
"system" : "You are a DUPLOCLOUD agent. Your name is DUPLO.",
"tools" : [{
"name":"tone",
"description":"Gives the tone of the message while also giving your regular response to the previous message",
"input_schema": {
"type":"object",
"properties": {
"content" : {
"type":"string",
"description" : "The normal response you would have for the user"
},
"tone" : {
"type":"string",
"enum": ["happy", "angry", "sad", "funny"],
"description" : "Should be the tone of the message"
}
}
}
}],
# Here we are forcing LLM to make use of the 'tone' tool we defined by passing in a tools_choice parameter
"tool_choice" : {"type": "tool", "name" : "tone"}
}
tool_use_respone = call_bedrock(client, tool_use_body)
print(tool_use_respone)