Add 2 PyRIT orchestrators ((Crescendo, PAIR)) and re-strucutre PyRIT code.

[samailguliyev]

Summary

This PR adds comprehensive PyRIT orchestrator enhancements including new orchestrator types (Crescendo, PAIR), tools, and CLI adjustements.

Changes Made

• Added system prompt file for SelfAskTrueFalseScorer
• Added agent instruction for orchestrator type input in agent mode
• Added clean_json() method to LLMAdapter
• Changed from inheritance to wrapper class approach for orchestrator agnostic functionality
• Added one runner function per orchestrator
• Added one tool per orchestrator
• Added 1 CLI command per orchestrator
• Added 1 attack specification case per orchestrator
• Added tools to agent

Tested by:

• Running individual CLI scripts for each PYRIT attack
• Running main.py and running vulnerability scan
• Running main.py and asking to run pyrit separately and passing inputs

[marcorosa]

Some notes while I do the code review, in no particular order:

• Missing note (in /backend-agent/data/pyrit) to explain to the agent how to use these attacks (the current one is still the old one, and does not explain to the agent that there are 3 sub-attacks in case the users ask for "pyrit".
• do not mix ' and " in your strings. Be consistent (and better use ' for strings in code). For reference, this random guy on the interned explains it very well: link
• Chose wisely the names of the attack/attack specification. Indeed, consider the name that will be written in the db, and think in advance of any issue it may cause using a name too long or containing _ or - or with spaces inside. Will users write this correctly?
• Please, re-run a python linter because I am not fully convinced it worked correctly

[marcorosa]

I confirm the linter action did not work:

Error: This action does not have permission to create annotations on forks. You may want to run it only on `pull_request_target` events with checks permissions set to write. See https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#permissions for details.

 Flake8 found 82 errors (failure)

So, you have 82 linter violations to fix 😄

[marcorosa]

If you fetch the upstream develop branch, I may have fixed the linter setup and it would run automatically

[samailguliyev]

I confirm the linter action did not work:

Error: This action does not have permission to create annotations on forks. You may want to run it only on `pull_request_target` events with checks permissions set to write. See https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#permissions for details.

 Flake8 found 82 errors (failure)

So, you have 82 linter violations to fix 😄

@marcorosa I fixed all your comments and all linter violations locally and fetched upstream develop branch, but backend and frontend actions fail to run for some reason.

[samailguliyev]

Some notes while I do the code review, in no particular order:

• Missing note (in /backend-agent/data/pyrit) to explain to the agent how to use these attacks (the current one is still the old one, and does not explain to the agent that there are 3 sub-attacks in case the users ask for "pyrit".
• do not mix ' and " in your strings. Be consistent (and better use ' for strings in code). For reference, this random guy on the interned explains it very well: link
• Chose wisely the names of the attack/attack specification. Indeed, consider the name that will be written in the db, and think in advance of any issue it may cause using a name too long or containing _ or - or with spaces inside. Will users write this correctly?
• Please, re-run a python linter because I am not fully convinced it worked correctly

done, github action does not work though.

[marcorosa]

small adjustments still needed, mostly in the note phrasing

[github-actions]

This update introduces a significant overhaul of LLM attack functionalities, specifically enhancing the PyRIT attack framework. It refines how attacks are executed by expanding the variety and specificity of attack types available. The changes aim to make the attack process easier to configure and more versatile, thereby improving the user experience for security professionals working with LLM vulnerability assessments.

Walkthrough

• New Feature: Introduced new attack types — "redteaming", "crescendo", and "pair" — to the PyRIT framework, along with corresponding tools and orchestrators.
• Refactor: Updated naming conventions and reframed orchestrator usage, providing more modular and maintainable code architecture.
• Chore: Expanded attack configuration capabilities and adjusted default attack parameters for better personalization.
• Documentation: Enhanced instructions and notes to guide users on specifying attack types and parameters efficiently.
• Style: Minor styling adjustments for improved readability and consistency in code formatting.

_{Model: gpt-4o | Prompt Tokens: 8804 | Completion Tokens: 191}

[github-actions]

Here's a collaborative code review enhanced by AI assistance. These insights offer suggestions and observations that may help improve your work, though they're not absolute truths. You remain the expert on your project's needs and goals. Consider these recommendations as supportive guidance while you make the final decisions that align best with your vision and requirements.

---

Always critique what AI says. Do not let AI replace YOUR I.
_{Model: anthropic--claude-4-sonnet | Prompt Tokens: 14851 | Completion Tokens: 2533}

[marcorosa]

File true_false_system_prompt.yaml is copied in both backend-agent/data and backend-agent/libs/data. It should appear only in the latter.

[samailguliyev]

File true_false_system_prompt.yaml is copied in both backend-agent/data and backend-agent/libs/data. It should appear only in the latter.

done