diff --git a/source/img/guides/add_assessment.png b/source/img/guides/add_assessment.png index bac015f2..5d1e1e0d 100644 Binary files a/source/img/guides/add_assessment.png and b/source/img/guides/add_assessment.png differ diff --git a/source/img/guides/rubricfinal.png b/source/img/guides/rubricfinal.png new file mode 100644 index 00000000..8a9dcc04 Binary files /dev/null and b/source/img/guides/rubricfinal.png differ diff --git a/source/instructors/authoring/assessments/add-assessment.rst b/source/instructors/authoring/assessments/add-assessment.rst index 3a9fdd38..d79702f1 100644 --- a/source/instructors/authoring/assessments/add-assessment.rst +++ b/source/instructors/authoring/assessments/add-assessment.rst @@ -47,14 +47,14 @@ To add a new assessment for a course, follow these steps: .. image:: /img/guides/add_assessment.png :alt: Add Assessment -3. Click **Create**. +3. Fill the appropriate fields and click **Create**. While creating/editing the assessment, you can still see/edit the guide page. Assessment Auto-Generation ++++++++++++++++++++++++++ -Assessments can be auto-generated based on the text found on a guides page. Currently only :ref:`Multiple Choice `, :ref:`Fill in the Blanks `, :ref:`Free Text ` and :ref:`Parsons Puzzle ` assessments can be auto-generated. For more information, please check out :ref:`Generating Assessments With AI ` +Assessments can be auto-generated based on the text found on a guides page. Currently only :ref:`Multiple Choice `, :ref:`Fill in the Blanks `, :ref:`Free Text `, :ref:`Standard Code Test `, and :ref:`Parsons Puzzle ` assessments can be auto-generated. For more information, please check out :ref:`Generating Assessments With AI ` Refer to the specific topics for the type of assessment: @@ -68,3 +68,4 @@ Refer to the specific topics for the type of assessment: - :ref:`Grade Book ` - :ref:`Parsons Puzzle ` - :ref:`Random Assessment ` +- :ref:`LLM Rubric and LLM Rubric Autograde ` \ No newline at end of file diff --git a/source/instructors/authoring/assessments/llm-based-rubric.rst b/source/instructors/authoring/assessments/llm-based-rubric.rst index 7344e2bc..7770843a 100644 --- a/source/instructors/authoring/assessments/llm-based-rubric.rst +++ b/source/instructors/authoring/assessments/llm-based-rubric.rst @@ -6,20 +6,27 @@ LLM-Based Rubric Grader ======================= -You can find the **LLM Based Rubric** assessment in the **Manually Graded** section of the Assessments tab. More information about adding assessments can be found in our :ref:`assessment documentation `. -There are three steps to this process. +Codio has provided two versions of the **LLM-Based Rubric Grader**. One requires the instructor to approve the LLM-generated feedback, and the other automatically displays it to the student. + +The **LLM Rubric Autograde** assessment is in the **Auto-Graded** section of the assessments menu and the **LLM Rubric** assessment is in the **Manually Graded** section. +More information about adding assessments can be found in our :ref:`assessment documentation `. + +The first two steps are the same for both versions: 1. Rubric generation 2. LLM-based grading using the generated rubric + +.. Note:: Step 3 is required only for the manually graded version - LLM feedback will not be released to the student until the instructor reviews and approves it. + 3. Manual approval of LLM-generated grades by the instructor -.. Note:: Step 3 is required - LLM feedback will not be released to the student until the instructor reviews and approves it. + LLM-Based Rubric Creation (Step 1) ---------------------------------- -Add an **LLM Based Rubric** assessment to your guide page and follow the steps below. +Add an **LLM Rubric** or **LLM Rubric Autograde** assessment to your guide page and follow the steps below. 1. On the **General** page, enter the following information: @@ -34,41 +41,39 @@ Add an **LLM Based Rubric** assessment to your guide page and follow the steps b - Add a solution file (1) if you wish the rubric creation process to consider your solution. - .. image:: /img/guides/llmbasedrubric.png - :height: 600 - :alt: Generate a rubric +.. image:: /img/guides/llmbasedrubric.png + :height: 600 + :alt: Generate a rubric - - Click the **Generate Rubrics** (2) button to initiate the process. +- Click the **Generate Rubrics** (2) button to initiate the process. - The **Rubric Creation Agent** uses the following items to generate the rubric items: +The **Rubric Creation Agent** uses the following items to generate the rubric items: - - The assessment name - - Instructions provided in the **General** tab of the assessment - - Content of the Guide Page where the assessment is being added - - Contents of the provided solution file - - The Course, Module, and Assignment name - - Requirements specified in the Rubric creation tab - - .. Note:: If you do not add rubric requirements; the process will use general code grading norms to supply rubric items. - - (optional) Add your requirements in the **Rubrics Requirements** dialog: +- The assessment name +- Instructions provided in the **General** tab of the assessment +- Content of the Guide Page where the assessment is being added +- Contents of the provided solution file +- The Course, Module, and Assignment name +- Requirements specified in the Rubric creation tab - .. image:: /img/guides/llmrubricreqs.png - :height: 600 - :alt: Area to add your rubric requirements +.. Note:: If you do not add rubric requirements; the process will use general code grading norms to supply rubric items. +Add your requirements in the **Rubric Requirements** dialog (optional): - - Once you are done, click **Generate Using AI**. +.. image:: /img/guides/llmrubricreqs.png + :height: 600 + :alt: Area to add your rubric requirements - - You can provide additional rubric items by clicking **Add Rubric** and entering information. - - Once you have reviewed the rubric items and other settings, click **Save** to save the assessment. +- Once you are done, click **Generate Using AI**. +- You can provide additional rubric items by clicking **Add Rubric** and entering information. +- Once you have reviewed the rubric items and other settings, click **Save** to save the assessment. LLM Grading Based on the Created Rubric (Step 2) ------------------------------------------------ -The LLM Grading agent uses the following to grade the student work: +The grading occurs when the student clicks the **Check It** button. The LLM Grading agent uses the following to grade the student's work: - Instructions provided in the **General** tab of the assessment - Contents of the Guide page where the assessment is located @@ -76,20 +81,34 @@ The LLM Grading agent uses the following to grade the student work: - The student file - The rubric generated in the previous step to identify the grading criteria -.. Note:: The grading occurs when the student clicks the **Check It** button. The student receives information about whether they have passed or failed each rubric item, but does not see the rest of the LLM-generated feedback until after the instructor conducts their review. +Sample feedback for the Auto-Graded Version: + +.. image:: /img/guides/rubricfinal.png + :height: 600 + :alt: Final grading information displayed to the student + + +Manual Approval by Instructor (Step 3 - only for the manually-graded version) +----------------------------------------------------------------------------- +.. Note:: The student receives information about whether they have passed or failed each rubric item, but does not see the rest of the LLM-generated feedback until after the instructor conducts their review. + + +Sample student view of the feedback, before instructor approval: .. image:: /img/guides/rubricprelim.png :height: 600 :alt: Preliminary grading information displayed to the student +In the case of the manually graded version of this assessment, the instructor must open the student's assignment and select the passing test cases as part of the :ref:`grading process `. The instructor can also edit the comments generated by the LLM. Once this process is complete, click on **Apply Grade**. + +.. Warning:: Once feedback is released to students, it cannot be modified. -Manual Approval by Instructor (Step 3) --------------------------------------- -The final step involves the instructor opening the student assignment and selecting the passing test cases as part of the :ref:`grading process `. The instructor can also edit the comments generated by the LLM. Once this process is complete, click on **Apply Grade**. Once the feedback is released to students, it cannot be modified. +Sample Instructor view of LLM-generated feedback: + +.. image:: /img/guides/rubricapproval.png + :height: 600 + :alt: Instructor view of LLM-generated feedback for approval - .. image:: /img/guides/rubricapproval.png - :height: 600 - :alt: The teacher approves each comment generated by the LLM grader Rubric Requirements Example diff --git a/source/instructors/getstarted/support/changelog.rst b/source/instructors/getstarted/support/changelog.rst index 3970c9ce..6fb518c7 100644 --- a/source/instructors/getstarted/support/changelog.rst +++ b/source/instructors/getstarted/support/changelog.rst @@ -18,6 +18,7 @@ We do not report small bug fixes or issues that affect individual users, who we - `[I] [24th]`: Pyret support has been deprecated. - `[I] [22nd]`: Pagination is now persistent across pages. - `[I] [20th]`: Added support for the Math markdown code block. +- `[F] [17th]`: Add Ability to :ref:`Auto-Grade using an LLM-based rubric grader ` - `[I] [14th]`: Behavior insight information is available in the assignment CSV file. :ref:`Assignment data ` - `[I] [9th]`: New Guides button for :ref:`uploading files`.