Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified source/img/guides/freetext-grading.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified source/img/guides/freetext_navigate.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified source/img/guides/freetextanswer.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified source/img/guides/freetexticon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified source/img/guides/notpartial.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified source/img/guides/partial.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ Complete each section to set up your advanced code test. For more information on
- **Rationale** - Enter guidance for the assessment. This is always visible to the teacher when the project is opened in the course or when opening the student's project.
- **Use maximum score** - Toggle to enable assessment final score to be the highest score attained of all runs.

5. **(Optional)** Complete **Metadata**.
4. **(Optional)** Complete **Metadata**.

6. **(Optional)** Complete **Files**.
5. **(Optional)** Complete **Files**.

7. Click **Create** to complete the process.
6. Click **Create** to complete the process.


See a Working Example
Expand Down
97 changes: 60 additions & 37 deletions source/instructors/authoring/assessments/auto-grade-scripts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,19 @@

Assignment Level Scripts
========================

You can use assignment level scripts to evaluate student code, normalize points, and mark for participation grading. Assignment level scripts are added in the **Script Grading** field on the :ref:`Script Grading <grade-weights>` settings page. These scripts can then transfer the grading value into the grading field. Assignment level scripts are run when an assignment is **Marked as Complete**.

.. Note:: The script must execute within 3 minutes or a timeout error occurs. There is a maximum size for the feedback that can be returned of 1Mb. If this limit is exceeded, the message **Payload content length greater than maximum allowed: 1048576** will be returned.

If you are using an LMS platform with Codio, be sure to enter a percentage value in the **Grade Weight** field to maintain compatibility with LMS gradebooks. This value is then transferred into your LMS gradebook once you :ref:`release the grades <release-grades>`.

Secure scripts
Secure Scripts
--------------
If you store grading scripts in the **.guides/secure** folder, they run securely and students cannot see the script or the files in the folder. Only instructors can access this folder.
You can find more information about assessment security :ref:`here <assessment-security>`.
You can find more information about assessment security in the :ref:`Assessment Security <assessment-security>` page.

Access assessment results
Access Assessment Results
-------------------------
You can access student scores for the auto-graded assessments in an assignment. All this information is in JSON format and can be accessed in the ``CODIO_AUTOGRADE_ENV`` environment variable. The following tabs show the format of this data. The first tab shows just the assessment portion of the data and the second depicts all the available values.

Expand Down Expand Up @@ -258,42 +259,45 @@ The student grade is calculated based on whether they answered the question, not
- In the course assignment settings :ref:`Grade Weights <grade-weights>` section, enable **Script Grading** set **Set custom script path** to that file and disable **Assessments Grading**.


Regrade an individual student's assignment
------------------------------------------
If students have clicked **Mark as Complete** and the custom script is triggered, you can regrade their work by resetting the `complete` switch, and then set it to *complete* again, which triggers the custom script to run again.
Regrade an Individual Student's Assignment
-------------------------------------------
If students have clicked **Mark as Complete** and the custom script has been triggered, you can regrade their work by clicking the three vertical dots next to the student's name to access additional actions. Select **Mark as Incomplete**, then click the three vertical dots again and select **Mark as Complete** to retrigger the custom script.

Regrade all student's assignments
Regrade All Students' Assignments
---------------------------------
You can regrade all student's assignments that have already been auto-graded from the **Actions** button on the assignment page.
You can regrade all students' assignments that have already been auto-graded by clicking the **Regrade Completed** button at the bottom of the assignment page.

1. Navigate to the assignment and click it.
2. Then click **Regrade Completed** at the bottom of the page. This is useful if you have found a bug in your assignment level grading script. **Regrade Completed** does not run individual code test assessments.

1. Navigate to the assignment and open it.
2. Click the **Actions** button and then click **Regrade Completed**. This is useful if you have found a bug in your assignment level grading script. **Regrade Completed** does not run individual code test assessments.
Test and Debug Your Grading Scripts
------------------------------------

Test and debug your grading scripts
-----------------------------------
.. Note:: Codio provides the ability to test your auto-grading scripts when creating your project, this should be done before publishing your project to a course. Once an assignment has been published to the course, any changes made to files in the student workspace (/home/codio/workspace) are not reflected in the published assignment. Grading scripts should be stored in the **.guides/secure** folder. Files in the .guides and guides/secure folders can be published even if students have already started.
Codio provides the ability to test your auto-grading scripts when creating your project, this should be done before publishing your project to a course. Once an assignment has been published to the course, any changes made to files in the student workspace (/home/codio/workspace) are not reflected in the published assignment. Grading scripts should be stored in the **.guides/secure** folder. Files in the **.guides** and **.guides/secure** folders can be published even if students have already started.


Test your script in the IDE
...........................
Test Your Script in the IDE
----------------------------

You can test your auto-grading script in the Codio IDE from the **Education > Test Autograde Script** on the menu bar. This option allows you to specify the location of your auto-grading script and run it against the current project content. It also allows you simulate scores attained by any auto-graded assessments located in the Codio Guide and select which autograded assessments to test.

.. image:: /img/autograde-test.png
:alt: Autograde Test

Be sure to take the following into account when using this feature:

- When you click **Test Script**:
1. When you click **Test Script**:

- All output to ``stdout`` and ``stderr`` are displayed in the dialog.
- The grade returned by your test script is at the bottom of the output section.

- All output to ``stdout`` and ``stderr`` are displayed in the dialog.
- The grade returned by your test script is at the bottom of the output section.
2. ``stdout`` and ``stderr`` output is not available when running the actual auto-grading script (not in test mode) because it runs invisibly when the assignment is marked as complete. Because of this, you should only generate output for testing and debugging.
3. If you want your script to provide feedback to the student, you should output it to a file that can be accessed by the student when opening the project at a later date. In this case, you should allow read-only access to the project from the assignment settings after being marked as complete.

- ``stdout`` and ``stderr`` output is not available when running the actual auto-grading script (not in test mode) because it runs invisibly when the assignment is marked as complete. Because of this, you should only generate output for testing and debugging.
- If you want your script to provide feedback to the student, you should output it to a file that can be accessed by the student when opening the project at a later date. In this case, you should allow read-only access to the project from the assignment settings after being marked as complete.
Test Your Script Using Bootstrap Launcher
------------------------------------------

Test your script using bootstrap launcher
.........................................
You can also use a simple bootstrap launcher that loads and executes the script from a remote location so that you can edit and debug independently of the Codio box. The following example bash script shows a Python script that is located as a Gist on GitHub. This script might be called **.guides/secure/launcher.sh**.
You can also use a simple bootstrap launcher that loads and executes the script from a remote location so that you can edit and debug independently of the Codio box. The following example is a bash launcher script that downloads and runs a Python script from a GitHub Gist. This script would be saved as **.guides/secure/launcher.sh**.

.. code:: bash

Expand All @@ -312,7 +316,7 @@ Sending Points to Codio

Codio provides a Python library to facilitate reporting points from your custom scripts. There are four functions in this library: `send_grade`, `send_grade_v2`, `send_partial` and `send_partial_v2`.

.. Note:: Partial points are not used in assignment level scripts, see :ref:`Allow Partial Points <partial-points>` for more information about setting up partial points.
.. Note:: Partial points are not used in assignment level scripts. See :ref:`Allow Partial Points <partial-points>` for more information about setting up partial points.

In order to use this library you need to add the following code to the top of your grading script:

Expand All @@ -336,23 +340,31 @@ The calls to use these functions are as follows:

send_grade(grade)

`grade` - Should be the percent correct for the assessment.
or:

.. code:: python

send_grade_v2(grade, feedback, format=FORMAT_V2_TXT, extra_credit=None)

`grade` - Should be the percent correct for the assessment.

`feedback` - The buffer containing the feedback for your student - maximum size is 1 Mb.

`format` - The format can be Markdown, HTML or text and the default is text.
.. list-table::
:widths: 20 80
:header-rows: 1

`extra_credit` - Extra points beyond the value for doing this correctly. These do not get passed to an LMS system automatically, just the percentage correct.
* - Field
- Description
* - ``grade``
- Should be the percent correct for the assessment.
* - ``feedback``
- The buffer containing the feedback for your student - maximum size is 1 Mb.
* - ``format``
- The format can be Markdown, HTML or text and the default is text.
* - ``extra_credit``
- Extra points beyond the value for doing this correctly. These do not get passed to an LMS system automatically, just the percentage correct.

.. _autograde-enhance:

Auto-grading enhancements
Auto-Grading Enhancements
-------------------------

The V2 versions of the grading functions allow you to:
Expand All @@ -362,14 +374,25 @@ The V2 versions of the grading functions allow you to:
- Notify (instructors and students) and reopen assignments for a student on grade script failure.


If you don't use the send_grade_v2 functions, this URL (passed as an environment variable) can be used:```CODIO_AUTOGRADE_V2_URL```
If you don't use the send_grade_v2 functions, this URL (passed as an environment variable) can be used:``CODIO_AUTOGRADE_V2_URL``

These variables allow POST and GET requests with the following parameters:

- **Grade** (```CODIO_AUTOGRADE_V2_URL```) - return 0-100 percent. This is the percent correct out of total possible points.
- **Feedback** - text
- **Format** - html, md, txt - txt is default
- **Penalty** - Penalty is number between 0-100,
.. list-table::
:widths: 20 80
:header-rows: 1

* - Field
- Description
* - **Grade** (``CODIO_AUTOGRADE_V2_URL``)
- Return 0-100 percent. This is the percent correct out of total possible points.
* - **Feedback**
- Text
* - **Format**
- html, md, txt - txt is default
* - **Penalty**
- Penalty is number between 0-100


If you want to calculate penalties in the grading script you can use the **completedDate** (in UTC format) in ``CODIO_AUTOGRADE_ENV`` to calculate penalties. See Python example below.

Expand Down Expand Up @@ -430,7 +453,7 @@ These Python and Bash files that can be loaded by a bootstrap script or as expla

main()

Example grading scripts
Example Grading Scripts
-----------------------
This section provides example assignment level scripts using the older methods to send grades.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@ Edit Assessment Points
======================
To edit assessment points, follow these steps:

1. In the Guide Editor, click the **Assessments** button to view the list of all assessments.
1. In the Guide Editor, click the **Assessments** button.

2. Click the assessment to open it and modify the points.
2. Then click **"View Existing Assessments"** in the bottom right corner to view the list of all assessments.

3. Modify the point value in the box to the left of the assessment you want to update. Once done, click the **Close** button on the bottom left.

.. image:: /img/assessmentpoints.png
:alt: Edit Assessment Points
8 changes: 6 additions & 2 deletions source/instructors/authoring/assessments/edit-assessment.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@ Edit an Assessment
To edit an assessment, either:


1. In the Guide Editor, click the **Edit: <Assessment Name>** button to the right of the assessment.
- In the Guide Editor, click the **Edit: <Assessment Name>** button to the right of the assessment.

.. image:: /img/guides/editassessmentbutton.png
:alt: Edit Assessment

2. Click the **Assessment** button to view the list of all assessments and click the assessment to open it and make your changes.
Or, in the Guide Editor, use the following steps:

1. Click the **Assessment** button.
2. Click **"View Existing Assessments"** in the bottom right corner.
3. Click the assessment to open it and make your changes.

.. image:: /img/guides/editassessmentlist.png
:alt: Edit Assessment List
Loading