Configure Assessors and Artifacts

Once a Juried Assessment is scheduled and the assignment linking end date has been met, the number of assessors required to score each artifact and the distribution of the sample size across assessors can be configured until the scoring start date is met. The platform supports the following user roles as Juried Assessment assessors:

Juried Assessments do not utilize the organizational hierarchy; therefore, the college or department to which assessors are aligned will not affect the access granted to an assessor.

Configure Assessor and Artifact Scoring

1. To configure assessors and artifact scoring, navigate to the Assessors and Artifacts Manager and select Settings from the expanded kebab menu.

   

   1. Utilize the sample size slider bar (1) to configure the sample size of artifacts to be assessed. By clicking % or # (2), the sample size can be configured as a percentage of the total artifacts available for scoring or as a numerical value based on the total artifacts available for scoring. The total count of assessors required to score each artifact can also be configured (3).

      1. In the example below, 36 artifacts are available for assessors to score. Based on the configuration of the sample size slider bar, 51% of the artifacts will be assessed, which results in 18 out of the 36 total artifacts being reviewed and scored. The artifact scoring sample size of 18 is multiplied by the count of assessors, being 3, to determine that the total amount of artifact scoring opportunities is equal to 54.

         

   2. After clicking Next, users can be added as assessors; the total number of assessors must equal the total count of assessors selected to score each artifact in the previous step. For example, if three was selected as the total count of assessors, then three assessors must be added. Click +Add to search for and select one or more assessors via the Add Assessors popup. Multiple assessors can be added, and assessors can be searched for by first/last name, email, username, or university ID.

      

      1. After clicking Add Assessors, the selected users will be displayed, and the distribution of artifacts can be configured for each assessor. The type of distribution can be defined as a percentage, numerical, or a number of hours (1); once the distribution type has been defined, the artifact distribution can be manually entered for each assessor (2); for percent and numerical distribution types, the Scale option can be clicked to automatically and equally distribute artifacts among assessors. Before continuing, the total distribution must equal 100%.

         1. In the example below, three assessors have been assigned to review and score artifacts. The distribution type was defined as a percentage by selecting the percentage option, and the artifacts were distributed equally in increments of 33.33% by selecting the Scale option. This configuration results in each assessor being responsible for the scoring of 33% of the artifacts.

            

2. After clicking Next, a review of the configured assessors and artifact distribution can be performed; if changes need to be made, Back can be clicked, or click Next to proceed.

   

3. Finally, all selections can be reviewed, and Save Settings can be clicked to complete the configuration of assessors and artifacts.

   1. Once settings are configured and saved, the artifact samples that have been randomly selected, based on sample size, should be reviewed to ensure all artifacts are valid for assessor scoring. Learn more about managing artifacts.

      

Replacing An Assessor

Replacing an assessor may be necessary due to scheduling conflicts, changes in assigned responsibilities, or other unforeseen circumstances. Assessors can be replaced while Juried Assessments are in Pending Scores status. When replacing an assessor, scoring from the previous assessor can be either retained or erased. When an assessor is replaced, the platform will automatically regenerate any associated action items and assign them to the new assessor.

1. To replace an assessor, navigate to the Assessors and Artifacts Manager and select Assessors from the expanded kebab menu.

   

2. From the expanded assessor kebab menu, select Replace Assessor.

   

   1. Search for and select the new assessor and decide how to handle the previous assessor’s progress (1); clicking the magnifying glass will open the advanced search with filter options. Only one user may be selected as the new assessor. After clicking Replace Assessor (2) and confirming the replacement, the assessor will be replaced, and the assessor's progress will be retained or removed, as configured.

      • Retain Previous Work: Selecting this option will transfer all work (artifacts scored, un-scored, and in progress) to the new assessor. Saved work will be intact for the new assessor to either keep or modify.

      • Start Fresh: Selecting this option will transfer all work to the new assessor and erase any previously saved work and comments from the original assessor. This means the new assessor will start scoring from scratch.

        

Adding External Artifacts

Learning Management System (LMS) assignments that have been linked to outcomes are automatically imported from the LMS during the assignment linking period, when the Juried Assessment is in Pending status. Additionally, external artifacts can be uploaded for an outcome through the Assessors and Artifacts page. When adding external artifacts, the file must be a .zip file and should be less than 1 GB in size per file upload, and users can upload multiple files one at a time. Artifacts can be added when a Juried Assessment is in Pending or In Progress statuses.

Juried assessments utilizing only externally uploaded artifacts will skip the Pending status and proceed directly to the In Progress status.

1. To add external artifacts for assessors to review, navigate to the Juried Assessment Homepage and select the Assessors and Artifacts page from the left-hand menu. By expanding the kebab menu, select Add Artifacts.

   

   1. Zip files can be added by clicking the Upload File section to open a file explorer or by dragging and dropping the file to the File Attachments section.

      

2. Once uploaded, the Upload Artifacts drawer will automatically close, and the page will refresh to display a success message. The user will then receive an email once processing is complete.

   

Managing Artifacts

Artifacts should be reviewed to ensure they are valid for scoring by the assessor. Artifacts that aren’t valid for scoring can be removed and replaced with a new randomly selected artifact, given that additional artifacts are available in the sample pool. The total count of discarded artifacts will be displayed via the Discard Artifacts column of the Assessors and Artifacts Manager once the Juried Assessment transitions into the status of Pending Scores.

1. To review artifacts, navigate to the Juried Assessment Homepage and select the Assessors and Artifacts page from the left-hand menu. By expanding the kebab menu, select Review Artifacts.

   

   1. Artifacts can be reviewed (1), and given that additional artifacts are available in the sample pool, artifacts that aren’t valid for scoring can be discarded and replaced (2) with a new randomly selected artifact. Discarding an artifact submission will add it to a list of non-usable artifact submissions for the assessment.

      

      1. View options and search functionality can be utilized for each artifact (1); the view functions can be used to open the thumbnail panel for an artifact, zoom, rotate, etc.; additionally, if an artifact contains multiple pages, page navigation can be used (2) to review all artifact contents.

         

      2. By clicking the artifact information option, a pop-up will display the assignment name, associated course section, and term information. Artifacts can be downloaded, and the last icon displays the count of pages within the sample.

         

   2. When artifact review is complete, click Exit to return to the Assessors & Artifacts page.