Configuration A: AI Enabled, IRR Enabled
This is the intended full configuration and is suitable for Institutions wanting to utilize AI, IRR agreements, and PII redaction. All features are available. AI Shadow Baseline runs in parallel with human assessors for calibration. All reliability metrics include AI comparison data. PII redaction is active.
-
Available: Rubric scoring, assignment linking, artifact distribution, blind scoring, AI analysis, AI-suggested scores, PII redaction, IRR configuration, IRR agreement policies, Kappa tracking, adjudication, Reliability metrics, and all IRR-based reporting.
Configuration B: AI Enabled, IRR Disabled
The most suitable for single-assessor or low-stakes workflows. AI assistance is fully available during scoring. Assessors can access criterion-level analysis, evidence links, and suggested scores. PII redaction is active. However, there is no IRR agreement tracking, no adjudication, and no reliability reporting.
-
Available: Rubric scoring, assignment linking, artifact distribution, blind scoring, AI analysis, AI-suggested scores, PII redaction, and basic scoring workflow.
-
Not available: IRR configuration, IRR agreement policies, Kappa tracking, adjudication, Reliability metrics, and all IRR-based reporting.
Configuration C: AI Disabled, IRR Enabled
The most common configuration for Institutions with AI concerns. Full IRR capabilities are available, and the reliability-based Calibration & Operations and Student Proficiency dashboards load, but they show only human data (AI comparison columns are hidden). PII redaction is not available.
-
Available: Rubric scoring, assignment linking, artifact distribution, blind scoring, all IRR and adjudication features, Reliability metrics, Kappa tracking, and human-only reporting dashboards.
-
Not available: AI analysis, AI scores, PII redaction, AI Shadow Baseline, AI comparison statistics in reporting.
Configuration D: AI Disabled, IRR Disabled
The assessment runs as a basic juried workflow. Assessors score artifacts independently against the rubric. No IRR agreement tracking, no adjudication, no AI assistance, no PII redaction. This is the basic Juried Assessment functionality.
-
Available: Rubric scoring, assignment linking, artifact distribution, and blind scoring.
-
Not available: AI analysis, PII redaction, IRR, adjudication, IRR agreement policies, and reliability reporting.