Critical AI Writing Assessment Workbench — Shibani et al. CHI '24 Framework

1. Turn-by-Turn AI Dialogue Classifier

Classify each prompt/response against the 5 CHI '24 dimensions. Toggle depth to observe metric shifts.

Course Context:

Student ID:

CHI '24 Assessment Design guidelines to verify human agency, critical scrutiny, and hallucination containment.

Q1. What did you ask AI to do, and why?

Q2. How did you evaluate its response?

Q3. What did you verify using other sources?

Q4. Which suggestions did you accept, revise, or reject?

Q5. What intellectual contribution remains yours?

Empirical baseline of 49 graduate student assignments

Total Turns

Deep Turns

Shallow Turns

Planning & Generating Ideas Shallow: 100% (Cohort: 84%)

Finding & Evaluating Info Shallow: 100% (Cohort: 90%)

Writing & Revising Shallow: 100% (Cohort: 100%, 0% Deep)

Dialogue & Conversational Probing Deep: 100% (Cohort: 8% Deep, 91% Shallow)

Reflecting on AI Learning Deep: 100% (Cohort: Deepest Area)

Student Transcript Shibani '24 Baseline

Assessment Diagnostic Verdict

Critical Depth in Reflection & Dialogue; Shallow Planning and Revision typical of baseline cohort

Explicitly Documented with Independent Cross-Verification