Quantifying Deliberate Practice Using Multiple Student Response Metrics on Auto-Graded Homework

Auto-graded homework questions with randomized numbers and content provide frequent, formative assessments for students that create big data for instructors and researchers. Online homework questions combine defined and repetitive practice, feedback on correctness, explanation of solutions, and availability of repeated assessment, which are tenets of deliberate practice. Over 800 questions and 170,000 attempts were studied from three cohorts of students (n=284) using a fully interactive online textbook, Material and Energy Balances zyBook. Question sets consisted of multiple parts where easier levels preceded harder levels. Question types included multiple choice, single numeric, and multiple numerical responses. When students were allowed an unlimited number of attempts before a fixed due date, a median correct of 94% was measured. To investigate student responses beyond correct and incorrect, multiple metrics were aggregated into a deliberate practice score. Deliberate practice scores between 0 and 8 encompassed fraction correct, modified correct, first quartile attempts before correct, and median attempts before correct at different thresholds. The deliberate practice score distinguished between question type and order. Furthermore, students disproportionately practiced questions with higher deliberate practice scores without data beyond their own correctness.