CORE Grant

Computerized Oral Reading Evaluation (CORE)

PI: Joseph F. T. Nese
Co-PI: Akihito Kamata (Southern Methodist University)

This research project is supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A140203 to the University of Oregon.

Purpose:  The purpose of this project is to develop and validate a new computerized assessments system of oral reading fluency (ORF), called Computerized Oral Reading Evaluation (CORE). CORE will contain an automated scoring algorithm based on a speech recognition engine and a latent variable psychometric model to overcome both practical and psychometric inadequacies of traditional ORF assessments for screening and progress monitoring, which have direct implications for instructional decision-making. CORE will produce scaled scores that are appropriate for use with students in Grades 2-4 and technically adequate for improving reading outcomes for typically developing students across reading proficiency levels.

Setting:  CORE research is conducted in Grade 2-4 classrooms distributed across school districts in several states. A representative sample of over 7,000 public school elementary students in Grades 2-4 will be recruited across the four years of the study. Students will encompass a wide range of reading proficiency levels, and socio-economic, ethnic, and language backgrounds.

Process:  A three-phase process will structure the development and validation of CORE. The first phase is the development of the CORE test administration and data collection platform. This includes the incorporation of the CORE reading passages with automated scoring, achieved by incorporating an existing ORF assessment system that uses a speech recognition engine. The second phase includes the development of the foundation of the scaled-score scoring algorithm. We will develop a psychometric model that simultaneously incorporates response time and response accuracy to realize ORF as a latent factor. The third phase involves collecting additional validity evidence of CORE through a series of planned studies.

Research Design and Methods:  In 2014-2015, we developed (a) the preliminary content and CORE reading passages: 162 passages total; 54 at each of Grades 2-4, with 9 long passages (≈ 85 words), 15 medium passages (≈ 50 words), and 30 short passages (≈ 25 words), and (b) the CORE assessment platform. In 2015-2016, the remaining reading passages were developed (168 passages total; 56 at each of Grades 2-4, with 11 passages, 15 medium, and 30 short passages), and we collected content evidence of validity. We compared wcpm scores from CORE and a traditional ORF measure (easyCBM) to test the mean differences between: CORE and easyCBM; the three passage length versions of CORE; hand scoring in real time, hand scoring of reading recordings, and CORE. Finally we have developed and tested our psychometric model.

We have completed the administration of the 2016-2017 calibration/equating/linking study. We recruited 11 schools, with 86 teachers, and potentially, approximately 900 students at Grade 2, 810 students at Grade 3, and 870 students at Grade 4. In 2017, we will conduct calibration/equating/linking analyses, the passage parameters will be horizontally equated and vertically linked, and the scoring algorithm for will be developed for the psychometric model.

In 2014-2018, progress monitoring CORE data will be collected to obtain further validity evidence. We will: (a) compare the mean ORF performance between subpopulations to detect potentially biased passages; (b) use latent growth curve modeling for CORE and traditional ORF results to compare the reliability estimates of the growth slopes and compare the standard error (SE) of the point and slope estimates; and (c) solicit feedback from teachers about the utility and interpretability of the system and score reporting to ensure that CORE has direct applicability to real-world school settings and evidence of consequential validity.

Key Outcomes: This project will produce (1) a fully developed and validated assessment of computerized ORF to provide teachers with real-time assessment data to inform subsequent instruction for all students in Grades 2-4; and (2) a psychometric model that incorporates students’ response time and response accuracy to produce scaled ORF scores, with horizontally equated and vertically linked passages.