Meeting of the Board of Regents | November 2010

THE STATE EDUCATION DEPARTMENT / THE UNIVERSITY OF THE STATE OF NEW YORK / ALBANY, NY 12234

TO:	P-12 Education Committee
FROM:	John B. King, Jr.
SUBJECT:	Revised Comprehensive English Regents Exam: Standards Revisitation Update
DATE:	November 8, 2010
STRATEGIC GOAL:	Goals 1 & 2
AUTHORIZATION(S):

Summary

Issue for Discussion

Implementation of the new Comprehensive English Regents Exam in January 2011.

Reason(s) for Consideration

For Discussion.

Procedural History

This item will come before the P-12 Education Committee for discussion at the November 2010 meeting.

Background Information

In June 1999, the Department implemented a new design for the Comprehensive English Regents Exam; it was administered in two separate three-hour sessions over two days. Students had to complete both sections to receive a score, and it was not possible to “score bank” grades, i.e. take one section in a June administration and another section at another time, e.g. August/January. The format of this test created significant hardship for both students and administrators around the State. On three separate occasions, adverse weather conditions disrupted the administration of the assessment, and students who started the test were unable to complete it; they were forced to retest during the next testing opportunity. Superintendents, including the New York City Department of Education, repeatedly requested that the State redesign the exam so that it could be administered in one three-hour session. Department staff recommended to the EMSC Committee that we honor the request, and the Board directed staff to redesign the test and maintain a parallel level of rigor despite the loss of three hours of testing time. In November 2009, the Department released the test specifications for the new assessment and informed the field that the new exam would be administered in January 2011.

           

Scheduling the exam poses challenges to both the Department and the field. The January Testing Period is four days and inclement weather is always possible. School Districts expect the Department to post scores by noon on January 28 to ensure that: 1) second semester scheduling can occur; 2) course grades and promotion decisions can be determined; and 3) graduating seniors have the opportunity to enroll in postsecondary education institutions. There is no one optimal scheduling solution. In order to return scores by January 28, the exam will have to be administered prior to the regular January Testing Period.

The January Testing Period starts on the morning of Tuesday, January 25 and concludes Friday, January 28. Sixteen exams will be administered during this time. Some school districts have objected to losing an extra day of instruction since the Department will have to administer the new exam prior to the Regents Testing Period. For example, although many high schools remain open during the January Testing Period, NYCDOE closes their high schools to administer the tests. An alternative approach would be for the Department to administer the remaining exams from Wednesday-Friday. This condensed schedule poses both an advantage and challenges.

Advantage

 

Challenges

Feedback from key stakeholders suggests a global preference for maintaining the traditional four-day schedule.

After the administration of the exam, the Department will need to conduct a Standards Revisitation Process. In July 2010, the Department reset cut scores on the Grades 3-8 Testing Program. This process included analyzing the relationship between Grade 8 ELA Exam performance and subsequent performance on the Comprehensive English Regents Exam. This was done to help strengthen the predictive validity of the Grade 8 assessment and to inform administrators, teachers, and parents as to whether or not students are on track for college and/or career readiness.  The Department is also working to study the college readiness scale score of 75 with the College Board. The Standards Revisitation Process plan is included with this Item as Appendix A.

The Department is committed to providing a seamless transition to the Common Core Standards. The Department has issued an RFP for an updated Grades 3-8 Testing Program that will include exams for English Language Arts in grades 9-11. In addition, the Department is participating in the PARCC consortium which is designing next generation assessments with funding from a USED Race to the Top Assessment Grant. The PARCC assessments are scheduled to be available for administration in 2014-15. The PARCC assessments will also include grades 9-11.

Next Steps

The Department will administer the Comprehensive English Regents exam on the morning of Tuesday, January 11, 2011; schools will be required to score the test that afternoon and ship answer sheets to our vendor for preparation for the Standards Revisitation Process by Wednesday, January 12, 2011.  No changes will be made to the length of the regular January Testing Period; we recognize that this is a unique situation and that this will require a one-time only solution.

Appendix A

Comprehensive English Regents Exam


Performance Standards Revisitation

Standards Revisitation Plan

In response to a request from Superintendents, including the New York City Department of Education, the Board of Regents directed staff to develop a new three-hour, one-session test format for the Comprehensive English Regents Exam. This new exam will be based on the 2005 high school level English Language Arts Core Curriculum, Performance Indicators, and the Key Ideas of listening, reading, and writing.    

The new exam retains the following components from the current exam:

• Learning Standards, Key Ideas, &  Performance Indicators
• One listening passage
• Critical Lens Essay Task
• Paired literature passages
• Inclusion of non-fiction reading materials

 

	Old Exam	New Exam
Testing Time	Two days, 6 hours	One day, 3 hours
Number of multiple-choice items	26	25
Number of essay questions	4	1
Number of constructed response items	0	2

Subject area experts determined that the content being assessed remained constant, despite changes to the formatting and composition of item types. The standards and requirements did not change substantially for the two tests; therefore, a full standard setting is not required. However, changes to the test design require a revisitation of the performance standards to ensure that the State maintains consistent performance standards across time.

Assumptions

The performance Standards Revisitation Process is based on four assumptions:

1. Maintaining rigor — the expected rigor of the new cut scores on the exam should be comparable with the current cut scores applied to the Comprehensive English Regents Exam.
2. Alignment with the performance standards from Grades 3-8 — It is important to continue to maintain the alignment between the adjusted standards for ELA Grades 3-8 and the new Exam.
3. Readiness for the next educational level — the performance standards provide information about students’ readiness for their next educational level.
4. College readiness indicator—colleges have indicated that a scale score of 75 denotes readiness to enroll in credit bearing English classes.

Goals

There are three goals for the Standards Revisitation Process: 1) revisit the standards and requirements due to the new test format; 2) describe the framework for collecting evidence to support the interpretation of the cut scores; and 3) continue the process for systematically collecting evidence to evaluate the use of a scale score of 75 as an indicator of college readiness.

Methodology

There are three stages for the Standards Revisitation Process: 1) pre-policy measurement review; 2) performance standards revisitation; and 3) post-policy measurement review.

Pre-policy Measurement Review

The pre-policy measurement review is designed to obtain recommendations for the expected percentage of New York students who would be classified in each achievement level: 0-64, 65-84, and 85-100. A committee, made up of members of advisory groups to the Department, will be asked to consider five guiding questions:

• Given the differences in the new format of the assessment, what type of differences in the percentage of students at each level does the panel expect?
• What percentage of students in each achievement level would the panel find acceptable on the newly formatted exam?
• What consistency is expected between the data from the current and new testing programs, given the consistency in content and standards requirements shown between the two exams?
• What type of consistency does the panel expect between ELA Grade 8 and the Comprehensive English Regents Exam?
• What consistency is expected between national data and New York results for this newly-formatted exam compared with the previously-formatted exam?

Following review of these guiding questions, panelists will review trend data from multiple assessment programs:

The results of a study by Dr. Howard Everson examining the relationship between Comprehensive English Regents Exam scores and the probability of a student earning a course grade of C or above in Freshman English courses will be reviewed. This will help panelists evaluate the relationship between the Comprehensive English Regents Exam and college readiness.  The panelists will also be asked to provide their recommendations on what subsequent data collection and analysis can be conducted to provide ongoing validity evidence supporting the interpretations of the performance standards and the use of a scale score of 75 to denote college readiness. 

Standards Revisitation Process: Content Review

Two Content Review committees will meet in Albany to revisit the performance standards. They will study the changes in format to the test and evaluate whether the cut scores mapped to the newly-formatted exam continue to support the interpretations about what abilities students have demonstrated at those achievement levels. Given the assumption that the performance standards will provide information about student readiness for their next educational experience, college professionals will participate in the process. The content experts will focus on consistency in content requirements for the cut scores across the current and newly-formatted tests.

It will be possible to identify where the original cut scores of 65 and 85 are located on the new exam. Panelists will use Achievement Level Descriptors (ALDs) to focus on the items around the two cut scores. They will evaluate the knowledge and skills these items are measuring, how they relate to the standards and ALDs, and decide whether to keep the original cuts mapped onto the newly-formatted test or to move them. If they recommend moving the cut points, they will indicate which direction and how far. The exercise will be conducted in two rounds. The panelists will also be asked to provide their recommendations on what subsequent data collection and analysis can be conducted to provide ongoing validity evidence supporting the interpretations of the performance standards. As a separate step, the scale score of 75 will also be identified in the Ordered Item Booklet. Panelists will evaluate the skills and knowledge of the items around this cut score and discuss why the content surrounding this scale score supports the observation from Higher Education that students have demonstrated the knowledge and skills necessary for college readiness.

Post-policy measurement review

The Post-policy measurement review panel will meet after the completion of the standards revisitation meetings with the content experts. The same committee who participated in the pre-policy measurement review will participate in this activity.  The purpose of the review panel is to integrate results from the pre-policy measurement review and the two content committees (including the synthesis meeting). Results from pre-policy measurement review and content expert recommendations on the standards revisitation will be presented and discussed. Following the review of the methods and results, panelists will be asked to independently integrate results from both meetings. They will then discuss their rationale for how they integrated the results.  Discussions regarding the use of the scale score of 75 as an indicator of college readiness will also occur. Further ideas for collecting validation evidence supporting interpretations from the three cut scores (65, 75, and 85) will be discussed and summarized in the technical report of the standards revisitation.