Meeting of the Board of Regents | May 2008

THE STATE EDUCATION DEPARTMENT / THE UNIVERSITY OF THE STATE OF NEW YORK / ALBANY, NY 12234

 

TO:
FROM:	Johanna Duncan-Poitier
SUBJECT:	Grade 3-8 Testing Administration and Scoring Timeline
DATE:	May 1, 2008
STRATEGIC GOAL:	Goals 1 & 2
AUTHORIZATION(S):

 

SUMMARY

 

Issue for Discussion

 

Should the Board of Regents consider short-term and long term strategies to improve the administration, scanning and scoring of the Grades 3-8 Math and English Language Arts examinations so that school districts receive scores in a timelier manner?

 

Reason(s) for Consideration

 

              For Information

Procedural History

 

During the March 2008 meeting of the Regents Committee on Policy Integration and Innovation, information about the administration, scanning, scoring and score returns for the Grades 3-8 Mathematics and English Language Arts examinations was presented to the Board of Regents.  The Board requested that staff continue to examine possible short-term and long-term solutions to the issues.

 

Background Information and Recommendations

 

Purpose of the New York State Grades 3-8 Testing Program

 

The New York State Testing Program (NYSTP) is designed to evaluate the implementation of the State Learning Standards at the student, school, district, and statewide levels.  New York State is required to administer tests in English Language Arts (ELA) and Mathematics under the No Child Left Behind Act (NCLB).  These tests must address the depth and breadth of the State’s Learning Standards; be valid, reliable, and of high technical quality; and be designed to provide a coherent system across grades and subjects.  The Grades 3-8 Testing Program in Mathematics and English Language Arts is administered to approximately 300,000 students per grade in both public and nonpublic schools.  With the exception of the small population of severely disabled students who qualify to take the New York State Alternate Assessment (NYSAA), all students, general education, students with disabilities, and Limited English Proficient/English Language Learners (LEP/ELL) in all public school districts are required to take these tests.  These tests present the opportunity to annually evaluate the implementation of the State’s learning standards; measure individual student and cohort progress; and gather data on student readiness for study at the next level.

 

The ELA tests target student progress toward three of the four content standards (we do not test speaking).  The mathematics tests target student progress toward the five content bands.  The established cut scores classify student proficiency into one of four levels based on their test performance:

 

Level 1:  Not Meeting Learning Standards

              Student performance does not demonstrate an understanding of the ELA/mathematics knowledge and skills expected at this grade level.

 

              Level 2:  Partially Meeting Learning Standards

              Student performance demonstrates a partial understanding of the ELA/mathematics knowledge and skills expected at this grade level.

 

              Level 3:  Meeting Learning Standards

              Student performance demonstrates an understanding of the ELA/mathematics knowledge and skills expected at this grade level.

 

              Level 4:  Meeting Learning Standards with Distinction

              Student performance demonstrates a thorough understanding of the ELA/mathematics knowledge and skills expected at this grade level.

 

All students who score below the State-designated performance Level 3 must be provided academic intervention services.

 

Test Structure and Psychometric Validation

 

              New York State has developed its testing program to reflect the depth and breadth of the New York State Learning Standards.  All assessments contain both multiple-choice and constructed-response items.  Multiple-choice items are scored by computer after they are scanned; the constructed-response items are scored locally by teachers.

 

              The Grades 3-8 Testing Program has been externally peer reviewed and fully approved by the United States Education Department.  The Department has adhered to the Joint Standards for Educational and Psychological Testing to ensure that the tests are valid and reliable to ensure accurate classification decisions for both student support (Academic Intervention Services) and accountability determinations.

Test Score Return:  Timelines from Other States

 

        •      
Critical Test Operations Question:  Is the timeline currently in place for the New York State Testing Program inconsistent with timelines used by other large states?

 

                Department staff have reviewed timelines from other large states to see if test turnaround times are significantly shorter or similar to New York State (see appendix A).  There are a number of variables that impact the return of test scores to schools, including, but not limited to: administration/scheduling; test structure (multiple choice and/or constructed-response); use of single or multiple test forms; scoring and scanning models; and psychometric validation processes.

 

                An analysis of the large seven states, California, Florida, Illinois, New York, Ohio, Pennsylvania, and Texas show New York’s timelines are not unusual:

 

        •       California allows all school districts a flexible window to administer the exams and the Department returns test scores to schools in August, long after the school year is completed.

        •       Florida administers exams in March and returns test scores in June, though Department staff have indicated that for the 2008 tests, scores will be returned during the summer break.

        •       Illinois and Pennsylvania administer tests in March and return scores in August.

        •       Ohio and Texas administer tests in late April and early May and have rapid turnaround times – Ohio in June and Texas within a month of test administration. However, unlike New York State, which must administer a new test form every year, states that reuse test forms (e.g., Texas) do not have as many additional psychometric processes to complete after each administration. 

 

Test Administration/Scheduling

 

        •       Critical Policy Question: Should the Department revise the current testing calendar?

 

                Tests are administered in January and March in a staggered format, Grades 3-5 and then Grades 6-8.  There is also a make-up period associated with the testing program to ensure that the school systems have enough time to test students who were absent, students with disabilities, and LEP/ELL students.  The current testing schedule was supported and influenced by multiple-stakeholders (e.g., New York State Council of School Superintendents, District Superintendents, The Big 5 School Districts, New York State United Teachers, New York State School Boards Association, and School Administrators Association of New York State).

 

        Recommendation:

 

                The testing calendar for the 2008-2009 school year has already been published and schools have set their testing and scoring schedules.  At this late date; we do not advocate changing the 2008-2009 testing calendar.  For the 2009-10 school year, we can signal to the field now that a change will be coming so school districts and BOCES will have enough time to design their calendars.  To increase test turnaround time, we can condense the testing schedule into approximately seven days and test all grades concurrently.  This will still allow schools to administer make-up tests and to make sure that students who require testing accommodations are appropriately assessed.

 

                Members of the Board have asked if we can administer all the exams during the same time of the year.  This is possible.  To do so we would need to select an appropriate time of year and possibly move one or both sets of tests.  This is a complicated process that we do not recommend at this juncture.  The mathematics exams are carefully designed to test material from March to March; the field has made their curriculum adjustments based upon this.  If we were to move these exams, we will be required to restandard—set the exams and break the current trend line for data analysis; this would also create additional costs and complicated work for school districts to adjust to these changes.  This would also complicate the data projections for the design and implementation of the growth model.  Rescheduling the tests would also require coordination between the vendor and the Department to complete a major revision to all aspects of test development, manufacturing, storage, and shipping processes.  For the current contract, we do not recommend moving the placement of the exams in the academic calendar, but we would recommend to the Regents, after consultations with impacted stakeholders, moving both sets of tests to an agreed upon time of year with the advent of the new Grades 3-8 Testing Program in 2 years.

 

Test Scanning

 

        •       Critical Policy Question: Should the Department change the process for scanning the Grades 3-8 Testing Program?  What is a possible short-term, cost effective solution?

 

                Approximately 25 days for math and 35 days for ELA are currently used to scan exams through the Regional Information Centers and scanning centers in Yonkers and New York City.  It is possible to reduce this time frame.

 

        Recommendation:

 

                For the 2008-2009 school year, the Regional Information Centers and New York City have submitted a plan to the Department to scan the Grades 3-8 Testing Program in approximately 10 days for each exam (a reduction of 15 days for the math exam and 25 days for the ELA exam) while still ensuring quality control procedures.   This plan requires the Department to work collaboratively with these entities and CTB/McGraw Hill to reengineer key parts to the process to facilitate quicker turnaround time in the scanning process.  Planning is underway to ensure that this process can be implemented for the 2008-2009 school year.  ELA results can be projected to be released in early April and math in mid-May.

 

                If the Department were to condense the testing calendar for the 2009-2010 academic year (as discussed above), we could save an additional 7 days which would enable ELA scores to be released in late March and math scores in early May.  We will continue to refine this restructured testing timeline and report back to the Board on final time savings.  There are no major costs associated with this approach.

 

        •       Critical Policy Question: Should the Department change the scanning and scoring process for scanning the Grades 3-8 Testing Program?  What is a possible long-term solution?

 

                The scoring of the constructed-response items is time intensive, labor intensive, and expensive.  This process is creating a great deal of strain as reported by all school districts.  The majority of school districts must complete this scoring during the contracted work day or incur additional labor costs.  Furthermore, school districts must find substitute teachers to cover classes while teachers are scoring the exams.  We have consulted with the field regarding the scoring timelines and despite their requests for more timely return of tests, they are adamant that we not shorten the scoring timelines due to the volume of work and scheduling difficulties that this process creates.

 

        Recommendation:

 

                To offset the demand of scoring exams at the local level, staff recommend that the Board consider the use of an electronic distributive scoring model.  This model requires that the Department produce or purchase a scannable answer sheet or test booklet so that the open-ended responses can be distributed electronically to be scored by qualified staff at designated scoring sites or directly to independent scorers anywhere in the country.  The SAT testing program uses a variation of this model which is why the results are available so soon after the test administration.  Staff are researching the impact on our manufacturing processes to determine if we can generate a testing booklet or answer sheet that can be scanned by the vendors capable of doing this work; this requires specific paper, ink, coding, and computer programming.

 

                Experts estimate the time that an external vendor will need to scan, hand-score constructed response items, and prepare a file for submission for psychometric research is approximately 28-38 days.  The psychometric scaling and data verification processes require an additional 30 days bringing the total number of work days to approximately 58-68 days from the time tests are administered to the time scores can be returned to schools.

 

                Local school districts would save money by not having to hire substitute teachers in order to free-up the teachers needed to score the exams, however there would be considerable cost to the State to use this scanning and scoring process.  In addition, since many states administer these exams at the same time of year, demand on the vendors to scan and score is great and this could impact time frames.  Estimated State costs run between $25 - $35 million per year, but until the Department puts forward a Request for Proposals (RFP) and receives proposals from vendors, we can only estimate total costs.  The Regents would need to advocate for funding to be included in future State Budgets to support this long-term recommendation.

APPENDIX A

 

LARGE SEVEN STATES*

COMPARISON OF ELEMENTARY AND INTERMEDIATE STATE EXAMINATIONS

 

State	Average Number of Students Tested per Grade	Test Content	Subject	Last Day of Test Administration	Date Schools Get Results
California	481,000	MC	Reading and Math	California Department of Education does not establish testing schedules for school districts.  Schools administer the California Standards Tests within a 21-day window that is comprised of ten days before and 10 days after the day on which the school completes 85 percent of all instruction for the year.  District STAR coordinators set the district window within the 21-day regulatory window for the Standardized Testing and Reporting (STAR) Program.	The state, county, school district, and school summary reports will be posted on the Internet on August 15th each year at http://star.cde.ca.gov/.
Florida	200,000	MC, CR	Reading and Math	March 24	No earlier than June 2-6 Summer 08: No formal projected date
Illinois	150,000	MC, CR MC, Short Answer	Reading and Math	March 3-14	August
New York	300,000	MC, CR	ELA and Math	Jan. 7-11 (Grades 3-5); Jan. 14-18 (Grades 6-8); Mar. 3-7) (Grades 3-5); Mar. 6-12 (Grades 6-8)	May: ELA June: Math
Ohio	132,000	MC, CR	Reading and Math	May 9	June 15
Pennsylvania	130,000	MC, CR	Reading and Math	March 31-April 11	End of August
Texas	305,000	MC	Reading and Math	Reading: April 30 (for most tests) Math: February	Most results available within a month

* Largest states by population