Gains achieved by students who exceeded the previous year: PPS compared to ACCESS

    It is well established that high-achieving students make lower gains than other students.  This has been found in districts across the country and it is also true for Portland Public Schools, where students who exceed benchmarks consistently make lower gains than other students.  See the results for this year and earlier years here.  I have often heard that this was a result of students hitting the "ceiling" on the tests and did not reflect the student's learning or instruction. 

Under a very silly provision of No Child Left Behind, state tests must measure students against grade level expectations, not against higher expectations.  This does put a lid on how much students can demonstrate that they know when they have been working above grade level, but the Oregon OAKS, as a computer-adaptive test, has a somewhat higher "ceiling" than paper and pencil tests.

I was curious to see whether the testing ceiling was in fact making it impossible for high-achieving students to make gains, so I compared the gains made by ACCESS students with those of other students in Portland who "exceeded" benchmarks.  ACCESS admits students in the top 1% of achievement--if there was a ceiling they should be hitting it harder than other students.  They might have higher scores to begin with, but they would not have higher gains. If, on the other hand, the difference is a reflection of real differences in instruction, then ACCESS students might continue to outperform other students in PPS who also exceeded benchmarks.

The results from this year are below.  I have included the mean achievement levels of the same group of students from the previous spring, so it is evident that ACCESS students who exceeded benchmarks are beginning the school year with higher scores than their classmates in other schools who also exceeded benchmarks. If they followed the normal pattern, the higher their starting score, the lower their gains would be.  

Because ACCESS is a small program, there is likely to be greater variability in test performance averages than in the district as a whole.  The scores should be compared over a period of several years before we can be confident about the story they tell. 

The results cast doubt on the common argument that regression to the mean, margins of error and ceiling effects inevitably depress the gains that can be obtained by high-achieving students. However, they don't tell us much more than that about how ACCESS performed.

Out-of-level testing for all students who exceeded would be more informative.    A recent study has found that the span of reading comprehension levels within a single classroom in a magnet school was as high as 10 grade levels and that the average range was 8.3 grade levels.  See Jenine R. Firmender, Sally M. Reis, and Sheela M. Sweeny, "Reading Comprehension and Fluency Levels Ranges Across Diverse Classrooms" Gifted Child Quarterly (Winter, 2013) 57 no. 1: 3-14, Table 3 p. 10.  In a small program with such large variations between students, a composite gain score based on a single test is not very meaningful.  Oregon, however, will only allow testing up ("targeting up") by one grade level for now as the Smarter Balanced tests are being phased in.  This new test will also make it impossible to compile comparable data for several years to come.

 

READING
		PPS	PPS	ACCESS	ACCESS
		exceeds	Exceeds	exceeds	exceeds
	Grade	12-13 mean	13-14 gain	12-13 mean	13-14 gain
	4	231.9	2.4	235.9	4.0
	5	234.8	-0.9	238.1	1.8
	6	236.5	2.3	242.3	-0.8
	7	242.6	3.2	244.9	3.6
	8	247.5	-1.0	249.2	1.2
MATHEMATICS
		PPS	PPS	ACCESS	ACCESS
		exceeds	exceeds	exceeds	exceeds
	Grade	12-13 mean	13-14 gain	12-13 mean	13-14 gain
	4	227.5	5.7	237.6	5.5
	5	234.9	1.9	243.2	1.7
	6	240.8	0.9	245.5	2.1
	7	244.0	8.9	248.0	11.1
	8	250.8	3.3	257.5	3.9