It's news when the nation's largest school district--New York City--announces a teacher incentive plan that would give cash bonuses to teachers at some of the city’s high-needs schools that raise student test scores. Union officials supported the plan because it gives the incentive bonus to the whole school and not just to individual teachers who raise student test scores. NYC's move comes in the midst of much discussion in D.C. about performance pay for teachers. Notably, House education committee Chairman George Miller included performance pay in his committee’s discussion draft for renewal of NCLB.
Those interested in this policy topic are advised to visit the National Center for Performance Incentives , a national research and development center for state and local policy at Vanderbilt University’s Peabody College. Under the leadership of Matthew Springer, this center sets out to answer one of the most contested questions in public education today: Do financial incentives for teachers, administrators, and schools affect the quality of teaching and learning? The Center is hosting a national conference on this topic in late February 2008, which promises to be very, very interesting.
Sunday, October 21, 2007
Monday, August 20, 2007
Improvement in Student Performance Slows Since NCLB
Is NCLB really raising student achievement as its proponents argue? Fuller et. al. report that test-score improvement among 4th graders in 12 states has fallen off in reading and slowed in math since the enactment of NCLB. Unlike a recent report from the Center on Education Policy, which found consistent and significant increases in state-test scores since NCLB became law in early 2002, this research study draws on data from both state assessments and the federally administered NAEP.
A link to the study, published in the July issue of Educational Researcher, can be found below.
A link to the study, published in the July issue of Educational Researcher, can be found below.
Wednesday, August 15, 2007
America Competes?
On August 9, President Bush signed into law H.R. 2272, the "America COMPETES Act" or the "America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Act." This bill authorizes various programs at the National Science Foundation and the Departments of Energy, Commerce, and Education intended to strengthen education and research in the United States related to science, technology, engineering, and mathematics. This bill shares many of the goals of the President's American Competitiveness Initiative (ACI), a strategy designed to maintain America's competitive position in the global marketplace by strengthening its scientific education and research, improving its technological enterprise, attracting the world's best and brightest workers, and providing 21st century job training.
This bill authorizes the President's Math Now proposal, which many have already disparagingly described as 'Reading First does mathematics.' Math Now focuses on the use of research-based instructional practices to prepare students for rigorous math courses in middle and high school, particularly Algebra I and II. Given appropriate allocations, Math Now will be implemented through federal grants that will flow to the states and then to local school districts to improve K-9 instruction.
While this focus on mathematics is certainly needed, I echo the concerns expressed recently by Chester Finn and Diane Ravitch (see link below to Beyond the Basics) that America's economic strength and innovation also spring from students receiving a rich education in other subjects, such as literature, history, and the arts: "America's true competitive edge over the long haul is not its technical prowess but its creativity, its imagination, its inventiveness. And those attributes are best inculcated not by skill-drill but through liberal arts and sciences, liberally defined."
This bill authorizes the President's Math Now proposal, which many have already disparagingly described as 'Reading First does mathematics.' Math Now focuses on the use of research-based instructional practices to prepare students for rigorous math courses in middle and high school, particularly Algebra I and II. Given appropriate allocations, Math Now will be implemented through federal grants that will flow to the states and then to local school districts to improve K-9 instruction.
While this focus on mathematics is certainly needed, I echo the concerns expressed recently by Chester Finn and Diane Ravitch (see link below to Beyond the Basics) that America's economic strength and innovation also spring from students receiving a rich education in other subjects, such as literature, history, and the arts: "America's true competitive edge over the long haul is not its technical prowess but its creativity, its imagination, its inventiveness. And those attributes are best inculcated not by skill-drill but through liberal arts and sciences, liberally defined."
Wednesday, July 25, 2007
Chairman Miller's NCLB Memorandum Link
Education Week's blogger extraordinaire, Alexander Russo, posted House education committee chairman George Miller's memorandum to House freshmen from earlier this month, which outlines Miller's stance on NCLB. The two-page memo dated July 7 outlines nine key proposals for revising the controversial law. Here's a link to the memorandum: http://blogs.edweek.org/edweek/thisweekineducation/upload/2007/07/the_miller_reauthorization_mem/G_Miller_Memo_July_2007.pdf
Mapping State Proficiency Standards onto NAEP Scales
Under NCLB, states are required to report the percentages of students achieving proficiency in reading and mathematics for grades 3 through 8. As tempting as it is to compare proficiency scores across states, researchers are well aware of the pitfalls of this approach, given the differences in state curriculum standards and assessments.
In a recently released report (see link below), the Department's Institute of Education Sciences compared state assessment proficiency percentages to the estimated
percentages of students achieving proficiency with respect to the standards established by the National Assessment of Educational Progress (NAEP). IES found large discrepancies between the two. It attributed this variation to differences in both content standards and student academic achievement from state to state, as well as from differences in the stringency of the standards adopted by the states.
In a recently released report (see link below), the Department's Institute of Education Sciences compared state assessment proficiency percentages to the estimated
percentages of students achieving proficiency with respect to the standards established by the National Assessment of Educational Progress (NAEP). IES found large discrepancies between the two. It attributed this variation to differences in both content standards and student academic achievement from state to state, as well as from differences in the stringency of the standards adopted by the states.
Thursday, July 5, 2007
State Graduation Rates: A Failing Accountability Measure
The heightened focus on accountability in education and growing interest in high school reform in recent years has focused attention on one of the most significant indicators of a high school’s performance: the student graduation rate. Yet there has been considerable debate in educational policy circles for several years about how many students actually earn a diploma. Various reports cite national graduation rates as low as 70 percent and as high as 83 percent. The state picture is even more confusing: 36 states report graduation rates between 80 and 97 percent, while an independent source has suggested that the rates in these same states are only 58 to 86 percent.
While statisticians and researchers argue about the best approaches for estimating graduation rates, the public is left to ponder why state governments fail to accurately count the number of students who graduate from their schools each year and what can be done to fix this problem. This paper sets out to explain the reasons for this inaccuracy in graduation rates and to identify future directions for assuring accuracy in reported graduation rates.
At the time when the No Child Left Behind Act of 2001 made reporting of an on-time graduation rate an accountability reporting requirement, few states had data collection systems adequate to produce an accurate rate. The accurate calculation of this rate requires cohort data, or student record data on student progress from grade to grade, data on graduation status, and data on students who transfer in and out of a school, district, or state during their secondary school studies. Given the absence of cohort data, the NCLBA regulations offered the states some flexibility in reporting graduation rates. Specifically, the states were provided with the option of developing their own definition of a graduation rate that “more accurately measures the rate of students who graduate from high school with a regular diploma” (Title I Final Regulations, 2002, section 200.19 (a) (1) (B)), provided that the state did not count drop-outs as transfers. The states individually responded with new proxy graduation rate definitions, most of which were calculated using cross-sectional data.
Independent researchers quickly pointed to flaws in several of these state-created graduation rate definitions that had been approved by the U.S. Department of Education (hereafter referred to as the Department). The Education Trust examined the first round of state graduation rate data reported to the Department for the 2001-02 school year and found that many state graduation rates appeared to diverge significantly from independent estimates. The report writers asserted that many states used questionable graduation rate definitions, which too often had the effect of overinflating graduation rates. North Carolina, for instance, reported an impressive graduation rate of 92.4%. However, when Jay Greene, an independent analyst, applied a cohort definition to enrollment data and diploma counts in the Department’s Common Core of Data, he produced a graduation rate of only 63% for the state.
The problems associated with the North Carolina graduation rate definition are illustrative of the complex problem of graduation rate definitions. The North Carolina graduation rate definition was not based on the percentage of students who entered in the 9th grade and received a degree four years later, but on the percentage of diploma recipients who received their diploma in four years or less. Students who dropped out of high school or transferred were excluded from the calculations altogether. As the Education Trust noted in its 2003 report, this meant that “if only 50% of students who enter 9th grade in North Carolina were to eventually obtain a high school diploma, but every one of those 50% did so in four years or less, then North Carolina would report a “graduation rate” of 100%.” North Carolina’s rate definition clearly produces a misleading graduation rate, yet their graduation rate definition met the NCLBA criteria for state-defined rates and had been approved by the Department.
There are two issues with North Carolina’s graduation rate definition that recur in many states. First, the rate definition uses cross-sectional data, since in 2003, as is the case today, the state of North Carolina does not have a student record data system in place and therefore cannot use cohort data to calculate its graduation rate. Because student record data are not available, the state cannot track student progress from the 9th through the 12th grade to provide an accurate calculation of on-time graduation. Secondly, the cross-sectional data provide annual snapshots of the number of students and dropouts at one point in time in the school year, but they cannot track individual students as they transfer in and out of a school, district, or state during their secondary school education. This failure to account for transfers produces graduation rates that misrepresent the true proportion of students receiving an on-time diploma.
The need to determine accurate graduation rates fueled efforts at the national level to improve the state of educational data systems. Following the published recommendations of a National Center on Education Statistics task force on Graduation, Completion, and Dropout Indicators, the Department directed resources to assist states with the development of student record data systems. The Department launched, for instance, a $50 million program of cooperative agreements to help states develop their student record data systems; initial awards were made to 14 states in November of 2005. The National Governors Association undertook its own measures to improve the quality of graduation rates, developing the Graduation Counts Compact in 2005. Under the Compact, states commit to developing a high-quality, student-level data collection system that tracks students from kindergarten through college. It also includes a four-year cohort graduate rate formula: the number of students who graduated divided by the number of students enrolling in 9th grade for the first time four years earlier—plus the students who joined this class of students (that is, the cohort) and minus the students who left. All 50 states signed the contract and promised to implement the reforms.
However, at present fewer than half of the states have student record data systems that will enable them to follow individual students from pre-kindergarten through high school, and only a handful are using them to calculate a four-year cohort graduation rate (Hoff, 2006). It may take years for other states to get such systems in place. Curran reported that the National Governors Association anticipates that by 2010, 39 states will report a graduation rate using the Compact (cohort) definition. These states will begin reporting only after they have developed four or five years of longitudinal data capable of tracking students’ progress from their first-time entry into the ninth grade through their exit from high school.
It is not sufficient to report on cohort graduation data without taking into account student transfers. As the U.S. Government Accountability Office reported in its 2005 report, No Child Left Behind Act. Education Could Do More to Help States Better Define Graduation Rates and Improve Knowledge about Intervention Strategies, the primary factor affecting the accuracy of graduation rates is student mobility. The American family in the early 21st century is very mobile, presenting a challenge to state and local education agencies to accurately account for student transfers and dropouts.
In order to accurately track students who transfer in and out of a local education agency, a state must have a student record data system that includes exit or “leaver” data. These data are typically entered in the system as codes that identify the reason a student left a particular district or school (Data Quality Campaign, 2006). States with exit data systems in place provide their districts with a set of codes with which to identify reasons for students’ exits, including events such as death, transfer out of state, transfer to a home school, transfer to another country, transfer to a private school, incarceration, General Educational Development (GED) certificate, and hospital-bound.
Since exit code systems are critical to the reliability of graduation rates reported by states, the Department’s National Forum on Education Statistics, a cooperative of state, local, and federal education agencies, produced Accounting for Every Student: A Taxonomy for Standard Student Exit Codes (2006), which outlines the need for an exit code system and provides recommendations on how to establish a classification system of exit codes within the architecture of the student record data system. An exit code system, the Forum argued in this report, can greatly assist states in producing correct and comparable calculations of completion and dropout rates.
However, an exit code system is only as useful as the policies and procedures governing its data. A local educational agency, for instance, without policies and procedures in place to sufficiently identify the reasons for students’ exits from a system, may inaccurately report all students of “unknown” status as dropouts. It may also count students who drop out, return to school, and then drop out again as a dropout each time, inflating the district’s drop-out rate. Alternatively, a district may record students who drop out as transfers before they receive documentation that the student actually enrolled in a different school, thereby artificially lowering the drop-out rate. These situations illustrate the problems posed by insufficient policies and procedures for identifying and tracking student exits.
Identifying students who transfer is important to calculating graduation rates, since a state-defined rate definition under NCLBA, as noted earlier, can measure the rate of students who graduate from high school with a regular diploma provided that the state did not count drop-outs as transfers. State adoption of a standardized procedure for documenting transfers was recommended by the National Governors Association Task Force on State High School Graduation Data, as part of its recommendations to states on how to develop a high-quality, comparable high school graduation measure. Their task force had recommended use of a transcript request or other documentation from a receiving school as the only form of acceptable documentation to record a student departure as a transfer. By default, a student for whom there is no information should be documented as a nongraduate or dropout. However, as of August 2006, few states had established this procedure, although Curran reported that many states were developing checks and audit procedures as their student record systems came online that enabled undocumented student transfers to be identified when the student registered at another school or for students who had transferred out but had not shown up as enrolled elsewhere. This approach, however, does not fully address the issue of transfers to private schools, home schools, or out-of-state schools, which are harder to verify.
The use of proxy graduation rate definitions will soon end as states, with the support of the Department and interested groups such as the National Governors Association and the Data Quality Campaign, complete the process of developing student record data systems. When student record systems come online, it is anticipated that many of the accuracy issues plaguing graduation rate reports will simply disappear. However, the high school graduation rates reported using cohort data will only be accurate if actions are taken now to ensure that policies and procedures are developed and respected in regards to documenting student exits, especially transfers. Ultimately the leadership at the school, district, and state levels is responsible for creating a data-driven culture that makes it a priority to rethink and possibly reorganize how education data is managed throughout the system and increase training and professional development for staff, including managers and users. Only a broad-based commitment to data quality will produce accountability measures that accurately portray the state of education in the U.S.
While statisticians and researchers argue about the best approaches for estimating graduation rates, the public is left to ponder why state governments fail to accurately count the number of students who graduate from their schools each year and what can be done to fix this problem. This paper sets out to explain the reasons for this inaccuracy in graduation rates and to identify future directions for assuring accuracy in reported graduation rates.
At the time when the No Child Left Behind Act of 2001 made reporting of an on-time graduation rate an accountability reporting requirement, few states had data collection systems adequate to produce an accurate rate. The accurate calculation of this rate requires cohort data, or student record data on student progress from grade to grade, data on graduation status, and data on students who transfer in and out of a school, district, or state during their secondary school studies. Given the absence of cohort data, the NCLBA regulations offered the states some flexibility in reporting graduation rates. Specifically, the states were provided with the option of developing their own definition of a graduation rate that “more accurately measures the rate of students who graduate from high school with a regular diploma” (Title I Final Regulations, 2002, section 200.19 (a) (1) (B)), provided that the state did not count drop-outs as transfers. The states individually responded with new proxy graduation rate definitions, most of which were calculated using cross-sectional data.
Independent researchers quickly pointed to flaws in several of these state-created graduation rate definitions that had been approved by the U.S. Department of Education (hereafter referred to as the Department). The Education Trust examined the first round of state graduation rate data reported to the Department for the 2001-02 school year and found that many state graduation rates appeared to diverge significantly from independent estimates. The report writers asserted that many states used questionable graduation rate definitions, which too often had the effect of overinflating graduation rates. North Carolina, for instance, reported an impressive graduation rate of 92.4%. However, when Jay Greene, an independent analyst, applied a cohort definition to enrollment data and diploma counts in the Department’s Common Core of Data, he produced a graduation rate of only 63% for the state.
The problems associated with the North Carolina graduation rate definition are illustrative of the complex problem of graduation rate definitions. The North Carolina graduation rate definition was not based on the percentage of students who entered in the 9th grade and received a degree four years later, but on the percentage of diploma recipients who received their diploma in four years or less. Students who dropped out of high school or transferred were excluded from the calculations altogether. As the Education Trust noted in its 2003 report, this meant that “if only 50% of students who enter 9th grade in North Carolina were to eventually obtain a high school diploma, but every one of those 50% did so in four years or less, then North Carolina would report a “graduation rate” of 100%.” North Carolina’s rate definition clearly produces a misleading graduation rate, yet their graduation rate definition met the NCLBA criteria for state-defined rates and had been approved by the Department.
There are two issues with North Carolina’s graduation rate definition that recur in many states. First, the rate definition uses cross-sectional data, since in 2003, as is the case today, the state of North Carolina does not have a student record data system in place and therefore cannot use cohort data to calculate its graduation rate. Because student record data are not available, the state cannot track student progress from the 9th through the 12th grade to provide an accurate calculation of on-time graduation. Secondly, the cross-sectional data provide annual snapshots of the number of students and dropouts at one point in time in the school year, but they cannot track individual students as they transfer in and out of a school, district, or state during their secondary school education. This failure to account for transfers produces graduation rates that misrepresent the true proportion of students receiving an on-time diploma.
The need to determine accurate graduation rates fueled efforts at the national level to improve the state of educational data systems. Following the published recommendations of a National Center on Education Statistics task force on Graduation, Completion, and Dropout Indicators, the Department directed resources to assist states with the development of student record data systems. The Department launched, for instance, a $50 million program of cooperative agreements to help states develop their student record data systems; initial awards were made to 14 states in November of 2005. The National Governors Association undertook its own measures to improve the quality of graduation rates, developing the Graduation Counts Compact in 2005. Under the Compact, states commit to developing a high-quality, student-level data collection system that tracks students from kindergarten through college. It also includes a four-year cohort graduate rate formula: the number of students who graduated divided by the number of students enrolling in 9th grade for the first time four years earlier—plus the students who joined this class of students (that is, the cohort) and minus the students who left. All 50 states signed the contract and promised to implement the reforms.
However, at present fewer than half of the states have student record data systems that will enable them to follow individual students from pre-kindergarten through high school, and only a handful are using them to calculate a four-year cohort graduation rate (Hoff, 2006). It may take years for other states to get such systems in place. Curran reported that the National Governors Association anticipates that by 2010, 39 states will report a graduation rate using the Compact (cohort) definition. These states will begin reporting only after they have developed four or five years of longitudinal data capable of tracking students’ progress from their first-time entry into the ninth grade through their exit from high school.
It is not sufficient to report on cohort graduation data without taking into account student transfers. As the U.S. Government Accountability Office reported in its 2005 report, No Child Left Behind Act. Education Could Do More to Help States Better Define Graduation Rates and Improve Knowledge about Intervention Strategies, the primary factor affecting the accuracy of graduation rates is student mobility. The American family in the early 21st century is very mobile, presenting a challenge to state and local education agencies to accurately account for student transfers and dropouts.
In order to accurately track students who transfer in and out of a local education agency, a state must have a student record data system that includes exit or “leaver” data. These data are typically entered in the system as codes that identify the reason a student left a particular district or school (Data Quality Campaign, 2006). States with exit data systems in place provide their districts with a set of codes with which to identify reasons for students’ exits, including events such as death, transfer out of state, transfer to a home school, transfer to another country, transfer to a private school, incarceration, General Educational Development (GED) certificate, and hospital-bound.
Since exit code systems are critical to the reliability of graduation rates reported by states, the Department’s National Forum on Education Statistics, a cooperative of state, local, and federal education agencies, produced Accounting for Every Student: A Taxonomy for Standard Student Exit Codes (2006), which outlines the need for an exit code system and provides recommendations on how to establish a classification system of exit codes within the architecture of the student record data system. An exit code system, the Forum argued in this report, can greatly assist states in producing correct and comparable calculations of completion and dropout rates.
However, an exit code system is only as useful as the policies and procedures governing its data. A local educational agency, for instance, without policies and procedures in place to sufficiently identify the reasons for students’ exits from a system, may inaccurately report all students of “unknown” status as dropouts. It may also count students who drop out, return to school, and then drop out again as a dropout each time, inflating the district’s drop-out rate. Alternatively, a district may record students who drop out as transfers before they receive documentation that the student actually enrolled in a different school, thereby artificially lowering the drop-out rate. These situations illustrate the problems posed by insufficient policies and procedures for identifying and tracking student exits.
Identifying students who transfer is important to calculating graduation rates, since a state-defined rate definition under NCLBA, as noted earlier, can measure the rate of students who graduate from high school with a regular diploma provided that the state did not count drop-outs as transfers. State adoption of a standardized procedure for documenting transfers was recommended by the National Governors Association Task Force on State High School Graduation Data, as part of its recommendations to states on how to develop a high-quality, comparable high school graduation measure. Their task force had recommended use of a transcript request or other documentation from a receiving school as the only form of acceptable documentation to record a student departure as a transfer. By default, a student for whom there is no information should be documented as a nongraduate or dropout. However, as of August 2006, few states had established this procedure, although Curran reported that many states were developing checks and audit procedures as their student record systems came online that enabled undocumented student transfers to be identified when the student registered at another school or for students who had transferred out but had not shown up as enrolled elsewhere. This approach, however, does not fully address the issue of transfers to private schools, home schools, or out-of-state schools, which are harder to verify.
The use of proxy graduation rate definitions will soon end as states, with the support of the Department and interested groups such as the National Governors Association and the Data Quality Campaign, complete the process of developing student record data systems. When student record systems come online, it is anticipated that many of the accuracy issues plaguing graduation rate reports will simply disappear. However, the high school graduation rates reported using cohort data will only be accurate if actions are taken now to ensure that policies and procedures are developed and respected in regards to documenting student exits, especially transfers. Ultimately the leadership at the school, district, and state levels is responsible for creating a data-driven culture that makes it a priority to rethink and possibly reorganize how education data is managed throughout the system and increase training and professional development for staff, including managers and users. Only a broad-based commitment to data quality will produce accountability measures that accurately portray the state of education in the U.S.
Wednesday, June 20, 2007
Incentive Pay for Teachers ... New Buzz about an Old Idea
The new report by TeacherSolutions takes a favorable view of one of the profession's most controversial issues--merit pay. Merit pay is certainly not a new idea in American education. Merit pay policies have been formulated, adopted, implemented, and evaluated throughout the twentieth century. Single salary schedules for teachers, typically based on teachers' qualifications and years of services, have dominated teacher compensation systems since the 1920s. But what the NEA won't share is that historically the consideration of merit has been an important part of these single salary schedules since this earliest period.
In a nutshell, pay for performance is a systematic process for measuring teacher behavior or results, and linking these measurements to changes in teacher pay. Performance indicators may include changing classroom behavior, improving professional skills, and producing desired outcomes. A teacher’s knowledge and skills, for instance, may be assessed via testing, certification, and classroom observation. Classroom performance is typically measured by observations such as those done as part of a teacher evaluation system. Instructional outcomes may be assessed via tests of student learning and sometimes other measures such as attendance and graduation rates. Teacher performance can be defined in terms of these outcomes, and pay changes can be linked to outcome measures.
In response to the rise in educational accountability in this country, pay for performance systems have become increasingly sophisticated.
All indications from Washington are that pay for performance policy is here to stay, at least through the Bush Administration. The FY06 federal budget included funding for the Teacher Incentive Fund, a grant program that seeds the development of pay for performance programs, and for a National Research Center on Performance Incentives, established through a $10 million grant to Vanderbilt University from the Department of Education's IES. The President's FY2008 budget request for the Department of Education proposes to expand support for the Teacher Incentive Fund. The National Governors' Association, one of Washington's most respected public policy organizations, has been a vocal supporter of pay for performance policy, which it views as critical for improving teacher quality. Not surprisingly, governors across the country have raised the profile of this issue. As the Education Commission of the States has pointed out, "20 governors outlined teacher compensation as one of their major education issues in their 2005 State of the State addresses, and nine specifically spoke of some type of performance-based or merit pay" (Azordegan, Byrnett, et. al., 2005). Far from being an educational fad, pay for performance policy will have continued and profound effects on the teaching profession for some time to come.
Subscribe to:
Posts (Atom)
