Can+we+make+legitimate+comparisons?

=//Can the comparison be legitimate?// = Legitimate comparisons are difficult at best using the resulting data from PISA. Factors that make a comparison legitimate consist of PISA 2003 allowed for a system in which different countries were disqualified from having their data fully calibrated. “The United Kingdom, having missed several participation criteria, was excluded from tables in the official report [from PISA 2003]. However, data from the UK were fully used in calibrating the international data set and in calculating the OECD averages - an inconsistency that is left unexplained” (Wuttke, 2007, p. 1-2). The PISA system therefore broke its own regulations that were designed, in part, to help ensure consistency and ensure legitimacy. PISA 2003 allowed for a system in which different countries were disqualified from having their data fully calibrated. “The United Kingdom, having missed several participation criteria, was excluded from tables in the official report [from PISA 2003]. However, data from the UK were fully used in calibrating the international data set and in calculating the OECD averages - an inconsistency that is left unexplained” (Wuttke, 2007, p. 1-2). The PISA system therefore broke its own regulations that were designed, in part, to help ensure consistency and ensure legitimacy. “In some countries, it is clear from the outset that PISA cannot be representative...But even in countries where school is obligatory beyond the age of fifteen, low participation rates are likely to introduce some bias” (Wuttke, 2007, p. 5). This is represented partially in the ways that some countries purposely keep students from taking the PISA exam by not including the same percentages of populations. “Rules allowed countries to exclude up to 5% of the target population: up to 0.5% for organizational reasons and up to 4.5% for intellectual or functional disabilities or limited language proficiency. Exclusions for intellectual disability depended on ‘the professional opinion of the school principal or by other qualified staff’” (Wuttke, 2007, p. 6). This leads to questions not only of concerns regarding the statistical calculations but of tampering with results as per excluding students due to learning disabilities. Other discrepancies are detailed by Wuttke (2007) when he finds that “In Austria, 1.6% of the target population were completely excluded, and 0.9% of the participating students got the short test [made available to some with learning disabilities]. In Hungary, 3.9% were excluded, and 6.1% did the short test” (p. 7). Other issues that bring questions to the issue of PISA’s legitimacy were posed by to Stephen Hopmann and Gertrude Brinek as follows:
 * sampling disparities
 * loosely defined target populations
 * unequal inclusion of learning disabled students
 * inconsistent response rates
 * 1) “PISA is by design culturally biased and methodologically constrained to a degree which prohibits accurate representations of what actually is achieved in and by schools. Not is there any proof that what it covers is a valid conceptualization of what every student should know.
 * 2) The product of most public value, the national league tables...are based on so many weak links that they should be abandoned right away...
 * 3) The widely discussed by-products of PISA, such as the analyses of ‘good schools’, ‘good instruction’, or of differences between school systems and on issues like gender, migration, or social background, go far beyond what a cautious approach to these data allows for. They are more often than not speculative, and would at least need a wider framing by additional research looking at the aspects, which PISA by design cannot cover or gets wrong.
 * 4) “Any policy making based on these data (whether about school structures, standards or the curriculum) cannot be justified. The use and misuse of PISA data in such contexts - done with or without PISA researchers consent or cooperation - belongs solely to the sphere of policy making. Of course PISA researchers have the same right as every citizen to pronounce their political convictions in public. However they cannot do so claiming research as an unquestionable basis for their arguments” (pp. 12-13).

//**References **// Wuttke, Joachim, Uncertainty and Bias in PISA (November 1, 2007). PISA ACCORDING TO PISA. DOES PISA KEEP WHAT IT PROMISES, Hopmann, Brinek, Retzl, eds., pp. 241-263, Wien, 2007. Available at SSRN: http://ssrn.com/abstract=1159042 <span style="font-family: 'Times New Roman',Times,serif; font-size: 12.5px; line-height: normal; white-space: pre-wrap;">

<span style="font-family: 'Times New Roman',Times,serif; line-height: normal; white-space: pre-wrap;">Give us feedback on the PISA Project Wiki by filling out this form