The first data from the comparative analysis of the results on TIMSS-2011 and PISA-2012 tests, administrated to the same sample of Russian students Dr. Yulia Tumeneva, Senior Researcher, Laboratory for analysis of educational police HSE Alena Valdman, student on «Measurements in psychology and education» master program HSE TIMSS PISA

Research question How knowledge and skills measured in TIMSS associate with the skills measured in PISA?

Sample TIMSS-2011: 4893 pupils from 229 classes (49,3% girls and 50,7% boys) Sample PISA-2012: 4399 pupils from 229 classes (49,6% girls and 50,4% boys) Russian sample (TIMSS-2011/PISA-2012)

Strategy of analysis Divide TIMSS classes into 6 groups from the top performers to poor performers Pick up 10 and 20 hardest PISA items Check what is the percent of the ten (and 20) hardest PISA items each TIMSS group did correctly

229 classes in TIMSS How have we divided the sample? 35 classes 15,3% 39 classes 17,0% 39 classes 17,0% 39 classes 17,0% 38 classes 16,6% The classes have highest PVs math The classes have lowest PVs math classes 17,0% 6

One-Parameter Rasch Model (Partial-Сredit ) The difficulty of each tasks in the PISA in logits Selection of 10 (20) of the most difficult tasks PISA (highest logits) Hardest items PISA. How to reach them?

Results: mathematics Only highly developed TIMSS skills differentiate success in PISA

groupAlgebraData and chanceNumberGeometry 1 17,7617,8918,3118, ,849,9110,2811,23 3 7,076,807,456,72 4 7,416,287,117,67 5 6,177,125,735,47 6 2,843,433,083,17 % correct items of 10 hardest items in each group Results: content domains The same situation.

groupKnowingApplyingReasoning 118,8218,0318,25 29,8510,1010,64 36,978,047,45 45,955,406,50 57,447,516,21 62,882,873,07 % correct items of 10 hardest items in each group Results: cognitive domains The same situation.

What we talk about when we talk about hardest PISA items To associate information presented in different ways To keep relationships between things (or concepts) through time To use information from one domain to solve a problem in another domain To model relationships and changes mathematically

Some preliminary conclusions In terms of PISA and TIMSS tests we can say that TIMSS skills very weakly differentiate PISA skills. Factually only highest level of mastery of TIMSS skills enable success in PISA BUT! We can consider TIMSS skills as mastery of subject content; PISA skills as ability to transfer knowledge from one domain to another. Then our results mean that only highest level of mastery of subject content enables the meta-domain transition.

Future ways of analysis To check these results on individual level To check how overall TIMSS success can affect solving the hardest TIMSS items To specify what TIMSS domain/items/cognitive process affect success in different PISA domain/items/cognitive process To define other cognitive abilities, except subject knowledge, (e.g. analogical thinking) that can affect meta-domain transferring

Questions? Comments? Suggestions? Thank you for your attention!

Main references Алейникова И., Нечего на зеркало пенять, коли школа стара // Управление школой, 05, 2005 Ковалева Г.С., Особенности проведения международного исследования PISA-2012 в регионах России, доклад, 2012 (2) Пушникова Г. А зачем нам TIMSS и PISA? // Учительская газета, 6, 2006 РОСРО. Материалы дискуссии о конкурентоспособности российского образования // Вопросы образования, 2005, 1. С Вопросы образования Тюменева Ю.А., Хавенсон Т.Е.. // Тенденции развития образования: проблемы управления и оценки качества. Тюменева Ю.А.Хавенсон Т.Е. Материалы 8 международной научно-практической конференции, Москва: Дело, 2012 (1) Falch T., Rønning M. (2011). Homework assignment and student achievement in OECD countries. Working Paper Series, Norwegian University of Science and Technology, 5 Goldhaber D. (2002). The mystery of good teaching: Surveying the evidence on student achievement and teachers characteristics. Education next 2(1) Huang Francis L. & Moon Tonya R. (2009). Is experience the best teacher? A multilevel analysis of teacher characteristics and student achievement in low performing schools. Educ Asse Eval Acc, 21, P. 209–234 Hutchison D., Schagen, I. In Loveless, T. (Eds.), (2007). Lessons Learned: What International Assessments Tell Us About Math Achievement. Neidorf T, Binkley M., Gattis K., Nohara D. (2006). Comparing Mathematics Content in the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) 2003 Assessments // Technical Report Washington, D.C.: Brookings Institute Press Rindermann H. (2007) The g-Factor of International Cognitive Ability Comparisons: The Homogeneity of Results in PISA,TIMSS, PIRLS and IQ-Tests Across Nations. European Journal of Personality Eur. J. Pers. 21: P. 667–706 Torney-Purta J., Richardson W. K., Barber C. H. (2005). Teachers Educational Experience And Confidence In Relation To Students Civic Knowledge Across Countries. International Journal of Citizenship and Teacher Education, Vol. 1, No. 1 Wu, M. (2009). A comparison of PISA and TIMSS 2003 achievement results in mathematics. Prospects, 39: P.33–46 (1) Wu, M. (2009). A. A Critical Comparison of the Contents of PISA and TIMSS Mathematics Assessments (2) Wu, M. (2010). Comparing the Similarities and Differences of PISA 2003 and TIMSS. OECD Education Working Papers. No. 32. OECD Publishing Zuzovsky R. (2009). Teachers qualifications and their impact on student achievement: Findings from TIMSS 2003 data for Israel. IERI Monograph Series. Issues and Methodologies in Large-Scale Assessments, Vol.2, P