Large Scale Data Initiatives, Child Policy, and Measuring what Matters
by Dominic Richardson
-
The goals and content of existing large-scale, statistical tools
-
Conclusion: How can we best use large-scale statistical tools to identify what to measure?
This article is the responsibility of the author, and does not necessarily reflect the opinions of UNICEF, or UNICEF’s Office of Research – Innocenti.
1. Introduction
Policy-makers globally are increasingly looking to large-scale data to inform their decision-making. And the field of social policy, and specifically child policy, is no exception. At present, global data initiatives (e.g. World Development Indicators, Millennium/Sustainable Development Goals) and cross-national child and family surveys (e.g. PISA, TIMSS, PIRLS, HBSC, and MICS) are key sources for global information on issues such as education, health, deprivation and family functioning. In an increasingly globalised world, these sources are feeding directly into public policy discourse (both internationally and nationally), policy guidance, and inevitably policy-making for children. But how might the proliferation of such data initiatives – and the influence they have on decision-makers – inform or stifle the goals of understanding and measuring what matters?
This article begins by asking how to measure what matters from the perspective of the Learning for Well-being (L4WB) framework, the purpose of this focus is to draw from a theoretical frame with a specific reference to ‘measuring what matters’ around a list of observable and measureable personal capacities, in order to assess what large-scale data initiatives can contribute. The paper goes on to look at available existing large-scale statistical tools (in the OECD and Europe) and then introduces their goals, content, and use. The penultimate section of the paper discusses the potential of these studies to inform the measuring what matters discussion. The final section concludes by asking: what steps can be taken to help this type of analytical approach better measure what matters?
2. How L4WB measures what matters?
A key principle at the heart of the L4WB vision and framework is measuring what matters – or the idea that living systems should feedback on what is happening to them in order to express and understand health and well-being. Measuring what matters is a principle derived from the L4WB vision in which individuals are equipped to realise their uniqueness, life purpose, to grow and learn in diverse ways (emotionally, physically, mentally and spiritually) to build relationships, to respect those around you, and to engage in their social ecosystem, as a process and an outcome in a self-determined way. Core capacities and practices are the mechanism by which the L4WB framework might most readily operationalise its vision, because they represent attributes and practices that ‘enhance our ability to realize our unique ways of contributing to the fullness of our own life and the lives of those around us’ (L4WB, 2015).
Core capacities and practices, according to the L4WB framework, include: relaxation, paying attention, reflection, listening, inquiring, empathy, discerning patterns and systemic processes (recognising interdependency), subtle sensing (including intuition and imagination) and enriching sensory awareness (nurturing stimulating and expanding the senses) (ibid).
The idea that personal attributes constitute an important part of the determinants of learning is not new, and some of the core capacity measures will not be new to scholars of the life skills literature. Nonetheless, core capacities represent a unique set of observable and measureable personal attributes which can be used as a starting point for assessing whether large-scale data initiatives and analyses are fit for measuring what matters.
3. Policy, evidence, and international data
Increasingly policy-makers are looking to large-scale data to inform their practices. Evidence-based policy is a catchphrase in politics, academia, and civil society; with each field understanding the need to support policy arguments with evidence and data, and to move from ideological propositions for change.
The increased demand for evidence to inform child policy is seen most readily in large international surveys of children and associated monitoring tools. Examples of international child surveys in the OECD and Europe include:
-
European School Project on Alcohol and other Drugs (ESPAD);
-
Health Behaviour in School-aged Children (HBSC) study;
-
International Civic and Citizenship Education Study (ICCS);
-
Progress in International Reading Literacy Study (PIRLS);
-
Programme for International Student Assessment (PISA); and
-
Trends in International Mathematics and Science Study (TIMSS).
These studies all survey children in schools, collecting data from children aged 9-16 (Richardson and Ali, 2014). The earliest study dates back to the early 80s (HBSC), and the most prominent of these surveys, PISA, has grown since its inception in 2000 to survey over half a million children in 65 economies worldwide in 2012 (OECD PISA, 2015a), and today is ever present in child policy debates across the industrialised world (both in specific policies [Breakspear, 2012] and during elections [Guardian, 2014]).
With the increasing availability of large-scale data comes larger-scale use. International data sources allow for international comparison, and the number of indices and monitoring tools for children have grown also. In the past two decades, at least 10 indices of child or youth well-being have been published that cover multiple countries of the developed world (see Richardson and Ali, 2014). These include the Millennium Development Goals, the forthcoming Sustainable Development Goals (SDGs), The Human Development index, and even OECD’s Better life Index, represent further multidimensional cross-national suites of social progress indicators designed to monitor life outcomes. The purpose of these indices is to encourage policy-makers to take a broad view of how policy can impact on a wide range of social progress measures for the populations they serve, through monitoring successes and failures, setting standardised targets, and providing a reference point by which policy effectiveness and social progress between countries can begin to be compared. In particular, the shift to a global focus in the SDGs underlines how far the use of data and evidence and their potential influence has come, and implicitly the size of the data challenge to be faced.
4. The goals and content of existing large-scale, statistical tools
The goal of each child survey is to understand one, or at most two, factors of child well-being. These are commonly education-based (non-curricula based literacies in PISA, curricula-based literacies in TIMSS and PIRLS) or health-behavioural (HBSC, ESPAD). All child surveys have a stated goal, the most common of which is to inform policy (see Richardson and Ali, 2014).
Series data, such as those derived from the World Development Indicators, the WHO mortality database, or various OECD databases on health, education or income, are primarily used for monitoring, and offer limited opportunities for the type of advanced statistical testing required to build detailed evidence that can be used to inform policy and practice.
In 2011, the OECD undertook a stocktake of questionnaire items in the major ongoing international surveys of children (Richardson and Ali, 2014). It included the above surveys, as well as household surveys commonly used to inform child well-being measurement (European Survey of Income and Living [EU-SILC], the European Social Surveys [ESS], and the European Quality of Life Survey [EQLS]), as well as the Civic Education Study undertaken in 1999. The authors found almost 480 separate items related to the measurement of child well-being as commonly defined in the literature (multi-dimensional), the majority related to material well-being, and behaviours and risk (see Figure 1).
Figure 1: Major household studies and health behaviour studies dominate present cross-national knowledge on child well-being. Source: Richardson and Ali, 2014 – based on summary data in online Annex.
So far, these studies have produced very little on social and emotional skills – an area within which the concept of core capacities might be most readily housed, and the most promising area to begin to assess how well these data initiatives can contribute to measuring what matters.
Table 1 below draws on the detailed annex of Richardson and Ali (2014) to map information available from ongoing surveys under broad ‘core capacities’ heading identified in the L4WB framework. The information is mapped in the Table at the level of ‘sub-dimension’ descriptions, meaning that for each description a question on that topic has been included in the relevant survey.
Table 1: Children are commonly asked to reflect on their experiences, and relationships. Note: Surveys questionnaires may have been updated since the publication of the source material. Source: Author, based on Richardson and Ali, 2014.
Each topic in Table 1 has been allocated to a core capacity on the basis of longer descriptions of core capacities in the L4WB documentation (O’Toole, 2014). For instance, the decision to include suicide attempts and self-harm under ‘relaxation’ was made on the basis of suicide attempts and self-harm being indicative of a lack of ‘emotional, mental and spiritual relaxation’. This of course is a first attempt at mapping, which is undoubtedly open to further interpretation and constructive criticism!
Though children are commonly asked to reflect on experiences and relationships, it is important to note that no items in the main international surveys actually measure the core capacities themselves. The items identified in the table are better described as proxies of life skills, either based on reported experiences where life skills are being applied, or opinion items that indicate aspects of the core capacities (such as resonance with others, openness and curiosity).
In terms of upcoming developments, as of 2015, PISA will include a measure of empathy in its survey (OECD PISA have long acknowledged social and emotional skills in their work [‘key competencies’ see OECD PISA, 2015b], but have only recently begun to operationalise the concept).
5. The use of existing large-scale, statistical tools
The most common ways international surveys of children, and international series data on children, are used are in the form of league tables that assess which countries are at the top of a literacy league, which country has the most overweight children, or the happiest children. More detailed analysis with these surveys and series is also possible, and the academic literature is full of examples of studies that attempt to identify the determinants of literacy outcomes for health or relationships outcomes at the individual, school or national levels. These studies are designed to inform policy strategies (system effectiveness, school functioning) or practices (reading to children) for improving child well-being.
However, these data sources are complex, and often require nuanced interpretation of the findings in order for the studies to be truly informative. In the case of analysis of children’s surveys, the most common and basic error is using school-based populations to represent all children (Richardson and Ali, 2014). The surveys, to different levels in different countries (and so involving a form of country-level bias), exclude children with special educational needs, younger children (under the age of 9), and will inadvertently miss children who have low attachment to school due to illness, poverty or behavioural challenges (Ibid). Missing and hidden populations of children result in the studies being biased towards ‘better-off’ children, and as a result, can bias policy responses.
These international surveys and series have further challenges, insofar as they are cross-sectional, and are limited sources for isolating causal determinants of the outcomes they measure. And, as noted above, all of the child surveys are designed to understand one or two factors of child well-being.
Together the lack of complete child samples, country-level bias, the lack of a longitudinal data, and the sectoral foci (education, health, so on), represent serious challenges for the researcher wishing to use these sources for informing ‘a child policy portfolio’ (OECD, 2009), or how policies and programmes might be managed across the life course of a child. Moreover, they are also a challenge for monitors, and policy-makers, whose targets and measures may be partial measures, and whose ‘news friendly’ league tables are misrepresenting the reality. In both cases this runs directly contrary to the stated goals of most of these surveys to ‘inform policy’.
6. The potential for measuring what matters?
So what does all of this tell us about how the proliferation of such data initiatives – and the influence they have on decision-makers – inform or stifle the goals of understanding and measuring what matters?
The strengths of the data sources, without which we would be much poorer in terms of understanding how children are faring across much of the world, are many, and they have great potential for measuring what matters.
For instance, these surveys and series are flexible to change, and will include new indicators when the argument is strong, and have been working towards building better outputs, more harmonised sources, and better guidance for appropriate use of the data they produce. Moreover, they do presently offer a number of promising routes for analysis, based on the available proxies from child surveys (see Table 1), and PISA’s development of an empathy item in the 2015 questionnaire. Their coverage is wide, covering many children in many countries, allowing for analysis of children’s voices and experiences across varied cultures and contexts – and they are growing. Finally, and importantly, policy-makers pay attention to these surveys and series, and their messages have a meaningful influence on decision-making for children across OECD and European countries (through league tables, monitoring and policy analysis).
However, what is evident, is that these surveys do not ask questions about core capacities directly, and the framework, as defined by L4WB, is only partially covered. To truly contribute to the understanding of measuring what matters, existing data sources will need to develop/reform, for which they will require a strong rationale.
Therefore, the potential for these initiatives to inform (or stifle) the goals of understanding and measuring what matters lies in how the L4WB community, and others, can engage with debates and structures around these collections. Successful engagement will lever the advantages of the large-scale data initiatives to understand more about measuring what matters, and bring the idea of core competences, and the process of learning through well-being, to the attention of policymakers.
7. Conclusion: How can we best use large-scale statistical tools to identify what to measure?
At present, these large-scale statistical tools will not answer all the questions about measuring what matters, so what can be done to make best use of these wide-reaching, politically-influential, and valid sources of data on children and their lives?
Below three initial steps are recommended for making the best use of existing large-scale data sources:
-
analyse the available indicators (Table 1) to assess how well they represent life skills and core capacities;
-
develop new international items (robust to cultural differences) from examples in national studies of life skills, and communicate these to established survey decision-makers, though coordinators or governing boards, with a strong rationale for inclusion;
-
identify entry points (rotating modules) in existing surveys to facilitate take up; new ‘core capacities’ studies should align output to established surveys and measures (through harmonisation, or relevant research methodologies), and political targets (such as educational outcomes) to clearly establish how core capacities contribute to achieving these goals.
Finally, under the circumstances that existing data initiatives do not engage with measuring what matters, and stifle opportunities for developing core capacity items, key stakeholders in measuring what matters should assess the feasibility of undertaking their own surveys (starting with a meta-analysis of available literature and surveys in the field).
The path to measuring what matters in a meaningful way, so that it becomes the focus of policy attention and intervention, is going to be long. Figuring out ‘how’ to measure what matters is only the beginning. Once core capacities are being measured, policy-makers will want to know why they matter for which children, whether measures are robust nationally and internationally, what factors (social, economic and political) might influence change in core competences, and how to implement mechanisms for change most effectively (including cost benefit analysis). At the very beginning of the discussion, it is going to be useful to think of ways not just to measure what matters, but question why it matters, what drives differences in what matters, and how to facilitate effective change.
Author
Dominic Richardson is Senior Education Specialist at UNICEF Office of Research – Innocenti. He joined Unicef – Office of Research in January 2015, working on issues of equity in education and the relationships between schooling, school outcomes and child well-being. Dominic previously worked in the OECD Social Policy Division where he undertook evaluations of child and family policies, designed frameworks through which indicators of child well-being outcomes can be compared, and led a project evaluating large international surveys of school children in developed countries.
References
Breakspear, S. (2012), ‘The Policy Impact of PISA: An Exploration of the Normative Effects of International Benchmarking in School System Performance’, OECD Education Working Papers, No. 71, Paris: OECD Publishing. OECD iLibrary website
The Guardian (2014), ‘As Swedish elections loom, cracks start to show in “Nordic” model’, The Guardian website, accessed December 2015.
O’Toole, L. (2014). ‘Well-being as Wholeness: The Perspective, Process and Practice of Learning for Well-being,’ Chapter 2 in Improving the Quality of Childhood in Europe 2014, Brussels: Alliance for Childhood Europe Network Group.
Richardson, D. and N. Ali (2014), ‘“An Evaluation of International Surveys of Children’”, OECD Social, Employment and Migration Working Papers, No. 146, Paris: OECD Publishing. OECD iLibrary website
OECD (2009), Doing Better for Children, Paris: OECD Publishing.
OECD PISA (2015a), About PISA. OECD website, accessed November, 2015.
OECD PISA (2015b), Definition and Selection of Competencies (DeSeCo). OECD website, accessed December 2015.