Measuring the Nuances of Youth Employment & Livelihoods

The challenge of youth unemployment in low and middle-income countries is well-known. According to the International Labor Organization (ILO), there are close to 60 million young people without work worldwide. Scalable solutions to address youth unemployment are emerging and advances are being made in effective measurement strategies and tools to know if we are making a difference.

This topic was front and center at the Global Youth Economic Opportunities Summit supported in part by USAID and held in Washington D.C. from October 1-4.

In a session organized by the Youth Employment Funders Group (YEFG), a coordination group for donors supporting youth employment programs, USAID and the ILO (which are two of the founding members of the group), presented measurement tools and guidance notes which are meant to enable development practitioners to understand youth livelihoods and employment situations in varying labor markets. Consistently and accurately measuring these elements is essential to improving programming, scaling up action, and informing future policies and interventions. 

Smarter Metrics
“Looking only at unemployment numbers is not enough,” said Susana Puerto Gonzalez, Senior Youth Employment Specialist with the ILO. “We have to be smarter and innovative by looking at metrics that analyze what is really going on with young people in the labor markets, especially those who are not engaging in the world of work.” 

“Smart” metrics is about diving deeper into research and looking for nuances. For instance, “globally, there are 136 million youth that experience working poverty,” said Puerto. “These are young people working but not earning enough to lift themselves out of poverty.” This indicator tells us more about the reality faced by youth in low- and middle-income countries.

Another metric is the NEET indicator, Not in Education, Employment, or Training. This tracks a person who is not employed, but also not in school or in some type of training. Puerto points out that young women make up three quarters of the youth in NEET.

“Globally, youth NEET rates from 2005 to 2020 are above 20 percent, much higher than if you measured unemployment only. Smart metrics,” said Puerto, “can improve how we invest, how we target, and how we program.”

Improving Targeting and Accountability
Better targeting of youth employment programs means understanding where young people are coming from, where they are going, and how satisfied they are in the workplace. “If we understand national realities and how young people are transitioning in a given context,” said Puerto, “we can improve the targeting of our interventions. We can also identify vulnerabilities and focus on the types of interventions that respond to specific needs.”

ILO’s Youth Employment Evidence Gap Maps is the outcome of a systematic review of impact evaluations that cover youth employment interventions from 1990 to 2014. The ILO also published a Guide to Measure Decent Jobs for Youth, which proposes other types of indicators. “This guide captures information about the quality of jobs that young people are in,” said Puerto. “Do they have a contract, do they have benefits, are they aware of their labor rights, and do they have social security? These are the questions we should be asking.”

Youth employment policies are as important as the programs that equip young people with the skills to enter the job market. The ILO, said Puerto, has developed indicators that track the impact of youth employment policies so that countries are accountable, including the implementation of national strategies for youth employment. 

WORQ: USAID’s New Employment and Earnings Measurement Package of Tools
“The way we’ve been collecting data from youth about their employment up until now,” said Rebecca Pagel, Senior Monitoring and Evaluation Advisor with USAID’s Center for Education, is “by asking them what we perceive to be objective questions and writing down exactly what they say. This has not always yielded reliable and valid data.” This is partially what prompted USAID to develop the Workforce Outcomes Reporting Questionnaire (WORQ) Package, a youth employment measurement tool that the Agency has been developing and testing for the past two years. 

The WORQ package of tools is intended for use by activities to measure youth employment and earnings.  It specifically tracks the following three USAID standard indicators: 

• EG.6-12: Percent of individuals with new employment following participation in U.S. Government (USG)-assisted workforce development programs.
• EG.6-11: Average percent change in earnings following participation in U.S. Government (USG)-assisted workforce development programs.
• ES.I -46: Percent of individuals who transition to further education or training following participation in U.S. Government (USG)-assisted programs.

WORQ also allows activities to collect more information about a youth’s employment situation. In the process of developing and piloting the WORQ package of tools, USAID found evidence of the complexity of youth employment and changed the way they require the tool to be implemented. 

The tool recognizes and captures that complexity through a methodology that is more typical of qualitative interviews than quantitative surveys. 

USAID’s WORQ package of tools offers adaptable, detailed questions that can help program managers track employment and earnings information for youth participants in workforce programs and ultimately better measure a program’s impact.

The WORQ package contains four tools:

1)    The WORQ: includes questions to measure and report on three key indicators;

2)    The Extra WORQ: includes all WORQ items as well as additional items that may provide helpful background information on participant’s work lives;

3)    The Impact WORQ: includes items that capture information on secondary sources of earnings to allow for a more accurate comparison between program participants and control or comparison group;

4)    The Quick WORQ: seeks to obtain trends over time and is particularly recommended for contexts in which seasonal work is common.

The WORQ package of tools will be required for new USAID activities reporting against these three indicators starting in FY19.

The Importance of Cognitive Testing
A young person’s type of employment determines the questions asked in the WORQ package. The tool accounts for regular and irregular wage work, self-employment, helping in a household enterprise, and even working for in-kind pay. The tool was administered in Kyrgyzstan, Rwanda, Philippines, Kenya, and El Salvador through cognitive testing, a process that entails sitting down with the participant and going through the survey together. In the cognitive testing phase of tool development, says Pagel, “we were less interested in the answer to the questions but in how respondents are understanding the questions.” 

In the cognitive testing process, the research team gleaned information that was used to revise the WORQ before pilot testing:

• Respondents struggled to recall information from three months prior and altered these questions to refer to the previous month.
• Respondents did not understand ‘permanent’ wage work. The tool was revised to instead talk about regular and irregular wage work.
• Originally, some respondents provided pre-tax earnings information and some provided post-tax earnings information. Questions on earnings were revised to ask about “usual take-home pay”

The tool was also pilot tested and test/re-tested in Rwanda and Kyrgyzstan. So what were some of the key take-aways?

Lessons Learned from WORQ
The test/retest made it clear that traditional methods of obtaining survey data do not always produce reliable information with youth. “In Kyrgyzstan, we had a 30 percent variation between the pilot and the test/re-test after re-administering the test one week later,” says Pagel. “We had to try different approaches to ensure that the questions were understood correctly.” USAID added probing questions and gave the interviewer additional notes. That is when reliability increased.

“Even though USAID tested this tool in five countries and revised it each time,” Pagel said, “we know that we need more testing and that the WORQ tool will continue to evolve.” The fact of the matter is that “youth’s work lives are often more complex than we think. So, we need to give them the opportunity to be more than a checkmark in an answer box. And this takes time.”