Introduction

Service providers, policymakers, and researchers need to know how likely specific interventions are to improve employment and related outcomes if implemented in a particular setting with clients. In practice, most employment interventions offer a combination of services that are designed to improve labor market outcomes (e.g., employment, earnings, education and training, and public benefit receipt). The sheer volume of research on these interventions, combined with the diversity of intervention characteristics, study participants, and findings, can make it challenging to identify what works.

The Pathways to Work Evidence Clearinghouse is dedicated to gathering information from research and making evidence more useful to decision makers. As part of these efforts, the Pathways Clearinghouse has undertaken a series of research syntheses to explore what we can learn by looking across all the data the Pathways Clearinghouse has collected. This report, one in that series, uses Bayesian meta-analysis—an analytic approach that synthesizes relationships across multiple studies—to provide new evidence on the likelihood that specific interventions will improve labor market outcomes and which combinations of services are most likely to be effective for different groups of people.

Research questions

  • What are the most effective interventions in the Pathways Clearinghouse?
  • Which services and combinations of services have the highest probabilities of improving outcomes?
  • Which services are most effective based on characteristics of study participants?

Purpose

Policymakers, practitioners, and researchers can use this report to identify the interventions, and the combination of services provided by those interventions, that are most likely to improve labor market outcomes for different groups of people.

Methods

The Pathways Clearinghouse team systematically identified, categorized, and assessed studies of interventions designed to improve the labor market outcomes of individuals with low incomes. The team recorded information about the study methods, the characteristics and impacts of the interventions they examined, and the populations served. We identified the primary service offered by each intervention and the populations served by each intervention, and standardized impacts across studies.

For this report, we synthesized the evidence using Bayesian meta-regression to distill all available information into an assessment of the probability that an intervention, service, or combination of services would improve outcomes. A meta-regression estimates the relationships between impact estimates and individual study and intervention characteristics while holding all other characteristics constant. Bayesian meta-regression is an advanced form of meta-regression that enables us to estimate relationships between impacts and specific combinations of intervention and study characteristics that might appear only rarely in the data. This approach allows us to model the relationship between these characteristics and estimated impacts in a way that captures some of the complexity inherent in these relationships. We used these findings to draw policy-relevant assessments of effectiveness by calculating the probability that interventions and intervention characteristics could be genuinely effective, given all the available evidence.

Key Findings and Highlights

  • Twelve of the 127 interventions examined for this analysis have more than a 90 percent probability of improving labor market outcomes for participants.
  • Interventions focused on work and work-based learning have the highest probability of improving outcomes relative to interventions focused on delivering other services.
    • Interventions focused on work and work-based learning have a 74 percent probability of improving labor market outcomes overall (including earnings, employment, education and training, and public benefit receipt), and a 94 percent probability of improving employment outcomes specifically.
    • Interventions focused on work and work-based learning were most effective when combined with employment services; these interventions have an 81 percent probability of improving outcomes when offered in combination with work and work-based learning.
  • No single primary service is particularly likely to improve outcomes by a substantial amount (by at least $1,000 in annual earnings).
    • Though interventions focused on work and work-based learning have the highest probability of improving outcomes by any amount, they are among the least likely to improve outcomes by a substantial amount relative to interventions focused on delivering other services (a 21 percent probability of improving outcomes by an amount equivalent to $1,000 or more).
    • Interventions focused on subsidized and transitional jobs are the most likely to improve outcomes by an amount equivalent to $1,000 or more (a 32 percent probability).
  • The likelihood that an intervention offering a particular primary service improves outcomes differs depending on the characteristics of the population being studied. However, interventions focused on work and work-based learning, employment retention services, and case management and other supports typically have the highest probabilities of improving outcomes in interventions serving primarily people from racial or ethnic minority groups, women, and people with disabilities.

Citation

Shiferaw, Leah and Dan Thal. (2022). Digging Deeper into What Works: What Services Improve Labor Market Outcomes, and for Whom? OPRE Report # 2022-161, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.