Frequently Asked Questions

The Pathways Clearinghouse maintains a list of services that we use to characterize interventions and conduct analyses. This list is located on the website within Find Interventions that Work, under the filter “Select services the intervention should provide.” The list nests some of the services underneath other services, to show services that are part of a broader service category. A version of the list is also included  in the Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards. Interventions can offer any combination of the services on the list, and any of the services listed may be primary services. As the Pathways Clearinghouse continues to expand, other services may be added to the list.

Pathways Clearinghouse meta-analyses currently combine groups of closely related services from the list to produce a meaningful number of studies for analyses based on a group of services. Those groups are used as part of analyses to determine what types of services are associated with better outcomes. For example, What Works to Improve Employment and Earning for People with Low Incomes combines education and training services and compares the effectiveness of these services to other types of services, such as work and work-based learning and incentives and sanctions.

Evidence Snapshots are short briefs on the effectiveness of interventions that have a specific primary service. In some cases, a single Evidence Snapshot will summarize the effectiveness of interventions that provide either of two closely-related primary services. For instance, the Work Experience and Work-Based Learning Snapshot reports on interventions with primary services of work and work-based learning or work experience because these are two similar service strategies. 

The Pathways Clearinghouse created a list of potential services and uses that list across the website and products. Some services are nested within other services. For example, occupational or sectoral training is nested within training, because it represents a type of training. The higher-level category of training includes other types of training, in addition to occupational or sectoral training. If you are interested in understanding the range of interventions that feature training, your search will identify interventions that include occupational and sectoral training, as well as other types of training. All potential services on the website can be the primary service.

Employment and training interventions commonly provide light-touch supports—such as money or vouchers for child care or transportation—to address barriers to training or work. The Pathways Clearinghouse categorizes these kinds of services as “supportive services,” because they are explicitly designed to support employment and training. The Pathways Clearinghouse also reviews interventions that may improve labor-market outcomes through strategies other than employment and training services, including by addressing barriers to work through provision of intensive housing, child care, or legal assistance services. The Pathways Clearinghouse categorizes these types of services as “housing,” “child care,” or “legal assistance” services to indicate that the supports are more intensive and occur outside the context of an employment and training program. The Pathways Clearinghouse added these services tags during the third round of review, which spanned 2021 and 2022—and conducted a targeted re-review of studies previously included in the Pathways Clearinghouse to determine if the “housing,” “child care,” or “legal assistance” services tags should be applied.

The Pathways Clearinghouse is an investment of the Office of Planning, Research, and Evaluation in the Administration for Children and Families within the U.S. Department of Health and Human Services. Launched in 2020, the Pathways Clearinghouse identifies interventions that aim to improve employment outcomes, reduce employment challenges, and support self-sufficiency for populations with low incomes, especially recipients of public programs such as Temporary Assistance for Needy Families (TANF). It is a comprehensive resource for direct employment service providers, TANF administrators, policymakers, workforce development experts, and researchers. The Pathways Clearinghouse is designed to help users easily access and understand the effectiveness of interventions that aim to improve employment outcomes for job seekers with low incomes. Information about an intervention’s effectiveness is presented alongside information for better understanding the context in which it has been shown to work and how it was implemented.

The Chief Evaluation Office within the U.S. Department of Labor established the Clearinghouse for Labor Evaluation and Research (CLEAR) in 2012 to increase the accessibility of research on a wide range of labor and employment topics. With more than 20 evidence reviews conducted to-date, CLEAR is a central source for labor-related research and evaluation evidence on a variety of interventions spanning employment, training, health and safety, worker benefits, employer compliance, and other topics. CLEAR aims to help practitioners, policymakers, researchers, and the public find and use research to inform their decisions about labor policies and programs, future research, and improve outcomes for a variety of populations.

The Pathways Clearinghouse and CLEAR are coordinated federal efforts that have complimentary areas of focus and use similar standards to assess the quality of causal evidence in studies. Both Clearinghouses primarily focus on reviewing and rating causal impact studies. The Clearinghouses have collaborated to share studies and to combine dissemination efforts.

While there is strong alignment between these Clearinghouses, they are also important differences between them. These differences include their distinct goals, the ways they aggregate data across studies, and the additional types of study designs they review. As a result, they may be used by stakeholders for different purposes.

Goals and scope

• CLEAR is designed to be responsive to the entirety of the work and programs overseen by the Department of Labor, including, for example, compliance with occupational safety guidelines and child labor.
• Pathways, on the other hand, is more narrowly focused on self-sufficiency for individuals with low incomes.

Data aggregation

• In addition to reviewing individual studies, the Pathways Clearinghouse aggregates information across studies to determine an intervention’s effectiveness. It rates an intervention’s effectiveness within each of four outcome domains: earnings, employment, public benefit receipt, and education and training.
• CLEAR primarily focuses on assessing individual studies and only rates interventions for effectiveness in one of its topic areas, Reemployment Services and Eligibility Assessments (RESEA), to help states meet program requirements.

Types of studies reviewed While the focus for both Clearinghouses is on causal research, they also provide information about other types of research.

• In some topic area evidence reviews, CLEAR also reviews descriptive and implementation studies.
• While the Pathways Clearinghouse does not systematically review noncausal studies, Pathways does provide information about interventions with an evaluation underway (developmental interventions).

Individually and in combination, these websites provide a rich array of information about what works to help individuals succeed in the labor market and users are encouraged to examine both websites.

Find more information about CLEAR by visiting: Clearinghouse for Labor Evaluation and Research: https://clear.dol.gov/

We will send occasional mailings to users who subscribe to our email list, alerting them to new website content and open calls for studies. For real-time updates on Pathways Clearinghouse activities, events, and presentations, follow the Office of Planning, Research, and Evaluation (OPRE) on Twitter or Facebook and use the #PathwaysClearinghouse hashtag to launch a conversation.

Use the intervention search tool to find the most up-to-date information on the evidence about helping youth and adults with low incomes succeed in the labor force. Begin by going to Find Interventions that Work, and then use the sorting options to show interventions that are rated  well-supported or supported by domain at the top of the search results. 

The Pathways Clearinghouse welcomes submissions of research studies that have evaluated a particular employment and training intervention. Researchers or others interested in having the Clearinghouse review studies of specific interventions may submit the intervention name and its associated citations or research studies to PathwaysClearinghouse@abtassoc.com. (See the question below: How do I submit my research for consideration?) You can also submit developmental interventions, which have research of their effectiveness underway but not yet completed. The team will log all recommendations. Those interested in submitting such studies should note, however, that the Pathways Clearinghouse does not review interventions on a rolling basis. Instead, we review interventions periodically, as resources allow. As a result, interventions submitted to the Pathways Clearinghouse will not be reviewed immediately upon request. In addition, submitting a recommendation does not guarantee that we will review studies of the intervention.

The Pathways Clearinghouse team uses prespecified keywords to systematically search databases of journal publications, evaluation reports, and unpublished literature (such as working papers). We also incorporate studies cited in literature reviews and those submitted in response to a public call for relevant studies. We screen studies for their eligibility to be included in the review against a set of predefined criteria. Then, we prioritize the eligible studies based on publication date (examining the most recent first and turning to older research as resources allow). For more information, please see the Protocol for the Pathways to Work Evidence Clearinghouse.

Occasionally, the Pathways Clearinghouse issues a call for studies and invites the public to submit research. When open, we will post a link to more information on the Pathways Clearinghouse home page and will circulate the call to users who subscribe to our email list. Interested stakeholders can submit research at any time to PathwaysClearinghouse@abtassoc.com, even when no call for studies is posted, and the team will log the submission for consideration alongside submissions from the next call for studies. Submitting research does not guarantee that the study will be reviewed. We decide which research to review using the systematic process outlined in the Protocol for the Pathways to Work Evidence Clearinghouse.

The Pathways Clearinghouse Quality Review Team (QRT) handles any challenges stakeholders make about a review’s findings, the inclusion of a study within the Pathways Clearinghouse, or other individual judgements the Pathways Clearinghouse team makes. The QRT addresses any issues with reviews that stakeholders raise, so long as they are (1) submitted in writing to PathwaysClearinghouse@abtassoc.com, (2) related to a specific study or well-defined set of studies, and (3) coherently explained (and the inquirer is available to answer any clarifying questions).

When a request is submitted to the QRT, a team member first verifies the request meets the criteria listed above. After this confirmation, the team member examines the study and any related materials, discusses the review with the original study reviewers, and presents a summary of the review and any potential flaws to the QRT. The QRT then determines whether the initial review should be revised, notifies OPRE and the inquirer of its findings and, if necessary, edits any Clearinghouse products to reflect the updated review.

Developmental interventions have a rigorous impact evaluation underway, but no available findings at the time that Pathways Clearinghouse identifies them. Nominations for developmental interventions can come from federal staff, evaluators, and Pathways Clearinghouse stakeholders. If the evaluations meet the eligibility criteria established in the Protocol for the Pathways to Work Clearinghouse: Methods and Standards, they are added to the Developmental interventions page. These evaluations need to be randomized controlled trials or comparison- group quasi-experimental designs of an employment or training intervention in the U.S. or Canada that focuses on populations with low-incomes.

It is important to note that the Pathways Clearinghouse is not a registry for impact evaluations and submitting evaluations does not count as registering research. If you have a developmental intervention to share with the Pathways Clearinghouse, please email the name of the intervention, a brief description, and links to any additional publicly available materials (such as websites or reports) to PathwaysClearinghouse@abtassoc.com.

For the Pathways Clearinghouse, the effects shown are the estimated changes in the percent of low-income adults who are employed, average annual earnings, average annual public benefits received, and percent of low-income adults with any education and training credential. The Pathways Clearinghouse takes each effect and estimates comparable effects in 2018 dollars and percentages by calculating the average impact of the intervention in standard deviation units (Hedges’ g effect sizes) across outcomes and studies and comparing that average to the distributions of outcomes for adults with low earnings potential in the 2019 Current Population Survey Annual Social and Economic Supplement. The impacts in standard deviation units are also combined into domain average effects by taking the average across studies and outcomes, giving more weight to studies with larger sample sizes. These averages are then also converted into 2018 dollars and percentages.

An effectiveness rating is the assessment of the Pathways Clearinghouse, based on the existing evidence from impact studies, of the extent to which a given intervention improves a specific type of outcome. The effectiveness rating is a holistic assessment of whether an intervention is likely to produce favorable results if faithfully replicated with a similar population. An effect size is a standardized measure that allows us to make direct and meaningful comparisons across different outcomes, settings, and interventions. Both the effectiveness rating and the effect size can be compared across studies in the Pathways Clearinghouse. The effect size is the impact found in the available research, and the effectiveness rating is the likelihood that an intervention would produce a similar, favorable effect if implemented again with a similar population and context.

An effectiveness rating assesses whether an intervention is likely to produce favorable results if faithfully replicated with a similar population. Outcome domains with well-supported ratings are those that the evidence suggests are most likely to improve if an intervention were replicated with a similar population. Outcome domains with supported ratings have some evidence that the intervention improves them, but the evidence is less conclusive. Outcome domains that receive a rating of not supported have strong or consistent evidence that the intervention is unlikely to produce favorable effects.

However, because implementation challenges and successes often vary and no two implementations of an intervention are identical, the well-supported and supported ratings do not guarantee success.

The four domains for which the Clearinghouse rates effectiveness are earnings, employment, public benefit receipt, and education and training. Within the first three of these domains, the Pathways Clearinghouse reports on short-term, long-term, and very-long term outcomes [the education and training findings are reported at the longest follow-up, and not segmented into these three time periods]. To make the intervention search results display easier to view and navigate, the effectiveness ratings on the search page represent the highest rating given to the short-term, long-term or very-long term outcomes for that intervention. For example, if an intervention has a supported effectiveness rating in the long-term for earnings, but not in the short-term or very-long term, we will display the supported icon for the earnings domain. Users can click on the individual interventions to see whether the effectiveness ratings apply to short-term, long-term, or very-long term outcomes.

The Pathways Clearinghouse’s ability to report on the effects of an intervention are tied to the existing evidence. Not all interventions have been studied along all the domains on which we report [earnings, employment, public benefit receipt, and education and training]. An intervention may have studies that examined effects in some outcomes but not in others. In other cases, the quality of evidence may vary across outcome domains. If we did not find any studies that rated moderate or high quality that studied the intervention’s effect on outcomes in a given outcome domain, that outcome receive an effectiveness rating of no evidence to assess support. In addition, we may not have all of the data to extract an effect size from the original study (as discussed in the previous FAQ). In those cases, we cannot report on the effects on an intervention for those domains.

Very long-term outcomes are those measured 5 years or more after participants in the study’s intervention group are first offered services.

This has to do with the difference between how we calculate whether or not an intervention has evidence of being effective on a given outcome domain and how we calculate the size of an intervention’s effects on that outcome domain. In order to receive a supported rating, an intervention must have at least one statistically significant, favorable finding and no statistically significant unfavorable findings in the given outcome domain. Effect sizes, on the other hand, are an average of all findings for a given outcome domain, including those that are not statistically significant.

Take, for example, a study that finds three effects on earnings, one of which is statistically significant, and favorable, and two of which are statistically insignificant, but unfavorable. Because the study identified a statistically significant, favorable effect on earnings and no statistically significant unfavorable findings, the intervention receives an effectiveness rating of supported on earnings. In calculating its overall effect on earnings, however, we average all the findings in this domain, the two unfavorable, but statistically insignificant findings, along with the statistically significant, favorable effect. The average of these three findings might result in an overall negative effect on earnings.

The Pathways Clearinghouse reports implementation details for interventions with at least one outcome domain that has a well-supported or supported rating across the 10 outcome domains (increasing earnings, increasing employment, or decreasing benefit receipt in the short, long, or very long term, or increasing education and training). We focus on well-implemented interventions with supported outcomes because practitioners and decision makers might be interested in learning more about how to implement these interventions.

The Pathways Clearinghouse identifies and reviews relevant studies, including impact, descriptive, and implementation studies, to collect details on how an intervention was implemented. We summarize information such as the amount of services received, the number of staff delivering services, and costs and benefits. For each intervention summary, we identify a key informant, who is usually the lead author on one or more of the studies. We share the draft implementation summary with the key informant to ensure accuracy and clarity. During their review, the key informant might provide additional information or suggest edits. Therefore, although the majority of information comes from the studies cited on the study pages associated with each intervention, some of the details come directly from exchanges with a key informant.

The Pathways Clearinghouse provides cost information for interventions that have at least one supported or well-supported effectiveness rating. Cost information is located on the details page for an intervention, the page which supplies more context about interventions with favorable outcomes. All cost information reported on the website is based on data reported by study authors. We first convert these figures to their present value at study baseline (that is, at the time intervention services began, or at the time of random assignment for interventions studied using randomized controlled trial designs). Specifically, we discount costs that occurred over multiple years to their value at baseline because the value of a dollar is greater in the present than in the future. We assume a real rate of return of 3 percent annually. We then report those costs in 2018 dollars. Where there are multiple studies of an intervention rated high or moderate quality, we compute the simple average of costs reported across those studies, and the cost figure we report reflects the average set of intervention services provided.

Because the cost information is based on data reported by study authors, cost information is not directly comparable across interventions. The types of costs reported across interventions vary, as does the amount of time that interventions lasted. The Pathways Clearinghouse does not factor information on cost into the effectiveness ratings displayed on the website. However, the cost details provide a useful starting point for thinking about the costs to organizations that implemented the interventions.

The Pathways Clearinghouse estimates effect sizes for each finding with a high or moderate rating when the study or study authors provide sufficient information to calculate an effect size. The Pathways Clearinghouse team contacts authors to obtain the information necessary to calculate the effect size, but in some cases, sufficient information is not available.

Studies with low quality ratings demonstrate little evidence that findings are attributable, in part or in full, to the intervention examined. A low quality rating suggests that there is a high risk of bias. The Pathways Clearinghouse focuses on providing detailed information for studies rated high or moderate quality because these studies have a lower risk of bias, and these interventions are more likely to have contributed to the reported outcomes. In other words, the Pathways Clearinghouse focuses on well-implemented studies because they provide the most relevant, useful information for practitioners and decision makers.

Whenever possible, the Pathways Clearinghouse follows federal standards for classifying data on race and ethnicity. For some studies—in particular, studies published before 2000—Pathways Clearinghouse uses earlier classifications of race and ethnicity to be consistent with how studies tended to report this information at the time. Pathways Clearinghouse may use either set of race and ethnicity terms, depending largely on when a study was published. There are three key differences between the two classification systems:

1. Some earlier studies asked individuals to select both their ethnicity (Hispanic) and their race (White, Black, Asian, and so on). In practice, this means that, in these older studies, an individual who self-identifies as both Hispanic and White could be identified in the data as both Hispanic and White. By comparison, under current standards the same individual would be identified only as “Hispanic or Latino”.
2. The language used to describe certain racial classifications has shifted over time. For example, some earlier studies might use the term Pacific Islander, whereas later studies use Native Hawaiian or Other Pacific Islander.
3. For some earlier studies, Pathways Clearinghouse combined into a single category (1) the cases in which race was unknown or not reported by the study author, and (2) cases in which another category was used that is not aligned with the Pathways Clearinghouse categories. For example, a study author might present a population that is 40 percent White, 40 percent Black, 10 percent Middle Eastern or North African (MENA), and 10 percent unknown. In this scenario, the authors use a race classification, MENA, that is not clearly aligned with a category in the current federal standards. If this study was described using the older race and ethnicity classifications, then the Pathways Clearinghouse would combine the percentage of individuals who identified as MENA with those whose race was unknown, reporting that the study sample was 40 percent White, 40 percent Black, and 20 percent unknown, not reported, or another race. If the study sample were described using the current standards, by contrast, the Pathways Clearinghouse would classify the sample as 40 percent White, 40 percent Black, 10 percent another race, and 10 percent unknown or not reported. The move towards disaggregation is intended to provide practitioners with the most complete information about the demographics of the population served by a given intervention.

The Pathways Clearinghouse strives to present a complete set of race and ethnicity data for each study, in accordance with the federal standards for classifying data on race and ethnicity. In practice, for most studies this means that an individual can fall only into single race or ethnicity category. For example, a person would identify as either Hispanic or Black, but not both. Because of this, most studies present race and ethnicity data that sums to 100 percent. In some cases, however, study authors treated race and ethnicity as non-exclusive categories. For example, an individual could self-identify both as Hispanic (ethnicity) and Black (race). In cases where individuals report on race and ethnicity separately, the sum total for race and ethnicity demographics might be greater than 100 percent.

Individuals within the study sample could fall into these categories because they chose not to identify their race, because the authors did not collect or report race and ethnicity data, or because the individual identified with a race other than one of the federal categories. Wherever possible, the Pathways Clearinghouse differentiates between cases where race and ethnicity were unknown or not reported, and cases where race and ethnicity were reported but fell into a different category. However, a number of earlier Pathways Clearinghouse reviews of studies published before 2000 did not differentiate between unknown, not reported, and other cases, instead reporting these as a single category.

Most of the studies reviewed by the Pathways Clearinghouse report information using the biological sex categories of male and female. Fewer studies report on gender demographics, including but not limited to transgender and nonbinary-inclusive gender categories. In order to present information consistently across all studies, the Pathways Clearinghouse has, at this time, limited itself to presenting biological sex categories rather than gender.

The Pathways Clearinghouse reports on the effects for study findings that are rated high or moderate quality in our four domains [earnings, employment, public benefit receipt, and education and training]. When the quality of a study is high, that means we can be fairly confident in the study findings because the study finding is solely attributable to the intervention examined. This rating is reserved for study findings from high quality RCTs with low attrition of sample members. The Pathways Clearinghouse also reports effects for moderate quality studies, where we can be somewhat confident in the study findings.

There are several reasons that some findings in a study may not be rated. Specifically:

• Studies may have outcomes that are rated low quality, where we cannot have much confidence in the study findings. Other important factors could have influenced the study findings, and the study did not account for them. The Pathways Clearinghouse does not report outcomes that are rated low quality.
• If the original study provides findings for multiple outcome measures in a given domain, the Pathways Clearinghouse prioritizes findings to review and report based on the outcome measure, following the prioritization process summarized in Exhibit III.2 of the Protocol for the Pathways to Work Clearinghouse: Methods and Standards. For example, if study authors used both surveys and administrative records to assess earnings, Pathways Clearinghouse reviewers select two sets of earnings findings for review: one measured using survey data and one measured using administrative data.

In addition, the Pathways Clearinghouse may not have sufficient information to report on the magnitude of study findings. We calculate an effect size using study-specific data, if we can obtain from the study authors the information needed to do so.

Begin by going to Find Interventions that Work, and use the "Select the settings" filter on the left under the heading "Limit results to interventions implemented: In urban or rural settings."

If you select "Rural only", the search will return interventions that have only been tested in rural settings plus those that have been tested in multiple studies and at least one of those studies occurred in an exclusively rural setting.

However, consider expanding your search to include interventions that were tested in multiple settings; a category that can include rural settings. Some interventions may have been tested and implemented in rural settings, but the study findings were not disaggregated by rural versus non-rural settings, so they will not be included in the "Rural only" search results. For example, a study may have tested an intervention throughout the state of Texas, which includes rural, urban, and suburban areas. To capture interventions like these, expand your search to include "Tested in multiple settings." This will provide a more comprehensive list of interventions relevant to rural settings.