How does the Pathways Clearinghouse choose research for review?
The Pathways Clearinghouse team uses prespecified keywords to systematically search databases of journal publications, evaluation reports, and unpublished literature (such as working papers). We also incorporate studies cited in literature reviews and those submitted in response to a public call for relevant studies. We screen studies for their eligibility to be included in the review against a set of predefined criteria. Then, we prioritize the eligible studies based on publication date (examining the most recent first and turning to older research as resources allow). For more information, please see the Protocol for the Pathways to Work Evidence Clearinghouse.
How do I submit an employment and training intervention for consideration?
The Pathways Clearinghouse welcomes submissions of research studies that have evaluated a particular employment and training intervention. Researchers or others interested in having the Clearinghouse review studies of specific interventions may submit the intervention name and its associated citations or research studies to PathwaysClearinghouse@Mathematica-MPR.com. (See the question below: How do I submit my research for consideration?) You can also submit developmental interventions, which have research of their effectiveness underway but not yet completed, and case studies of interventions using new and promising practices with no research of their effectiveness. The team will log all recommendations. Those interested in submitting such studies should note, however, that the Pathways Clearinghouse does not review interventions on a rolling basis. Instead, we review interventions periodically, as resources allow. As a result, interventions submitted to the Pathways Clearinghouse will not be reviewed immediately upon request. In addition, submitting a recommendation does not guarantee that we will review studies of the intervention.
How do I submit my research for consideration?
Occasionally, the Pathways Clearinghouse issues a call for studies and invites the public to submit research. When open, we will post a link to more information on the Pathways Clearinghouse home page and will circulate the call to users who subscribe to our email list. Interested stakeholders can submit research at any time to PathwaysClearinghouse@Mathematica-MPR.com, even when no call for studies is posted, and the team will log the submission for consideration alongside submissions from the next call for studies. Submitting research does not guarantee that the study will be reviewed. We decide which research to review using the systematic process outlined in the Protocol for the Pathways to Work Evidence Clearinghouse.
How can I stay up to date on new information released by the Pathways Clearinghouse?
We will send occasional mailings to users who subscribe to our email list, alerting them to new website content and open calls for studies. For real-time updates on Pathways Clearinghouse activities, events, and presentations, follow the Office of Planning, Research, and Evaluation (OPRE) on Twitter or Facebook and use the #PathwaysClearinghouse hashtag to launch a conversation. You can also follow Mathematica using @MathematicaNow across all social media platforms.
Can I appeal the rating that the Pathways Clearinghouse has applied to my study?
The Pathways Clearinghouse Quality Review Team (QRT) handles any challenges stakeholders make about a review’s findings, the inclusion of a study within the Pathways Clearinghouse, or other individual judgements the Pathways Clearinghouse team makes. The QRT addresses any issues with reviews that stakeholders raise, so long as they are (1) submitted in writing to PathwaysClearinghouse@Mathematica-MPR.com, (2) related to a specific study or well-defined set of studies, and (3) coherently explained (and the inquirer is available to answer any clarifying questions).
When a request is submitted to the QRT, a team member first verifies the request meets the criteria listed above. After this confirmation, the team member examines the study and any related materials, discusses the review with the original study reviewers, and presents a summary of the review and any potential flaws to the QRT. The QRT then determines whether the initial review should be revised, notifies OPRE and the inquirer of its findings and, if necessary, edits any Clearinghouse products to reflect the updated review.
Where can I find a list of well-supported or supported interventions?
Use the intervention search tool find the most up-to-date information on the evidence about helping youth and adults with low incomes succeed in the labor force. Begin by going to the Advanced Intervention Search, and then use the filtering options on the left to filter by well-supported or supported ratings in any outcome domain. This will display interventions that are rated well-supported or supported in at least one outcome domain.
The intervention search results show one effectiveness rating per domain, but when I click on the individual intervention, many domains have more than one rating. What does the effectiveness rating by domain on the intervention search page mean?
The four domains for which the Clearinghouse rates effectiveness are earnings, employment, public benefit receipt, and education and training. Within the first three of these domains, the Pathways Clearinghouse reports on short-term, long-term, and very-long term outcomes [the education and training findings are reported at the longest follow-up, and not segmented into these three time periods]. To make the intervention search results display easier to view and navigate, the effectiveness ratings on the search page represent the highest rating given to the short-term, long-term or very-long term outcomes for that intervention. For example, if an intervention has a supported effectiveness rating in the long-term for earnings, but not in the short-term or very-long term, we will display the supported icon for the earnings domain. Users can click on the individual interventions to see whether the effectiveness ratings apply to short-term, long-term, or very-long term outcomes.
What are very long-term outcomes?
Very long-term outcomes are those measured 5 years or more after participants in the study’s intervention group are first offered services.
What is the difference between the Pathways Clearinghouse and the Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR)?
The Pathways Clearinghouse, an investment of the Office of Planning, Research, and Evaluation in the Administration for Children and Families within the Department of Health and Human Services, identifies interventions that aim to improve employment outcomes, reduce employment challenges, and support self-sufficiency for populations who are low income, especially recipients for public programs such as TANF. The Pathways Clearinghouse was designed for use by direct employment service providers and TANF administrators. Potential users also include policymakers and researchers.
The Department of Labor’s Clearinghouse for Labor Research and Evaluation (CLEAR)’s mission is to make research on labor topics more accessible to practitioners, policymakers, researchers, and the public more broadly so that it can inform their decisions about labor policies and programs. CLEAR focuses a range of labor policies and programs including on workforce system, workforce training, unemployment insurance, worker benefits, and retirement.
These Clearinghouses are coordinated federal efforts given the overlap in content. While the scope of the Pathways Clearinghouse also includes research eligible for review by CLEAR, Pathways specifically focuses on programs designed to meet the needs of populations who are low income or who face chronic barriers to employment. Furthermore, Pathways focuses on summarizing all research within an intervention (which CLEAR does not) as well as across interventions that use similar strategies (which both Clearinghouses aim to do).
How does the Pathways Clearinghouse estimate the effects of an intervention?
For the Pathways Clearinghouse, the effects shown are the estimated changes in the percent of low-income adults who are employed, average annual earnings, average annual public benefits received, and percent of low-income adults with any education and training credential. The Pathways Clearinghouse takes each effect and estimates comparable effects in 2018 dollars and percentages by calculating the average impact of the intervention in standard deviation units (Hedges’ g effect sizes) across outcomes and studies and comparing that average to the distributions of outcomes for adults with low earnings potential in the 2019 Current Population Survey Annual Social and Economic Supplement. The impacts in standard deviation units are also combined into domain average effects by taking the average across studies and outcomes, giving more weight to studies with larger sample sizes. These averages are then also converted into 2018 dollars and percentages.
Why does the Pathways Clearinghouse only report the effects for some study findings?
The Pathways Clearinghouse reports on the effects for study findings that are rated high or moderate quality in our four domains [earnings, employment, public benefit receipt, and education and training]. When the quality of a study is high, that means we can be fairly confident in the study findings because the study finding is solely attributable to the intervention examined. This rating is reserved for study findings from high quality RCTs with low attrition of sample members. The Pathways Clearinghouse also reports effects for moderate quality studies, where we can be somewhat confident in the study findings.
There are several reasons that some findings in a study may not be rated. Specifically:
- Studies may have outcomes that are rated low quality, where we cannot have much confidence in the study findings. Other important factors could have influenced the study findings, and the study did not account for them. The Pathways Clearinghouse does not report outcomes that are rated low quality.
- If the original study provides findings for multiple outcome measures in a given domain, the Pathways Clearinghouse prioritizes findings to review and report based on the outcome measure, following the prioritization process summarized in Exhibit III.2 of the Protocol for the Pathways to Work Clearinghouse: Methods and Standards. For example, if study authors used both surveys and administrative records to assess earnings, Pathways Clearinghouse reviewers select two sets of earnings findings for review: one measured using survey data and one measured using administrative data.
In addition, the Pathways Clearinghouse may not have sufficient information to report on the magnitude of study findings. We calculate an effect size using study-specific data, if we can obtain from the study authors the information needed to do so.
The Pathways Clearinghouse reports the effects of an intervention for earnings, employment, public benefit receipt, and education and training. Why do some interventions lack effectiveness ratings or effects for some of those domains?
The Pathways Clearinghouse’s ability to report on the effects of an intervention are tied to the existing evidence. Not all interventions have been studied along all the domains on which we report [earnings, employment, public benefit receipt, and education and training]. An intervention may have studies that examined effects in some outcomes but not in others. In other cases, the quality of evidence may vary across outcome domains. If we did not find any studies that rated moderate or high quality that studied the intervention’s effect on outcomes in a given outcome domain, that outcome receive an effectiveness rating of no evidence to assess support. In addition, we may not have all of the data to extract an effect size from the original study (as discussed in the previous FAQ). In those cases, we cannot report on the effects on an intervention for those domains.
How does the Pathways Clearinghouse identify case study interventions?
Case studies are interventions that are innovative, noteworthy, and currently in use, but that lack a rigorous evaluation. Case studies are identified by nominations from federal staff, or Pathways Clearinghouse stakeholders. In order to be considered as a case study, interventions need to meet the following criteria:
- Have no rigorous evaluation
- Focus on employment, education, and/or training strategies
- Target populations with low incomes, such as Temporary Assistance for Needy Families (TANF) recipients, beneficiaries of other public benefit programs, disconnected youth, formerly incarcerated individuals, or non-custodial parents in the child support system that have relatively lower incomes
- Represent an innovative approach
- Have an explicit theory of change
- Have promising, preliminary outcomes
- Have enough publicly available information for the Pathways Clearinghouse team to summarize the intervention in the case study
If you have an intervention that the Pathways Clearinghouse should consider for a case study, please email the name of the intervention, a brief description, and links to any additional publicly available materials (such as websites or reports) to PathwaysClearinghouse@mathematica-mpr.com.
How does the Pathways Clearinghouse identify developmental interventions?
Developmental interventions have a rigorous impact evaluation underway, but no available findings at the time that Pathways Clearinghouse identifies them. Nominations for developmental interventions can come from federal staff, evaluators, and Pathways Clearinghouse stakeholders. If the evaluations meet the eligibility criteria established in the Protocol for the Pathways to Work Clearinghouse: Methods and Standards, they are added to the Developmental interventions page. These evaluations need to be randomized controlled trials or comparison- group quasi-experimental designs of an employment or training intervention in the U.S. or Canada that focuses on populations with low-incomes.
It is important to note that the Pathways Clearinghouse is not a registry for impact evaluations and submitting evaluations does not count as registering research. If you have a developmental intervention to share with the Pathways Clearinghouse, please email the name of the intervention, a brief description, and links to any additional publicly available materials (such as websites or reports) to PathwaysClearinghouse@mathematica-mpr.com.