View definitions of terms used throughout the Pathways Clearinghouse.

A

Apprenticeships

An organized or structured form of learning on the job, typically in a skilled trade, but typically not subsidized.

Attrition

After individuals in a study are randomly assigned to the group receiving the intervention (the intervention group) or the group not receiving the intervention (the comparison group), they might drop out of an intervention or the study team might lose contact with them. This means information about how they fare over time is no longer available for use in the in the study. This dropping out is called attrition. The proportion of people who drop out of a study is called the attrition rate. There are two different types of attrition that matter when deciding whether a study’s findings can be considered reliable. The first is overall attrition, which captures the total number of individuals missing from the original sample. The second is differential attrition which captures the difference between the percentage of individuals missing from the intervention group and the percentage missing from the comparison group. When either the overall attrition or the differential attrition is too high, the study results might not accurately capture the effects of the intervention. We use scientific standards to determine when either of these types of attrition is too high for a study’s findings to be considered reliable.

B

Baseline equivalence

A study’s ability to demonstrate that the group receiving the intervention (the intervention group) and the group not receiving the intervention (the comparison group) are similar to each other before the intervention begins is a key factor for determining the reliability of a study’s findings. Demonstrating that the two groups are similar according to important characteristics (such as race, ethnicity, gender, and time out of work) is called demonstrating baseline equivalence. We assess studies for how well they capture these kinds of characteristics about members of each group at baseline (that is, before intervention group members received intervention services) and how well they are able to show that the groups are largely similar across these key characteristics.

C

Case management

Meeting, typically one-on-one, with an employment specialist or counselor who helps assess needs and refers clients to other available services. Case management can take place before or during employment and could focus on employment or on mental health or substance abuse.

Case study

On the Pathways Clearinghouse site, case studies are brief summaries of noteworthy or emerging interventions for which there is neither planned nor completed research on their effectiveness. The interventions themselves might or might not be effective; no research yet exists with which to assess their effectiveness. Instead, the Clearinghouse captures descriptive information about these interventions such as how, by whom, and where an intervention is delivered, and anecdotal statements about the intervention’s outcomes.

Comparison group

A group with characteristics similar to those of intervention group members, except that those in the comparison group do not have an opportunity to receive the services of interest. The comparison group is intended to represent what would have happened to members of the intervention group if they had not been offered the services from the intervention of interest.

Confounding factor

A factor that might affect how well an intervention works and that applies differently to the intervention and comparison groups. Because this factor can affect one group and not the other, the presence of a confounding factor causes us to question a study’s findings. One type of confounding factor is an element external to the intervention that reaches only the members of one study group—for instance, if all members of the intervention group lived in one state and all members of the comparison group lived in another state. In this case, it would be impossible to separate the effect of the program or policy from that of local economic conditions.

D

Decrease long-term public benefit receipt

Reduction in the percentage of people receiving public benefits or the value of their benefits between 18 months and 5 years after participants are offered intervention services.

Decrease short-term public benefit receipt

Reduction in the percentage of people receiving public benefits (such as TANF, Supplemental Nutrition Assistance Program, and so on) or the value of their benefits during a period 18 months or fewer after the participant first received intervention services.

Decrease very long-term benefit receipt

Reduction in the percentage of people receiving public benefits or the value of their benefits more than 5 years after participants are offered intervention services.

Developmental intervention

Interventions that, to date, have no available research findings regarding their effectiveness, but for which research studying their effectiveness is underway. The effectiveness rating applies to the entire intervention and not to specific outcome domains.

E

Education

Services to support educational attainment, such as GED support, adult basic education, or post-secondary education.

Effect Size

A standardized measure of the magnitude of the impact of the intervention, or the difference in outcomes between the intervention group and the comparison group. The effect size is calculated by dividing the impact shown in the study by the standard deviation for that measure, which measures the diversity of the study sample. We can compare effect sizes for different outcomes, different settings, and different interventions.

Effectiveness rating

The assessment of the Pathways Clearinghouse, based on the existing evidence from impact studies, of the extent to which a given intervention improves a specific type of outcome. The effectiveness rating indicates whether the intervention is likely to produce favorable results if faithfully replicated with a similar population. After we review research on an intervention, we assign an effectiveness ratings to indicate how effective the intervention is at improving each of four types of outcomes: employment, earnings, public benefit receipt, and education and training. The ratings depend on (1) the quality of the impact study or studies conducted of the intervention and (2) the favorability (or lack thereof), statistical significance, and consistency of the study findings for that type of labor market outcome.

  • Well-supported  Well-supported. We have strong and consistent evidence that the intervention produces favorable results for a specific outcome domain, such as short-term earnings. These interventions have at least two impact studies of moderate or high quality that show evidence of favorable findings within the domain. However, because implementation challenges and successes often vary, and because no two implementations of an intervention are identical, Pathways Clearinghouse users should not view this rating as a guarantee of future success.
  • Supported  Supported. We have some evidence that the intervention improves outcomes. These ratings are domain specific, meaning that the intervention is considered supported only for the particular domains for which we have given this rating. These interventions have at least one study of moderate or high quality and show evidence of favorable findings in the domain, but the evidence is less conclusive than that for evidence-based interventions.
  • Not supported  Not supported. We have the strongest evidence that the intervention is unlikely to produce substantial favorable results in a given outcome domain. Studies of these interventions have found only a pattern of null and/or unfavorable findings. We only consider impact studies of at least moderate quality in determining this rating.
  • Mixed support  Mixed support. We have some evidence, from impact studies of moderate or high quality, that indicates the intervention improves outcomes in a given outcome domain, and some evidence that indicates it worsens outcomes in that domain. For instance, in the educational attainment domain, an intervention might have unfavorable effects on attainment of a high school diploma, but favorable effects on GED attainment.
  • Insufficient evidence  Little evidence to assess support. We have some research, from impact studies of moderate or high quality, on the intervention’s effect in a given outcome domain. But we do not have a sufficient body of evidence to assign one of the other ratings.
  • No evidence  No evidence to assess support. We did not find any studies that rated moderate or high that studied the intervention’s effect on outcomes in a given outcome domain. These interventions need further study to support conclusions about their effectiveness.
Employment coaching

Intensive assistance with identifying barriers and goals and helping clients address them. Also known as life coaching.

Employment retention services

Supplementary services provided when a client already has a job. These could include ongoing case management to address barriers or to assess progress toward career goals.

F

Financial education

Education that help individuals make informed decisions about their financial resources, such as providing information on budgeting or loans.

Financial incentives

Bonuses that clients receive for engaging in a specific activity or achieving a certain goal.

Finding

A finding summarizes the effect of an intervention on an outcome measure related to employment, earnings, public benefit receipt, education, or training. It is the smallest element we review and to which we give a study quality by finding rating. The direction of the finding might be favorable or unfavorable. We also categorize whether it is statistically significant (unlikely to have occurred by chance) and small or moderate-to-large.

  • Favorable. An impact on an outcome or an overall outcome domain in a direction that is socially desirable. For example, a favorable impact could be an increase in annual earnings or consecutive months of employment, or a reduction in months of Temporary Assistance for Needy Families (TANF) eligibility.
  • Unfavorable. An impact on an outcome or an overall outcome domain in a direction that is not socially desirable. For example, an unfavorable impact could be a decrease in annual earnings or consecutive months of employment, or an increase in months of TANF eligibility.
  • Moderate-to-large. We describe findings as being moderate-to-large if the intervention produced a sufficiently large change in the outcome. The change can be in a favorable or unfavorable direction. An impact receives this designation if the standardized effect size is equal to or greater than 0.25 or equal to or less than -0.25.
  • Small. We describe findings as being small if the intervention did not produce a large change in the outcome. The change can be in a favorable or unfavorable direction. An impact receives this designation if the standardized effect size is between -0.25 and 0.25.

H

Health services

Services to support the mental or physical health of clients.

High

The quality of the study is high, meaning we can be fairly confident in the study findings. Strong evidence shows that the study finding is solely attributable to the intervention examined. This rating is reserved for study findings from high quality RCTs with low attrition of sample members.

I

Increase education and training

Attainment of a degree or credential.

Increase long-term earnings

An increase in the amount earned through paid employment between 18 months and 5 years after the participant first received intervention services.

Increase long-term employment

An increase in the rate of employment between 18 months and 5 years after the participant first received intervention services.

Increase short-term earnings

An increase in the amount earned through paid employment during a period 18 months or fewer after the participant first received intervention services.

Increase short-term employment

An increase in the rate of employment during a period 18 months or fewer after the participant first received intervention services.

Increase very long-term earnings

An increase in the amount earned through paid employment more than 5 years after the participant first received intervention services.

Increase very long-term employment

An increase in the rate of employment more than 5 years after the participant first received intervention services.

Intervention

A specific bundle of services or policies implemented in a given context. For the Pathways Clearinghouse, interventions are defined based on the services offered to the intervention group but not offered to the comparison group. Two studies examine the same intervention only if the same services were offered in both cases. We use the following definitions for the services and policies included in the interventions we review.

  • Case management. Meeting, typically one-on-one, with an employment specialist or counselor who helps assess needs and refers clients to other available services. Case management can take place before or during employment and could focus on employment or on mental health or substance abuse.
  • Financial incentives. Bonuses that clients receive for engaging in a specific activity or achieving a certain goal.
  • Financial education. Education that help individuals make informed decisions about their financial resources, such as providing information on budgeting or loans.
  • Health services. Services to support the mental or physical health of clients.
    • Substance use disorder treatment and mental health services. Services to treat clients for substance use disorder or mental health diagnoses.
    • Physical health services. Services to address clients’ physical health concerns.
  • Employment retention services. Supplementary services provided when a client already has a job. These could include ongoing case management to address barriers or to assess progress toward career goals.
  • Pre-employment services. Services designed to help job seekers find a job that are not related to education or training. These can include initial assessments to identify employment barriers, formalized assessments to identify skills and interests, help designing a resume and cover letter, job search assistance, or help developing an individual employment plan.
    • Coaching. Intensive assistance with identifying barriers and goals and helping clients address them. Also known as life coaching.
    • Job development or job placement. Assistance getting placed in a job. Typically, a client visits a career center and meets with a counselor who works with employers to identify or create a specific opening for the client.
  • Sanctions. Reductions in payment for failing to comply with mandated services.
  • Training. Any training program.
    • Soft skills training. Training in so-called soft skills, such as punctuality, manners, professional dress, interactions with colleagues, or conflict management. Sometimes also called life skills training.
    • Occupational or sectoral training. Training that is tied to a particular occupation, such as truck driving or welding.
    • On-the-job training. An agreement between the workforce system and an employer in which the workforce system pays all or part of the wages for a client working for an approved employer in an approved occupation for a specified period. At the end of that time, the employer can hire the worker but without the wage subsidy.
    • Apprenticeships. An organized or structured form of learning on the job, typically in a skilled trade, but typically not subsidized.
  • Work experience. Paid or unpaid (such as internships) work experience.
    • Unpaid work experience. Work experience that is voluntary or unpaid, such as an unpaid internship.
    • Subsidized employment. Employment that is partially or fully paid for by an external funder (not the employer).
    • Transitional jobs. Jobs that are meant to integrate those who have been out of the workforce (for example, former prisoners) into the community. They can be paid or unpaid.

J

Job development or job placement

Assistance getting placed in a job. Typically, a client visits a career center and meets with a counselor who works with employers to identify or create a specific opening for the client.

Job search assistance

Assistance identifying potential jobs and preparing resumes and cover letters.

L

Little evidence to assess support

We have some research, from impact studies of moderate or high quality, on the intervention’s effect in a given outcome domain. But we do not have a sufficient body of evidence to assign one of the other ratings.

Long-term

For this review, a period lasting between 18 months and 5 years after participants are first offered services.

Low

The quality of the study is low, meaning we cannot have much confidence in the study findings. Other important factors could have influenced the study findings, and the study did not account for them. Study findings that do not meet the high or moderate ratings criteria receive the low rating.

M

Mixed support

We have some evidence, from impact studies of moderate or high quality, that indicates the intervention improves outcomes in a given outcome domain, and some evidence that indicates it worsens outcomes in that domain. For instance, in the educational attainment domain, an intervention might have unfavorable effects on attainment of a high school diploma, but favorable effects on GED attainment.

Moderate

The quality of the study is moderate, meaning we can be somewhat confident in the study findings. However, other factors not accounted for in the study might also have contributed to the findings. This applies to findings from random assignment studies that, because of flaws in the study design or analysis (for example, high sample attrition), do not meet the criteria for the high rating but satisfy other design criteria. We also assign the moderate rating to study findings from well-executed QEDs.

N

No evidence to assess support

We did not find any studies that rated moderate or high that studied the intervention’s effect on outcomes in a given outcome domain. These interventions need further study to support conclusions about their effectiveness.

Not supported

We have the strongest evidence that the intervention is unlikely to produce substantial favorable results in a given outcome domain. Studies of these interventions have found only a pattern of null and/or unfavorable findings. We only consider impact studies of at least moderate quality in determining this rating.

O

Occupational or sectoral training

Training that is tied to a particular occupation, such as truck driving or welding.

On-the-job training

An agreement between the workforce system and an employer in which the workforce system pays all or part of the wages for a client working for an approved employer in an approved occupation for a specified period. At the end of that time, the employer can hire the worker but without the wage subsidy.

Outcome domain

A group of related outcomes. The Pathways Clearinghouse includes seven outcome domains: short-term earnings, long-term earnings, short-term employment, long-term employment, short-term public benefit receipt (such as TANF, Supplemental Nutrition Assistance Program, and so on), long-term public benefit receipt, and education and training (which focuses on attaining a degree or credential).

P

P-value

The probability that if there actually was no real effect of the intervention, a finding at least as large as the observed finding would have been obtained by chance. For example, a sample might show a positive average difference, suggesting that the intervention group has better outcomes than the comparison group, with a p-value of 0.05. The p-value means that there is a 5 percent chance that the positive finding for the intervention group occurred by chance and does not occur in the population.

Physical health services

Services to address clients’ physical health concerns.

Q

Quasi-experimental design (QED)

A design in which the intervention and comparison groups are created through a process that is not random. For a QED to be rigorous, the intervention and comparison groups must have been comparable on predetermined characteristics at the start of the study. At best, strong QEDs will receive a moderate rating in the Pathways Clearinghouse.

R

Randomized controlled trial (RCT)

A design in which researchers randomly assign study participants to a group that receives the intervention services or a group that does not. RCTs are considered to produce the strongest possible evidence of effectiveness because random assignment ensures that no systematic differences existed between the study groups before the intervention.

S

Sanctions

Reductions in payment for failing to comply with mandated services.

Short-term

For this review, a period lasting 18 months or fewer after participants are first offered services.

Soft skills training

Training in so-called soft skills, such as punctuality, manners, professional dress, interactions with colleagues, or conflict management. Sometimes also called life skills training.

Statistical significance

The likelihood that a study finding is due to the intervention, rather than due to chance. The Pathways Clearinghouse will consider a finding to be statistically significant if the p¬-value of a two-sided hypothesis test of whether an effect is equal to zero is less than 0.05.

Study

An analysis of a distinct implementation of an intervention.

Study quality ratings

Studies vary in terms of their quality, or rigor. This variability in quality can result either from the way a study was designed or the way it was executed. Study quality affects the confidence we can have in the study’s findings. Three possible ratings—high, moderate, and low—describe our confidence that a given study’s finding is because of the intervention. We assign the study quality rating based on our assessment of its quality. For studies that include multiple findings, we might assess study quality for each finding. The study quality rating is the highest rating assigned to any of the findings. In the Pathways Clearinghouse, study quality ratings only apply to randomized controlled trials and quasi-experimental design studies.

  • HighHigh. The quality of the study is high, meaning we can be fairly confident in the study findings. Strong evidence shows that the study finding is solely attributable to the intervention examined. This rating is reserved for study findings from high quality RCTs with low attrition of sample members.
  • ModerateModerate. The quality of the study is moderate, meaning we can be somewhat confident in the study findings. However, other factors not accounted for in the study might also have contributed to the findings. This applies to findings from random assignment studies that, because of flaws in the study design or analysis (for example, high sample attrition), do not meet the criteria for the high rating but satisfy other design criteria. We also assign the moderate rating to study findings from well-executed QEDs.
  • LowLow. The quality of the study is low, meaning we cannot have much confidence in the study findings. Other important factors could have influenced the study findings, and the study did not account for them. Study findings that do not meet the high or moderate ratings criteria receive the low rating.

Overall, a study receives the highest study quality rating of any finding in that study. More information about study quality ratings is available in the Protocol for the Pathways to Work Evidence Clearinghouse.

Subgroup

A subset of individuals examined in a study who share a particular characteristic (for example, single parents or women).

Subsidized employment

Employment that is partially or fully paid for by an external funder (not the employer).

Substance use disorder treatment and mental health services

Services to treat clients for substance use disorder or mental health diagnoses.

Supported

We have some evidence that the intervention improves outcomes. These ratings are domain specific, meaning that the intervention is considered supported only for the particular domains for which we have given this rating. These interventions have at least one study of moderate or high quality and show evidence of favorable findings in the domain, but the evidence is less conclusive than that for evidence-based interventions.

Supportive services

Money or vouchers to fund child care, transportation (such as gas cards or tokens), or other supports to help clients search for work or engage in a training program.

T

Training

Any training program.

Transitional jobs

Jobs that are meant to integrate those who have been out of the workforce (for example, former prisoners) into the community. They can be paid or unpaid.

U

Unpaid work experience

Work experience that is voluntary or unpaid, such as an unpaid internship.

V

Very long-term

For this review, a period lasting five years or more after participants are first offered services.

W

Well-supported

We have strong and consistent evidence that the intervention produces favorable results for a specific outcome domain, such as short-term earnings. These interventions have at least two impact studies of moderate or high quality that show evidence of favorable findings within the domain. However, because implementation challenges and successes often vary, and because no two implementations of an intervention are identical, Pathways Clearinghouse users should not view this rating as a guarantee of future success.

Work experience

Paid or unpaid (such as internships) work experience.

Work experience and work-based training

Any paid or unpaid work experience or training that occurs in a work setting.

Work readiness activities

Services designed to help job seekers find a job that are not related to education or training. These can include initial assessments to identify employment barriers, formalized assessments to identify skills and interests, help designing a resume and cover letter, job search assistance, or help developing an individual employment plan.