1.0 Introduction

1.0 Introduction

The Evaluation of the Career Transition Service Program (CTS) and the Education and Training Benefit (ETB) was conducted by Goss Gilroy Inc. (GGI) on behalf of Veterans Affairs Canada's (VAC's) Audit and Evaluation Division. GGI conducted the planning and fieldwork stage of this evaluation between June and October 2021. A draft report, findings, and recommendations produced by GGI formed the basis of this final report prepared by VAC's Audit and Evaluation Division.

1.1 Background

1.1.1 Career Transition Services Program

Overview

The Career Transition Services Program works to ensure that eligible Canadian Armed Forces (CAF) Members, Members of the Reserves, Veterans, spouses, common-law partners and survivors have access to services that will help them develop the skills and tools they need to effectively search, apply for, and obtain civilian employment. With increased knowledge and skills, program participants are better prepared to navigate the civilian labour market not only on their initial transition out of the CAF, but also at any point where they may be faced with the transition from one type of employment to another.

Delivery

The CTS program is delivered by a contracted third-party service provider with policy guidance from the Department^{Footnote 1}. The program delivers services in three broad categories: 1) labour market information; 2) career counseling; 3) and job-finding assistance. Applicants can apply for this program in three ways. The fastest and easiest way is to apply on-line through My VAC Account. For those who prefer hard copy or are unable to apply online, they can apply by mailing a completed paper form or dropping it off at a VAC Area Office or CAF Transition Centre. Upon receiving the form, VAC will send a decision letter (approval or decline) to applicants within four weeks. If the client is eligible, the third-party service provider will work with the client to develop a plan to meet the client's employment needs.

With the purpose of improving employment prospects for participants, the program was re-designed by Veterans Affairs Canada in 2018. Currently, eligible individuals have access to career counselling, job-search assistance, and labour market information. Prior to the 2018 redesign, Veterans were required to find and pay for these services within the community and submit a request for reimbursement after the completion of the service^{Footnote 2}. As of April 1, 2018, services are provided by a third-party under contract who are then reimbursed by VAC^{Footnote 3}.

Eligibility criteria

The statutory framework for CTS is contained within the Veterans Well-being Act (VWA). Before 2018, only Veterans, Members, and survivors were eligible for the program. Currently, CTS is available for^{Footnote 4}:

Members who have completed basic training
Veterans (including Supplementary Reservists) who completed basic training and who were released from the CAF on or after April 1, 2006.
Spouse/common-law partners whose spouse completed basic training and released from the CAF on or after April 1, 2006
Survivors whose spouse completed basic training and died on or after April 1, 2006
Canadian Forces Income Support (CFIS) recipients (Veterans/survivors) who are currently entitled to Canadian Forces Income Support benefits.

To be eligible for CTS, all applicants must reside in Canada. Additionally, Veterans must demonstrate they have a need, and Veterans, spouses, common-law partners and survivors must not be in receipt of VAC rehabilitation services.

Resources

Veterans Affairs Canada allots $1.6 million in funding for CTS each year. A total of 3,813 CTS applications were approved by VAC between 2018-19 and 2020-21 (1,559 in 2018-19, 1,236 in 2019-20, and 1,018 in 2020-21). The forecast for program expenditures indicates increases over the next few years. Figure 1 details CTS program expenditures (actual and forecast) from 20117-18 to 2025-26^{Footnote 5}. It should be noted that 2017 expenditures were implementation dollars to develop and launch the contract prior to implementation of the re-designed CTS in April 2018.

Figure 1 – Career Transition Services expenditures actuals and forecast

Figure 1 – Career Transition Services expenditures actuals and forecast

Year	$ Millions
Réel
2017-18	$1.7
2018-19	$1.6
2019-20	$1.8
2020-21	$2.3
Prévu
2021-22	$2.8
2022-23	$3.0
2023-24	$3.3
2024-25	$3.5
2025-26	$3.7

Source: VAC Facts and Figures Book March 2021 Edition

1.1.2 Education and Training Benefit

Overview

Announced in Budget 2017^{Footnote 6}, the Education and Training Benefit (ETB) is designed to help Veterans successfully transition from military to civilian life, achieve their education and post-military employment goals, and better position them to be more competitive in the civilian workforce. It also permits Veterans to pursue activities that provide purpose and result in satisfaction with their main job or activity. In contrast with some other VAC programs and benefits, the ETB is not a needs-based program.

Complementing the CTS program, the ETB is a taxable benefit that provides up to $86,671.95^{Footnote 7} in funding (while a Veteran is participating in a formal program) to cover tuition, fees, supplies, and some incidental and living expenses. In addition, participants may access funding to contribute to the fees and costs associated with one-time courses aimed at self-fulfillment and personal interest and development. Veterans may be funded for most types of formalized educational programs, depending on whether they meet the eligibility criteria.

Delivery

Starting in 2018, VAC has provided ETB funding to eligible Veterans for short-term courses as well as formal education programs. Veterans can apply for ETB in different methods: online via a guided form on My VAC Account, dropping off the completed application form at a VAC Office or Canadian Forces Transition Center, or sending the form to Veterans Affairs Canada through the mail. VAC is expected to send a decision letter to the applicant within four weeks of receiving the completed application^{Footnote 8}.

The applicant wishing to pursue formal education programs must apply for the ETB and submit an ETB plan for approval before the start date of the educational program. The education institution must be listed on the Employment and Social Development Canada (ESDC) Master List of designated educational institutions, or if the institution is not listed on the ESDC Master List, applicants must submit documentation proving that it is a government recognized post-secondary education institution.

After an application and plan are approved by VAC, funding is released directly to the student for the first term of study^{Footnote 9}. A maximum of $21,077.81 is available per year to eligible individuals pursuing formal education programs. In order to continue to receive funding for the future periods of study, applicants must submit a completed monitoring form, provide proof of completion and proof of enrollment for the following term to VAC. In contrast, for Veterans who wish to apply for funding for short courses, a completed Short Course application must be received by VAC prior to the first day of classes. Veterans who take short-term courses can receive funding for the actual costs charged directly by the institution or provider up to a maximum of $5,269.45.

Eligibility criteria

The administration of the ETB falls under the Veterans Well-being Act. Veterans are eligible to apply for the ETB if:^{Footnote 10}

They were honourably released from the CAF (Regular or Reserve Force) on or after April 1, 2006, or are a member of the Supplementary Reserve; and
They meet the “length of service” requirement as follows:
- At least 12 years of authorized days of CAF Service (4382 days) to receive up to $86,671.95 (2022) or
- At least 6 years of authorized days of CAF service (2191 days) to receive up to $43,335.98 (2022).

Veterans have up to 10 years following their release date to receive ETB funding:

Veterans released from the CAF between April 1, 2006, and March 31, 2018, will have until April 1, 2028, to receive funding.
Members of the Supplementary Reserve who are honourably released between 1 April 2018 and 4 July 2019 have 10 years from date of release to receive funding.
Members of the Supplementary Reserve who are honourably released on July 5, 2019, have until July 5, 2029, to receive funding.
Those who transferred to the Supplementary Reserve after 5 July 2019 have 10 years from the date of transfer.

Resources

With total expenditures in the first fiscal year 2018-2019 being $12.3 million, ETB helped 1,072 Veterans pursue education and personal development goals.^{Footnote 11} Expenditures for ETB increased to $20.6 million in the fiscal year 2019-2020, supporting a total of 1,700 students. As Figure 2 shows, forecasted expenditures and recipients are expected to grow over the next five years.

Figure 2 – Education and Training Benefit recipients and expenditures actual and forecast

Figure 2 – Education and Training Benefit recipients and expenditures actual and forecast

Year	Expenditures (Actual)	Recipients
2018-19	$12.3	1072
2019-20	$20.6	1700
2020-21	$23.4	1933
Year	Expenditures (Forecast)	Recipients (Forecast)
2021-22	$28.8	2230
2022-23	$25.4	1950
2023-24	$29.4	2210
2024-25	$34.7	2560
2025-26	$35.5	2560

Source: VAC Facts and Figures Book March 2021 Edition

1.2 Evaluation purpose, scope and questions

1.2.1 Purpose

The evaluation of CTS and ETB is required to meet the Financial Administration Act (section 42.1) and Policy on Results requirements to conduct “Evaluations of ongoing programs of grants and contributions with five-year average actual expenditures of $5 million or greater per year.” The evaluation also fulfills the department’s responsibility to provide evaluation coverage of organizational spending and the programs in the Program Inventory as per section 4.3.15.2 of the Policy on Results. The evaluation is being conducted as part of the 2020-25 Departmental Evaluation Plan that was approved by the Deputy Minister on August 10, 2020.

1.2.2 Scope

The scope of the evaluation includes all the activities of the CTS and ETB programs conducted during the years 2018-19 to 2020-21. Where available and appropriate, updated data may be referenced in this report.

1.2.3 Objectives

The objective of this evaluation is to assess the programs’ relevance, effectiveness and efficiency. In addition, the evaluation also considered:

Alignment between the CTS and ETB programs;
Application of a Gender-based Analysis Plus (GBA+) lens to the evaluative process to assess how diverse groups of women, men and non-binary people served by VAC experienced the programs; and
Understanding the extent to which the programs are producing outcomes that contribute to the well-being of eligible Veterans.

1.2.4 Evaluation questions

The evaluation of the CTS program and ETB examines issues of relevance, effectiveness and efficiency. The following questions are addressed:

Relevance

What is the relationship between CTS and ETB and between these programs and other VAC programs in the continuum of supports for Veterans? Are these relationships appropriate/working effectively to meet Veterans’ needs?
To what extent are the eligibility criteria for CTS and ETB meeting Veterans’ needs?
To what extent do Veterans have unmet needs for career transition and/or accessing education or training?

Effectiveness

Are the programs achieving the expected/desired reach?
Are there alternative program delivery approaches that could be considered that would increase outcomes for Veterans?
How effectively have the programs supported Veterans to meet their personal and professional goals and advance their well-being as per VAC’s domains of well-being?
Were there unexpected outcomes (positive or negative) as a result of receiving services/benefits, including unintended GBA+ related impacts that should be resolved?

Efficiency

Are the available program data for CTS and ETB adequate to measure its outcomes/impact?
Have the activities of the CTS and ETB been delivered in an efficient and economical manner?

The evaluation questions along with the associated indicators and data sources are presented in Appendix A.

1.3 Methodology

Consistent with TBS guidelines and recognized best practice in evaluation, a range of methods and sources were used to triangulate evaluation findings. This section provides a description of the methodological approach for the evaluation. The data collection and analysis were guided by the evaluation questions and indicators outlined in Appendix A.

1.3.1 Review of documents, program and performance data and files

The GGI team reviewed relevant program documents and data related to the CTS and ETB programs. These documents include:

Annual reports and other sample reports from the CTS third-party service provider
ETB applicant guide, FAQ, policies, business process, etc.
CTS policies, processes, FAQ, previous evaluation report, etc.
Transition Services Performance Information Profile (PIP)

1.3.2 Key informant interviews

To help scope the evaluation, GGI held 7 meetings with 11 individuals to better understand the program context and explore areas of interest for the evaluation. GGI also held two meetings to prepare for subsequent interviews. One was held with internal VAC contracting officers and one was held with the third-party service provider (and also attended by a VAC contracting officer). These meetings were held to better understand the third-party contract requirements, gather descriptive information about the third-party service provider processes and data collection, and to discuss the best approach for the interview with the third-party service provider.

During the evaluation fieldwork stage, VAC Audit and Evaluation Division worked with the program areas to identify individuals who were familiar with the two programs and who could comment on the evaluation questions. A total of nine interviews were conducted with 12 participants as follows:

VAC program managers and senior advisors (6 interview sessions were held with 8 participants);
VAC appeals officers (2 individual interviews);
Representatives of third-party service provider (1 interview/group session were held with 2 participants).

Key informant interviews were conducted by telephone or online meeting software and lasted approximately 1 hour. In one instance, a written response to questions was provided.

1.3.3 Focus groups with front line staff

The Audit and Evaluation Division of Veterans Affairs Canada contacted front line and adjudication staff who were identified by their managers as individuals that would add value and insight from a hands-on perspective. A total of four virtual focus group sessions were conducted with the following number and types of participants:

One group with 5 Benefits Program Officers;
One group with 2 Veteran Service Agents and 1 Case Manager;
One group with 1 Veteran Service Agent and 1 Case Manager;
One group with 2 Case Managers.

Focus group sessions lasted approximately 1.5 to 2 hours. The purpose of the focus group sessions was to understand front line staff members’ current awareness of each of the programs, obtain their feedback on information available for each of the programs, and their suggestions for improvements to the programs. Focus groups were also used to gain insight into the adjudication process and the efficiency/performance of the program from the adjudicator’s perspective.

1.3.4 Survey of participants

An online survey was conducted of Veterans in receipt of CTS and ETB services, or both services.

Sample

The sample was selected by the Veterans’ Affairs evaluation team and included Veterans in receipt of services within the scope of the evaluation. This included Veterans who participated in CTS activities during the years 2018-19 to 2020-21, and Veterans in receipt of ETB since its launch on April 1, 2018. The survey was administered via My VAC Account.

The population was comprised of a total of 7,259 Veterans with unique IDs who participated in the CTS, ETB, or both programs since 2018-19. Of the 7,259, the survey was administered to 3,932 Veterans comprised of those that had their My VAC Account notifications enabled. Of the 3,932 Veterans, the online survey was administered to three separate groups of Veterans with the following program involvement:

CTS Only = 848
ETB Only = 1,941
CTS and ETB = 1,143

Response rates

A total of 672 Veterans completed the survey overall, of which 129 reported they received CTS services only, 377 received ETB services only, and 166 received both CTS and ETB services.

The response rates^{Footnote 12} are as follows:

Overall response rate: 17.1% (672/3,932)
CTS only response rate: 15.2% (129/848)
ETB only response rate: 19.4% (377/1,941)
Both CTS and ETB response rate: 14.5% (166/1,143)

Sampling and margin of error

A representative sample is a subset of a population that seeks to accurately reflect the characteristics of the larger group. In total, the sampling error is calculated based on Veterans in scope for this research, including those with notifications enabled and disabled. In total, 7,259 Veterans were eligible to receive the survey, with 672 completing the survey (8.9%). At the overall level, the survey has a margin of error of plus or minus 5 percent^{Footnote 13} at a 95 percent level of confidence.

1.4 Analytical approach

The collection of evaluation data from all lines of evidence formed the input for the analysis. The data/evidence gathered was compiled into matrices to enable triangulation. This report includes findings from all lines of evidence.

1.5 Strengths and limitations

The evaluation addresses the evaluation questions to the extent that accurate and complete information was available. Where there is not sufficient information, this is noted in the report, and the findings are interpreted accordingly. The following is an overview of the strengths and limitations of this evaluation.

1.5.1 Strengths

Overall, this evaluation presents a strong methodology. The use of multiple lines of inquiry enhances confidence in the findings and compensates, to an extent, for any non-response bias. In addition, by using both surveys and interviews/focus groups to collect information from clients and internal VAC employees, the evaluation provides an opportunity to make statements on the extent to which a theme is common (survey) and explain reasons why it is the case (interviews). Additionally, the analysis of the program data allowed for a greater understanding of the administrative processes associated with ETB and CTS, as well as the demographic characteristics of applicants.

Other strengths of the evaluation design include:

Accessibility – The evaluation team is trained on strategies for improving accessibility and making data collection easy for respondents. This included the use of multiple modes for each line of inquiry (e.g., virtual meetings, online survey). Bilingual options were offered to all respondents.
Confidentiality – The evaluation made every effort to protect the privacy and confidentiality of participants. This included using the My VAC Account for clients, and a range of strategies for recruiting VAC employees.
Reach – The survey used a census approach in which all eligible respondents who enabled the My VAC Account correspondence feature were invited to complete an online survey. With respect to internal VAC employees, invitations for interviews and focus groups offered multiple dates and times to accommodate their schedules and availabilities. Interviewees and focus group members had the option of participating in the official language of their choice.

1.5.2 Limitations

Survey of Participants

While every effort was made to minimize limitations, there were challenges with regards to accessing potential respondents:

In order to protect client confidentiality, it was not possible to access a complete list of clients. Instead, only Veterans that opted-in to be contacted on their My VAC Account were included in the sampling frame.
Data collection occurred over the summer months at a time when potential respondents may have been less available or inclined to participate.
It is possible that there are differences between stakeholders who participated in data collection and those who did not. For example, clients who had either very positive or very negative experiences may have been more enthusiastic about participation. This means that findings from individual lines of inquiry are not, generalizable to the whole population. To mitigate, alternative lines of evidence were used to support survey findings.

Program data

The program data was used mainly to address evaluation questions associated with program effectiveness and the degree to which the programs achieved the expected and desired reach, including delineating the demographic characteristics of clients. The data files did not contain information related to eligibility decisions, or program utilization over time, other than application processing data. To mitigate this challenge, data from the third-party service provider's Three-Year Annual Report was summarized for this purpose. It is felt that while the data from the third-party service provider may not be encompassing of all program data, it does provide an acceptable level of representation of outcomes and trends over time.