Evaluation of the Career Transition Services Program and the Education and Training Benefit Summary

Evaluation of the Career Transition Services Program and the Education and Training Benefit Summary

Veterans Affairs Canada - May 2022

Evaluation results

Relevance and Need

ETB is largely meeting Veterans’ expectations, while Veterans identified unmet needs associated with the CTS program. Eligibility is appropriate for both programs, although there are areas of confusion amongst potential clients. Improvement could be made to ETB short course funding.

CTS and ETB generally work well together, particularly for Veterans who access education planning services through CTS. The complementary nature of the programs is reinforced in program policies and other documentation.

Effectiveness

VAC front-line staff have a general knowledge of the CTS and ETB programs. There is an opportunity to improve communications with front-line staff around the programs.

Veterans surveyed indicated that their awareness of the CTS and ETB programs while they were serving was limited. The evaluation found that being able to access ETB does not impact Veterans’ decision to leave the Canadian Armed Forces.

The evaluation finds the ETB program is being delivered effectively. Eighty percent of Veterans surveyed indicated the training funded by the program met their needs, with 59% reporting that ETB provided them with the skills and knowledge they needed to engage in activities important to them.

Approximately 85% of survey respondents who participated in the ETB program indicated that they had completed their course of study or were still enrolled. Most indicated a satisfactory outcome from the program.

Client satisfaction with CTS suggests that delivery is not meeting needs or expectations. Overall, 56% of CTS clients surveyed (including clients who received both CTS and ETB) indicated satisfaction with the services received from the third-party service provider (55% indicated satisfaction with VAC services).

Most clients surveyed report their overall health and wellbeing, their adjustment to life after service and satisfaction with their life as good to excellent. Clients who only accessed ETB are more likely to report better outcomes in all these areas than clients who only accessed CTS.

Efficiency

Program expenditures for both CTS and ETB are on the rise historically and are forecast to increase in the coming years. Increased client uptake and expenditures are more pronounced for ETB in the forecast data.

Program data does not allow for easy measurement of program outcomes; the evaluation relied on the survey of clients to assess outcome achievement. There is an opportunity to improve data capture and integration the next time system changes are made.

Recommendations

Recommendation 1 Management responses and action plans

It is recommended that the DG-SDPM improve communications about the programs among Members and Veterans, including:

  1. Information about eligibility for the ETB and the two-stage application process.
  2. Information to improve program awareness for Members planning for their release, including additional training and tools to support VAC front-line staff who support the transition.

With respect to Recommendation 1A, the Director General, Service Delivery and Program Management will work in collaboration with Communications, Centralized Operations Division (COD), Policy, Business Integration Directorate (BID), National Learning Unit (NLU), National Client Center Network (NCCN), and Field Operations (FO) to:

  1. Review and analyze the current situation to better understand the needs for enhanced communications by engaging directly with internal stakeholders;
  2. As a result of the feedback collected above, develop a strategy to improve communication as it relates to eligibility and the two-step application process; and
  3. Design, update, and release communication products to highlight information about eligibility and the two-stage application process.

With respect to Recommendation 1B, the Director General, Service Delivery and Program Management will work in collaboration with, Communications, National Client Center Network (NCCN), Transition, Field Operations (FO), and National Learning Unit (NLU) to:

  1. Review current external and internal communication channels including analytics to better understand the most effective mechanisms for positive engagement;
  2. Review Communication products to implement enhancements and ensure consistent and accurate program information is concisely presented; and
  3. Develop and disseminate training materials to front-line staff who counsel releasing Members and Veterans.
Recommendation 2 Management responses and action plans

It is recommended that the DG-Policy undertake efforts to review the maximum amount available for ETB short courses and consider whether costs of licensing exams and fees should be included.

With respect to Recommendation #2, the Director General, Policy and Research will:

  1. Work in collaboration with SDPM, CPC, Finance, and IT to:
    1. Collect data on short course applications (numbers of favourable/unfavourable decisions, types of courses, costs, etc.).
    2. Undertake an analysis on the information gathered to determine if there are gaps and possible options.
  2. Collect data and conduct policy analysis to determine whether to include the costs of licensing exams and fees as eligible benefits.
Recommendation 3 Management responses and action plans

It is recommended that the DG SDPM explore client satisfaction results and associated performance data to identify areas for further improvement, and gaps where further data collection is required.

VAC’s current contract with a national service provider for Career Transition Services is due to expire in 2022 and VAC is in the process of re-procurement.

  • New contract will be in place (early 2023)
  • Contract statement of work (issued in Fall 2021 after the evaluation scope period) includes more robust language to address areas where client needs are not being met
  • VAC will work closely with the service provider during contract implementation to:
    • Develop a methodology to measure client satisfaction with the program
    • Identify gaps where further data collection is required
    • Develop/change business processes where needed to ensure correct data is collected to identify client needs
  • Using the results of client satisfaction data, implement business processes where necessary to address client needs.

Program Description

  • The Career Transition Services Program (CTS) works to ensure that eligible Canadian Armed Forces (CAF) Members, Reservists, Veterans, spouses, and survivors have access to services that will help develop the skills and tools they need to effectively search, apply for, and obtain employment.
  • Services are delivered by a contracted service provider and focus on three broad categories: 1) labour market information; 2) career counseling; 3) and job-finding assistance.
  • The Education and Training Benefit (ETB) is designed to help Veterans successfully transition from military to post-service life, achieve their education and post-military employment goals, and better position them to be more competitive in the civilian workforce.
  • The ETB provides a taxable benefit of up to $86,671.95 (while a Veteran is participating in a formal program) to cover tuition, fees, supplies, and some incidental and living expenses.
  • Veterans may be funded for most types of formalized educational programs and can also receive funding for one-time courses aimed at self-fulfillment and personal interest/development.

About the Evaluation

Scope and Methodology

  • The CTS and ETB evaluation was conducted by Goss Gilroy Inc. on behalf of VAC between February 2021 and October 2021. It was conducted as part VAC’s 2020-25 Departmental Evaluation Plan in accordance with Treasury Board’s Policy on Results.
  • The evaluators assessed the programs’ relevance, effectiveness, and efficiency over the period April 1, 2018, to March 31, 2021.
  • The evaluation also considered:
    • Alignment between the CTS and ETB programs;
    • Application of a Gender-based Analysis Plus (GBA+) lens to the evaluative process to assess how diverse groups of women, men and non-binary people served by VAC experienced the programs; and
    • Understanding the extent to which the programs are producing outcomes that contribute to the well-being of eligible Veterans.

Constraints and Limitations

  • The evaluation team conducted an on-line survey of participants in the CTS and ETB programs. Only Veterans that opted-in to be contacted on their My VAC Account were included in the sampling frame.
  • Data collection, including the survey, occurred over the summer months at a time when potential respondents may have been less available or inclined to participate.
  • It is possible that there are differences between stakeholders who participated in data collection and those who did not. For example, clients who had either very positive or very negative experiences may have been more enthusiastic about participation. This means that findings from individual lines of inquiry are not, for the most part, generalizable to the whole population. To mitigate, alternative lines of evidence were used to support survey findings.
  • VAC data files did not contain information on indicators related to eligibility decisions, or program utilization over time, other than application processing data. To mitigate, data from the third-party service provider’s Three-Year Annual Report was used to supplement VAC data.

Evaluation Report