3.0 Conclusions and Recommendations

3.0 Conclusions and Recommendations

3.1 Conclusions

3.1.1 Relevance

ETB is largely meeting Veterans’ expectations, while CTS appears to be falling somewhat short in comparison. Eligibility is considered appropriate for both programs, although there are some areas of confusion among potential clients of the programs. There were no suggestions for how CTS could be better designed to meet needs. However, some design features of ETB were noted as not meeting expectations including the size of the cap for short courses, and the type of training and educational institutions that are eligible.

The programs generally work well together, particularly for Veterans who access CTS’s education planning services. The complementary nature of the programs is reinforced in program policies and other documentation. The programs are viewed as separate from other VAC programs (such as the Rehabilitation Program), which is understandable given the very different eligibility for these other programs. For those consulted for the evaluation who saw a relationship, it was considered complementary in nature. A few noted some confusion between eligibility between the programs.

3.1.2 Effectiveness

Awareness of the programs among VAC front-line staff is quite general. Clients indicated that their awareness of the program while they were in service was low. The evaluation found that being able to access ETB does not impact a Veteran’s decision to leave the Canadian Armed Forces.

Program delivery for ETB appears to be functioning well. The program or courses Veterans accessed met their needs 80% of the time, and 59% reported that ETB provided them with the skills and knowledge they needed to engage in activities important to them. However, client satisfaction with CTS would suggest that delivery is not meeting needs. Indeed, an overall satisfaction rating of 56% among CTS clients (or CTS and ETB clients) for services received from the third-party service provider (and a 55% satisfaction rating for VAC services) would suggest that delivery is not meeting expectations. The most poorly rated aspects of CTS delivery were: self-marketing (31%); labour market information and research techniques (31%); interview skills and techniques (32%); job finding and placement assistance (34%); job search techniques and supports (37%); and employment transition support (45%). This is in contrast to quite high satisfaction ratings reported by the third-party service provider based on their client satisfaction survey they issue shortly after a file is closed.

Most opportunities for improvement identified by VAC employees consulted for the evaluation centred around improving communications.

In terms of the extent to which the programs help Veterans achieve their goals, the evaluation found this is happening to a moderate extent. Many CTS clients report that CTS helped them achieve one or more of their career transition goals with nearly two-thirds reporting they were employed at the end of services. For ETB (or ETB and CTS), 85% of survey respondents indicated that they had completed their program or were still enrolled. Most said they were satisfied with the outcome of their ETB participation.

Most clients report their overall health and wellbeing, their adjustment to life after service and satisfaction with their life as good to excellent (70%, 65% and 75% respectively). Clients who only accessed ETB are more likely to report better outcomes in all these areas than clients who only accessed CTS. The reasons for this difference are unclear, although it is possible that CTS and ETB clients are in different places in their lives and are seeking different outcomes from the program.

3.1.3 Efficiency

Program expenditures for both CTS and ETB are on the rise historically and are forecast to increase in the coming years. Increased client uptake and expenditures are more pronounced for ETB in the forecast data.

The evaluation also explored the extent to which the program data allows for the measurement of outcomes. It was found that there are several gaps in data and errors in data entry. The evaluation had to rely on the survey of clients to assess the achievement of outcomes.

3.2 Recommendations

The evaluation makes the following recommendations:

Recommendation #1

It is recommended that the DG-SDPM improve communications about the programs among Members and Veterans, including:

  1. Information about eligibility for the ETB and the two-stage application process.
  2. Information to improve program awareness for Members planning for their release, including additional training and tools to support VAC front-line staff who support the transition.

Management Response: Veterans Affairs Canada agrees with this recommendation.

Action and Rationale Expected Completion/ Implementation Date ADM Accountable for Action

With respect to Recommendation #1 a., the Director General, Service Delivery and Program Management will work in collaboration with Communications, Centralized Operations Division (COD), Policy, Business Integration Directorate (BID), National Learning Unit (NLU), National Client Center Network (NCCN), and Field Operations (FO) to:

  1. Review and analyze the current situation to better understand the needs for enhanced communications by engaging directly with internal stakeholders; and
  2. As a result of the feedback collected above, develop a strategy to improve communication as it relates to eligibility and the two-step application process; and
  3. Design, update and release communication products to highlight information about eligibility and the two-stage application process.
 March 31, 2023 Assistant Deputy Minister, Service Delivery

With respect to Recommendation #1 b., the Director General, Service Delivery and Program Management will work in collaboration with, Communications, National Client Center Network (NCCN), Transition, Field Operations (FO), and National Learning Unit (NLU) to:

  1. Review current external and internal communication channels including analytics to better understand the most effective mechanisms for positive engagement;
  2. Review Communication products to implement enhancements and ensure consistent and accurate program information is concisely presented; and
  3. Develop and disseminate training materials to front-line staff who counsel releasing Members and Veterans.
 December 31, 2023 Assistant Deputy Minister, Service Delivery

Recommendation #2

It is recommended that the DG-Policy undertake efforts to review the maximum amount available for ETB short courses and consider whether costs of licensing exams and fees should be included.

Management Response: Veterans Affairs Canada agrees with this recommendation.

Action and Rationale Expected Completion/ Implementation Date ADM Accountable for Action

With respect to Recommendation #2, the Director General, Policy and Research will:

  1. Work in collaboration with SDPM, CPC, Finance and IT to:
    1. Collect data on short course applications (numbers of favourable/unfavourable decisions, types of courses, costs, etc.).
    2. Undertake an analysis on the information gathered to determine if there are gaps and possible options.
    3. Collect data and conduct policy analysis to determine whether to include the costs of licensing exams and fees as eligible benefits.
 March 31, 2023 Assistant Deputy Minister, Strategic Policy and Commemoration

Recommendation #3

It is recommended that the DG SDPM explore client satisfaction results and associated performance data to identify areas for further improvement, and gaps where further data collection is required.

Management Response: Veterans Affairs Canada agrees with this recommendation.

Action and Rationale Expected Completion/ Implementation Date ADM Accountable for Action

With respect to Recommendation #3, VAC’s current contract with a national service provider for Career Transition Services is due to expire in 2022 and VAC is in the process of re-procurement.

  • New contract will be in place (early 2023)
  • Contract statement of work (issued in Fall 2021 after the evaluation scope period) includes more robust language to address areas where client needs are not being met
  • VAC will work closely with the service provider during contract implementation to:
    • Develop a methodology to measure client satisfaction with the program
    • Identify gaps where further data collection is required
    • Develop/change business processes where needed to ensure correct data is collected to identify client needs
  • Using the results of client satisfaction data, implement business processes where necessary to address client needs.
 March 31, 2024 Assistant Deputy Minister, Service Delivery
 
Date modified: