2.0 Scope and Methodology

2.0 Scope and Methodology

The CTS Program, a grant program, was evaluated in accordance with the Financial Administration Act (Section 42.1) which requires a relevance and effectiveness review of any Grants and Contributions program every five years. The Program was last reviewed in February 2011, with the Terms and Conditions expiring on March 31, 2016.

2.1 Scope

The evaluation covered the period from April 2010 to September 2015 and was conducted between October 2015 and March 2016.

The following elements were scoped out of the evaluation:

  • An investigation into the specific transition employment needs of medically-releasing CAF members,
  • Opportunities for additional training and education for releasing CAF members.

As per the TBS Directive on the Evaluation Function, five core issues were examined with the results intended to assist VAC senior management in making decisions regarding the design and delivery of career transition services to CAF releasing members or Veterans. Table 1 matches the TBS five core issues with the specific CTS evaluation objectives.

Table 1 – Core Issues and Evaluation Objectives
Core Objectives (as per TB requirements) Evaluation Objectives
Relevance
1. Alignment with Government Priorities Assess the linkages between the objectives of the Program and federal government priorities.
2. Alignment with Federal Roles and Responsibilities Assess VAC’s roles and responsibilities and strategic initiatives with respect to delivering the Program.
3. Continued Need for the Program Assess the extent to which the Program continues to address a need.
Performance
4. Achievement of Expected Outcomes Assess progress toward expected Program outcomes.
5. Demonstration of Efficiency and Economy Assess the Program resource utilization in relation to the production of outputs and progress toward expected outcomes.

2.2 Multiple Lines of Evidence

Multiple lines of evidence were employed to assess the Program’s relevance and performance, as detailed in Table 2.

Table 2 - List of Methodologies
Methodology Source
Documentation Review
  • Previous VAC evaluations
  • Evaluations from other government departments
  • Program documents and data from other countries (United Kingdom and United States)
  • Recipient survey results
  • Departmental Acts, Regulations, program and planning documents
  • TB Submissions
Research Studies
  • VAC Research Directorate studies
  • Studies conducted by other federal government departments
  • Studies conducted by other countries
  • Evidence-based non-VAC literature
Key Informant Interviews
  • Combination of 34 in-person or telephone interviews with VAC staff and government department subject matter experts
Statistical/Program Data
  • VAC Finance Division
  • Program Performance Measurement
  • VAC Statistics Directorate
  • VAC Research Directorate
Site Visits
  • CAF bases/wings
  • VAC Area Offices
  • Other federal government offices
  • Provincial agency
  • Private entities

2.3 Considerations, Strengths and Limitations

A number of considerations, strengths and limitations were factored into this evaluation, with varying degrees of impact.

Considerations:

  • Program design and eligibility changes resulting from the 2012 Deficit Reduction Action Plan (DRAP) effectively reset the Program such that the review period actually covered two significantly different career transition programs, the latter with a smaller target population and an alternate delivery method. This evaluation focused primarily on the current version of the Program.
  • All three phases of the New Veterans Charter Evaluation provided baseline information for this evaluation.

Strengths:

  • VAC’s Research Directorate has been leading several initiatives which have resulted in a collection of evidence and data pertaining to the Veteran population. This information afforded the evaluation team a better understanding of Veteran needs.

Limitations:

  • Evaluation time constraints did not allow for direct participant outcome research (e.g., focus groups). To mitigate this limitation, the evaluation team reviewed previous Public Opinion Research (focus group testing) results, interviewed VAC and CAF personnel directly involved with those Veterans who have employment challenges as well as public and private agencies involved in assisting Veterans find employment.
  • Similarly, the evaluation team was not able to contact Veterans. The team partially mitigated this limitation by conducting interviews with VAC employees who deal directly with Program recipients with the intent to obtain the perspectives of Veterans.
  • There were challenges accessing some of the data required from the Statistics Directorate due to a change in data query tools. Because the team’s data needs were not extensive, a combination of Statistics-produced data and existing information was adequate.

Overall, the evaluation team is confident these limitations do not impact the ultimate findings or conclusions.

 
Date modified: