2.0 Scope and Methodology

2.0 Scope and Methodology

The evaluation, conducted in accordance with the TBS 2009 Policy on Evaluation, examined Program relevance and performance. Methodologies were selected to ensure a thorough evaluation and recommendations to improve services for program recipients.

2.1 Evaluation Scope and Duration

Conducted between April 2015 and March 2016, the evaluation covered program activities for the period of April 2010 to March 2015, and assessed processes in place during the fieldwork phase (June 2015 to December 2015).

The evaluation was summative in nature and relied on a mix of qualitative and quantitative data sources to obtain a broad perspective and mitigate risks with data collection. As per the TBS Policy on Evaluation, five core issues were examined with the results intended to assist VAC senior management in making decisions regarding the design and delivery of the Program. The evaluation will focus on the core evaluation issues as outlined in Table 2:

Table 2 – Five Core Evaluation Issues
TBS Requirements Financial Benefit Evaluation Objectives
1. Continued Need for the Program Assess the extent to which the Program continues to address a demonstrable need
2. Alignment with Government Priorities Assess the linkages between the objectives of the Program and (i) federal government priorities and (ii) departmental strategic initiatives
3. Alignment with Federal Roles and Responsibilities Assess VAC’s roles and responsibilities with regards to delivering the Program
4. Achievement of Expected Outcomes Assess progress towards expected outcomes
5. Demonstration of Efficiency and Economy Assess the Program’s resource utilization in relation to the production of outputs and progress toward expected outcomes

The evaluation was calibratedFootnote 13 by placing more emphasis on areas of higher risk; i.e., ELB, PIA and PIAS, which represented 99% of all Program recipients at March 31, 2015. The introduction of the RISB in 2015 and the announcement of increasing the ELB from 75% to 90% of a Veteran’s monthly military salary as part of Budget 2016, were not included in the evaluation scope.

2.2 Multiple Lines of Evidence

The evaluation assessed both relevance and performance using multiple lines of evidence.

The evaluation methodology is outlined in Table 3.

Table 3 – List of Methodologies
Methodology Source
Documentation Review
  • Departmental Acts and Regulations
  • VAC reports, policies, processes, strategies and planning documents
  • Previous VAC audits and evaluations
  • Parliamentary reports
  • Pre-existing recipient survey results (e.g. Rehabilitation Intake and Program Completion Survey Results)
Review of Research Studies
  • Studies conducted by VAC Research Directorate
    • Income Adequacy and Employment Outcomes of the New Veterans Charter
    • Compensating for Permanent Losses: Totally and Permanently Incapacitated
    • Low Income Lines and Financial Security in Retirement
    • Labour-Market Outcomes of Veterans
    • Economic Loss: Is it related to age or disability rating?
    • Income Recovery after Participation in the Rehabilitation Program
File Review
  • A sample of 309 Veteran files were reviewed
    • 164 ELB recipients
    • 36 CFIS recipients
    • 42 SRB recipients
    • 67 PIA/PIAS recipients
Key Informant Interviews
  • A total of 69 (44 Area Office and 25 Head Office) in-person interviews with and questionnaire responses from VAC staff and departmental subject matter experts.
Direct Observation
  • Information Technology systems, as well as business processes and practices, were observed at VAC Head Office.
Statistical/Program Data
  • Analysis of statistical data obtained from
    • VAC Finance Division
    • Service Delivery Branch

2.3 Evaluation Limitations

The following limitations and analytical challenges were identified during the evaluation:

  1. The evaluation team did not speak directly to Program participants. To mitigate this limitation, the team used the existing Rehabilitation Intake and Program Completion Survey Results as well as internal and external studies to assess sufficiency of benefits. As well, interviews with VAC staff who speak directly with Program recipients helped gauge their needs and views.
  2. Existing research and economic measures were used to assess appropriateness of benefit levels as professional expertise in measurement and management of risk were not available to the evaluation team.

These limitations should be considered when reviewing the evaluation findings.

 
Date modified: