The evaluation, conducted in accordance with the TBS 2009 Policy on Evaluation, examined Program relevance and performance. Methodologies were selected to ensure a thorough evaluation and recommendations to improve services for program recipients.
2.1 Evaluation Scope and Duration
Conducted between April 2015 and March 2016, the evaluation covered program activities for the period of April 2010 to March 2015, and assessed processes in place during the fieldwork phase (June 2015 to December 2015).
The evaluation was summative in nature and relied on a mix of qualitative and quantitative data sources to obtain a broad perspective and mitigate risks with data collection. As per the TBS Policy on Evaluation, five core issues were examined with the results intended to assist VAC senior management in making decisions regarding the design and delivery of the Program. The evaluation will focus on the core evaluation issues as outlined in Table 2:
TBS Requirements | Financial Benefit Evaluation Objectives |
---|---|
1. Continued Need for the Program | Assess the extent to which the Program continues to address a demonstrable need |
2. Alignment with Government Priorities | Assess the linkages between the objectives of the Program and (i) federal government priorities and (ii) departmental strategic initiatives |
3. Alignment with Federal Roles and Responsibilities | Assess VAC’s roles and responsibilities with regards to delivering the Program |
4. Achievement of Expected Outcomes | Assess progress towards expected outcomes |
5. Demonstration of Efficiency and Economy | Assess the Program’s resource utilization in relation to the production of outputs and progress toward expected outcomes |
The evaluation was calibratedFootnote 13 by placing more emphasis on areas of higher risk; i.e., ELB, PIA and PIAS, which represented 99% of all Program recipients at March 31, 2015. The introduction of the RISB in 2015 and the announcement of increasing the ELB from 75% to 90% of a Veteran’s monthly military salary as part of Budget 2016, were not included in the evaluation scope.
2.2 Multiple Lines of Evidence
The evaluation assessed both relevance and performance using multiple lines of evidence.
The evaluation methodology is outlined in Table 3.
Methodology | Source |
---|---|
Documentation Review |
|
Review of Research Studies |
|
File Review |
|
Key Informant Interviews |
|
Direct Observation |
|
Statistical/Program Data |
|
2.3 Evaluation Limitations
The following limitations and analytical challenges were identified during the evaluation:
- The evaluation team did not speak directly to Program participants. To mitigate this limitation, the team used the existing Rehabilitation Intake and Program Completion Survey Results as well as internal and external studies to assess sufficiency of benefits. As well, interviews with VAC staff who speak directly with Program recipients helped gauge their needs and views.
- Existing research and economic measures were used to assess appropriateness of benefit levels as professional expertise in measurement and management of risk were not available to the evaluation team.
These limitations should be considered when reviewing the evaluation findings.