The VEF was implemented with approved funding for three fiscal years (2018-19 to 2020-21). To access ongoing funding for the VEF, the Minister of Veterans Affairs was required to return to Central Agencies in 2020/21 with information on need for the program and its level of ongoing funding, information on the performance, usage, and results of the program over the first three years, and strengthened governance based on the experience over the first three years of the program.
In consideration of risk and need, it was determined that the VEF would benefit from an evaluation. First appearing on Veterans Affairs Canada Risk-Based Audit and Evaluation Plan 2018-23, the evaluation was conducted in accordance with Treasury Board of Canada’s Policy on Results.
2.1 Evaluation Scope and Questions
The evaluation was conducted between February 2020 and October 2020 and primarily focussed on the time period of April 1, 2018 to August 30, 2020.
The objective of the evaluation is to assess the relevance, performance, effectiveness, economy and efficiency of the VEF. Findings from the evaluation will be used to support decision making, planning, and implementation of future departmental initiatives for the program. Evaluation findings will also be used to determine the need for ongoing program funding.
Preliminary discussions, document review, and data analysis identified potential program risk areas and determined the areas of priority and focus for this evaluation. Upon completion of an initial review, evaluation questions were developed to help assess program relevance, performance, efficiency and economy, and effectiveness in order to achieve the evaluation objective. A list of evaluation questions, highlighting the key lines of inquiry, can be found in Table 1 below. For further details and methodologies, see Appendix A - Evaluation Matrix.
Table 1: Evaluation Questions
To what extent does the program continue to address a demonstrable need, now and in the future? (Relevance)
To what extent does the program align with Government of Canada priorities and with federal roles and responsibilities? (Relevance)
To what extent is the Program responsive to the needs of recipients and target audiences? (Relevance)
Are processes currently in place to ensure the program is being administered as indicated in departmental guidance? (Performance)
To what extent is the program achieving its outcomes? (Performance)
- Immediate
- Intermediate
- Ultimate
Are there opportunities to improve the efficiency and economy of the program or ways to improve the effectiveness of the program? (Efficiency/economy/effectiveness)
Are there any unintended impacts resulting from the program (positive or negative)? (Efficiency/economy/effectiveness)
2.2 Multiple Lines of Evidence
Multiple lines of evidence have been used to support the evaluation findings. The methods undertaken to support these lines of evidence are identified in Table 2.
| Methodology | Source |
|---|---|
| Departmental Documentation and Secondary Research Review | Departmental documentation/information has been reviewed to understand the program objectives/intent, authorities and requirements, complexity, context and any key issue areas. Documents included departmental planning documents, foundational documents (Order in Council, Terms and Conditions, etc.) policies, mandate letters, business processes, guidelines, strategic documents, performance reports, research papers, survey results, and correspondence. |
| Non-Departmental Document Review | Various non-departmental documents were reviewed, including, Parliamentary reports, Budget Speeches/Plans, and Speeches from the Throne. |
| Interviews | Interviews were conducted with VAC staff in the following areas: program policy, strategic policy, field operations, program management (service delivery), as well as other subject matter experts within the Department including Senior Management. |
| Statistical Analysis | Statistical analysis included a review of VAC Facts and Figures, a review of client data stored in VAC’s Client Service Delivery Network (CSDN), and analysis of financial and operational data collected by VAC Finance Division and the program area. |
| File Review | A file review was completed to assist in evaluating the performance and efficiency of the VEF. All VEF applications for April, May, and June 2020 (405 applications) were reviewed to determine how effective the program was at providing assistance during the first three months of the Covid-19 pandemic (declared a public health emergency in all provinces). The evaluation team also reviewed additional files to identify whether the VEF is being used by family members and whether the VEF is encouraging Veterans to contact the department for the first time. |
| Survey | A survey was disseminated to all frontline decision makers (Veterans Service Agents, Case Managers, and Veteran Service Team Managers). Recipient feedback has been used as a data source. The survey was completed by 286 frontline staff of 715 contacted (40% response rate), providing a confidence level of 95% +/- 5%. See Appendix B for survey questions. |
2.3 Considerations and Limitations
- The VEF is a relatively new benefit implemented in April 2018. Baseline data has been collected, however, it is still early in the program life cycle, limiting year-over-year trend analysis.
- VAC’s Client Service Delivery Network (CSDN) limited the evaluation team’s ability to access historical data. While CSDN can provide a point-in-time picture of the program’s client population on the date the data is requested, the data does not depict the benefit amounts each Veteran was receiving on the date of VEF application. Also, the data does not show which indicators on the system might have been active on the date of application, but may have since been removed (e.g., a homeless indicator may be active on the date of VEF application, but will not show up in CSDN data if it has since been removed due to the Veteran finding permanent accommodations).
- CSDN payment data is not completely accurate as it often does not reflect changes in the amount disbursed where the amount required was more/less than that identified at the time of decision, or when a VEF payment is returned to VAC. To mitigate, the evaluation team reviewed financial data collected by the program area, with supplemental figures provided by VAC’s Financial Division.
- Decision Dockets are only available on the Veteran’s Notebook on CSDN and family members who apply for the VEF are tied to the Veterans decision docket. Data queries return information for the Veteran only, regardless of who received the VEF. Therefore, the ability to accurately determine whether it is a Veteran, spouse, child, or survivor who initiated a VEF application cannot be determined.
- The use of decision dockets tied solely to the Veteran also makes determining the number of applications per Veteran difficult. In some cases where the data shows a Veteran made multiple applications, the Veteran made may have made one application while another application was made by a spouse, child or survivor.
- In the initial field work stage of the evaluation, the evaluation team consulted with various areas across the organization to gain an in-depth understanding of the VEF and how the program is administered. Restrictions imposed mid-fieldwork due to the Covid-19 pandemic impacted the evaluation team’s ability to complete interviews and conduct on-site observation of VEF claim processing/payment. As a mitigation strategy, a survey of frontline employees was conducted to build on information gathered through early interviews.
- VEF recipients were not contacted as part of the evaluation. However, results of interviews conducted with VEF recipients by VAC’s Business Intelligence Unit (BIU) and subsequent analysis were reviewed as part of the evaluation.
- The evaluation team did not fully assess other interventions that may be available for Veterans in crisis. A survey conducted of frontline staff identified many federal, provincial, and community resources that can assist Veterans in crisis, depending on their needs. See Appendix C for a list of resources VAC frontline staff identified in the decision maker survey.