2.1 Evaluation objective and scope
This evaluation was conducted in accordance with VAC’s 2022-23 to 2026-27 Departmental Evaluation Plan, which was developed to align with Treasury Board of Canada Secretariat’s (TBS’s) Policy on Results. The policy stipulates that evaluations of all ongoing grant and contribution programs that have a five-year average expenditure of $5 million per year must be evaluated every five years. The policy also indicates that all organizational spending and programs in the Program Inventory must be considered within departmental evaluation planning based on risk and need.
The past performance of the Fund was reviewed along with current activities to determine program relevance, effectiveness, and economy. The evaluation was conducted from April 2022 to March 2023. The evaluation covers the time period from 1 April 2018 to 31 March 2022. When available and applicable, more up-to-date information was incorporated into the evaluation (e.g., observations of the Fund’s call for applications in Fall 2022 and the associated application assessment and funding decision process).
Since VAC’s 2021 Audit of the Veterans Family Well-Being Fund focused on governance, documentation, processes and tools, the evaluation placed less emphasis on the efficiency of the Fund, as staff continue to implement changes resulting from audit recommendations. Numerous recommendations in the area of efficiency were made in the audit, including updating processes, tools and supports for the Fund. Where appropriate, the evaluation highlighted early findings with regards to changes that have occurred as a result of addressing audit recommendations.
2.2 Evaluation questions
Upon completion of an initial review (interviews, document review, and data analysis), evaluation questions were developed to help assess program relevance and program performance (especially in the areas of effectiveness). A list of evaluation questions, highlighting the key lines of inquiry, can be found in Table 1 below.
Table 1: Evaluation questions
| Relevance |
| To what extent does the Fund address a need, now and in the future? |
| To what extent does the Fund align with Government of Canada priorities and departmental roles and responsibilities? |
| Performance |
| Have outcomes been identified and measured to determine the Fund’s success and support program management? |
| Are there any unintended impacts resulting from the Fund (positive or negative)? |
| Are there opportunities to improve the effectiveness of the Fund? |
| To what extent are Fund resources used efficiently and economically? |
2.3 Evaluation methodology
The Evaluation was summativeFootnote4 in nature and relied on a mix of qualitative and quantitative data sources. The methodology incorporated multiple lines of evidence to ensure reliability of collected information and reported results. This information is further outlined in Table 2 below.
Table 2: List of evaluation methodologies and sources
| Methodology | Source |
|---|---|
| Departmental Documentation and Secondary Research Review | The following departmental documents/information were reviewed to better understand the program objectives/intent, their authorities and requirements, complexity, context and any key issue areas: departmental planning documents, central agency submissions, previous audits, terms and conditions, business processes, guidelines, strategic documents, performance reports, and internal tracking documents. |
| Non-Departmental Document Review | Various non-departmental documents such as, parliamentary reports and transcripts, legislation, budget speeches, media documents, and Speeches from the Throne were reviewed for context purposes. |
| Interviews | 42 Interviews (19 planning interviews and 23 fieldwork interviews) were conducted with VAC staff (including senior management, present and past program managers, head office employees, as well as other subject matter experts within the department) and a sample of representatives from organizations who submitted successful funding applications. |
| Data analysis | Data analysis of available financial and operational data collected by VAC pertaining to the Fund since 1 April 2018. |
| Observation | Observation of the 2022 project assessment process was conducted with Program staff. As travel restrictions allowed, site visits occurred in Halifax, Nova Scotia, Stratford Prince Edward Island, and Summerside Prince Edward Island. The site visits provided an opportunity to meet Fund recipients, gather information, observe recipient projects, discuss the funding application process, the funding distribution process, and processes relating to reporting and corresponding with the Fund. |
| File review | A file review of completed funded initiatives (35 projects) was done to assist in evaluating the effectiveness of the Fund. Findings from the review were used to assess the success of funded projects completed to date, help determine whether the Fund is meeting its objectives as set out in the original terms and conditions, and measure progress towards achieving program outcomes. |
2.4 Considerations and limitations
The evaluation identified the following considerations and limitations:
- The evaluation team consulted with various areas across the department to gain an in-depth understanding of the Fund, the original intent of the Fund, how it operates, what is working well, and where there may be areas for improvement.
- The evaluation team was able to visit select Fund recipients in Nova Scotia and Prince Edward Island, and then conducted virtual interviews with other organizations, as scheduling logistics and air travel as well as COVID-19 continued to provide risks and challenges with travel.
- Gender based analysis (GBA+) was incorporated into the evaluation based on overarching project theme.
- There was an inconsistent level of information to fully assess Fund success. There was a limited set of completed project results. Many projects are multi-year projects, with only one third of projects considered completed. Additionally, the scope, size and themes of projects varied significantly. The nature of the funding program (research and innovation) also presented challenges in defining and measuring ‘success’ (e.g., what is considered innovative/how measure success of innovation). To mitigate this challenge, the evaluation looked at multiple information sources (e.g., file review, interviews with various departmental staff and funding recipients, and document review). Where available, the evaluation team noted any short-term, observable indications of success.
- Reporting for the Fund is primarily manual in nature. Throughout the evaluation, the team noted a lack of documented statistical information. Where possible, the information was created by the program area. The evaluation team noted discrepancies in some instances, which were highlighted with the program area.
- The Fund is still maturing and has seen many changes in funding levels, processes, and management since its inception in 2018. This was taken into consideration when evaluating the effectiveness and efficiency of the Fund to date.
- The VAC Research Funding Program was launched by the department in April 2020 and is further elaborated in section 3.2.1, Alignment of Government of Canada priorities. This new program is separate from the Veteran and Family Well-Being Fund and was not considered in scope for this evaluation. It will be evaluated at a later date. Some findings in this evaluation may be linked or impact the Research Funding Program, as there are some similarities among the two funding programs.
The above noted information should be taken into consideration when reading this evaluation report.