2.0 Scope and Methodology

2.0 Scope and Methodology

The Government of Canada’s Policy on Results requires that every department conduct evaluations of all ongoing programs of grants and contributions that have a five-year average actual expenditure of $5 million or greater per year. This evaluation was calibrated in a manner that is appropriate for the risk, complexity and materiality of the program.

The Program was last evaluated in 2014 and the conclusion, at that time, was that the Program was relevant, effective and efficient with no recommendations required. In addition, the Terms and Conditions for the Program were last renewed in 2005 and are valid until March 31, 2021. This evaluation will support the renewal of the Terms and Conditions in 2021.

The Evaluation covered the time period of April 1, 2014 to September 30, 2018.

In support of developing the scope for the evaluation, the following activities were completed:

  • Preliminary interviews with key informants including staff from Head Office and Field Operations.
  • A document review, including; policy/business processes/procedures, relevant legislation/regulations, internal and external reports, media articles, and the Minister’s Mandate letter.
  • A review and analysis of available program data including a Gender Based Analysis (GBA+) analysis of available program data.

Programs such as the Health Care Benefits Program, the Long-Term Care Program and the Veterans Independence Program, which can be obtained via a favourable decision, were excluded from the scope of this evaluation.

2.1 Multiple Lines of Evidence

The evaluation incorporated multiple lines of evidence to ensure the reliability of collected information and reported results. The lines of evidence used to evaluate the Program are shown in Table 2.

Table 2 - List of Methodologies
Methodology* Source
Departmental Documentation and Secondary Research Review The following departmental documents/information were reviewed to understand the program objectives/intent, their authorities and requirements, complexity, context and any key issue areas: departmental planning documents, previous audits and evaluations, legislation, regulations, policies, business processes, strategic documents, performance reports, and the National Client Survey results.
Non-Departmental Document Reviews Various non-departmental documents were reviewed such as: program literature from both federal and provincial government departments/agencies, Parliamentary reports, Budget Speeches, and Speeches from the Throne.
Interviews Interviews (nine individuals) were conducted with VAC management and staff involved in the management and operations of the Program; and subject matter experts.
Statistical Analysis Financial and operational data collected by VAC for fiscal years 2014-15 to 2017-18 was analysed, where available. Client forecasts and demographic data was also analysed.
File Review A file review of 182 applicants, whose applications received unfavourable decisions between April 1, 2017, and September 30, 2018, was conducted to develop a profile of the reasons why the decisions were unfavourable. The random sample size was calculated using a 95% confidence level.

* See Appendix A for the full evaluation matrix.

2.2 Considerations

  • The Program is a long established program which has evolved over the years to accommodate the needs of War Service Veterans, qualified civilians and their survivors, however, there have been no major changes implemented during the review period.
  • The most recent evaluation, published in March 2014, confirmed that the Program is relevant, effective and efficient and no recommendations were required.
  • Administrative costs for the Program were less than 0.2% of all VAC program administration costs and therefore, further analysis of administrative costs was not undertaken due to the low risk and low materiality of the Program.

2.3 Limitation

  • The evaluation team did not speak directly with individuals accessing the Program. Where available, the evaluation team used existing public opinion research, interviews with key informants and other available sources such as a file review of recent applications with unfavourable decisions.

This limitation had minimal impact on the evaluation process due to the longevity of the Program, minimal recent changes, and availability of other relevant sources such as key informant interviews.

 
Date modified: