2.0 Scope and Methodology

2.0 Scope and Methodology

The evaluation was conducted in accordance with the directive and standards specified in Treasury Board of Canada’s 2009 Policy on EvaluationFootnote 11. It covers the time frame from April 1, 2012 to March 31, 2016, and was conducted between April 2016 and January 2017. The evaluation covered three programs: Funeral and Burial, Cemetery and Grave Maintenance, and Honours and Awards.

Treasury Board of Canada Secretariat’s 2009 Directive on the Evaluation Function outlines five “core issues” to be addressed in all evaluations of federal programs: continued need, alignment with Government priorities, alignment with federal roles and responsibilities, performance, and efficiency and economy. Table 2 details how the evaluation covered those issues.

Table 2: Evaluation Scope
Relevance
  • Continued need for all three programs; based on intake, awareness and changes since the last evaluation.
  • Alignment with Government priorities for all three programs; assessment was calibrated and used relevant work completed in the previous evaluation.
  • Alignment with federal roles and responsibilities for all three programs; assessment was calibrated and used relevant work completed in the previous evaluation.
Performance
  • Achievement of outcomes; focusing on the outcomes for the Cemetery and Grave Maintenance Program and for the Funeral and Burial Program as well as a quality assurance assessment on whether the program is being administered as intended, particularly in relation to Matter-of-Right and Means Testing decisions pertinent to the latter.
Efficiency and Economy
  • Demonstration of efficiency and economy; focusing on administration of Veteran grave maintenance; efficiencies gained in the administration of the Funeral and Burial Program since the last evaluation; and efficient processing associated with the Funeral and Burial and Honours and Awards Programs.

The Honours and Awards Program is a component of VAC’s overall Ceremonies and Events Program; therefore, outcomes and economy associated with the former will be assessed to a greater extent during a 2017-18 evaluation of the latter. Similarly, VAC’s Cemetery and Grave Maintenance Program forms a part of VAC’s broader Memorial and Cemetery Maintenance program; the Memorial portion of this program will be assessed in 2017-18.

2.1 Multiple Lines of Evidence

The research methodology incorporated multiple lines of evidence, ensuring reliability of collected information and reported results. The lines of evidence used to evaluate the Programs’ relevance, performance, efficiency and economy are shown in Table 3.

Table 3: Sources of Information Reviewed During the Program Evaluation
Methodology Source
Non-departmental document reviews Various non-departmental documents such as: Parliamentary reports, Budget speeches, Speech from the Throne, program literature from other countries, and published journals relating to Veterans’ interests were reviewed.
Interviews and/or work observation More than 40 interviews were conducted with VAC management, staff involved in the delivery of the programs, provincial/municipal governments offering similar programs, and other subject matter experts. Interviews were also conducted with major third-party providers who play a role in the delivery of the programs.
Departmental documentation and secondary research review The following types of departmental documents/information were reviewed to gain an understanding of the Programs, their authorities and requirements, and key issue areas: Memoranda to Cabinet; Memoranda of Understanding and their Annexes; previous audits and evaluations; policies; business processes; performance reports; research papers; survey results; media articles; and complaints.
Statistical analysis Financial, demographical, and operational data collected by VAC and the Last Post Fund for fiscal years 2012-13 to 2015-16 were examined.
Site visits Site visits took place in Ottawa, Montreal and Halifax to observe and speak with staff and program delivery partners associated with all three programs. The evaluation team also visited 18 cemeteries located in Prince Edward Island, Nova Scotia and Quebec.
File review A file review was completed on a sample of 103 Veteran cases. In addition, judgmental samplingFootnote 12 was conducted on 113 files associated with Matter-of-Right decisions (Funeral and Burial Program) and issuance of Memorial Crosses (Honours and Awards Program). Further judgmental sampling (353 cases) was completed for specific medical conditions in relation to Matter-of-Right decisions.

2.2 Limitations and Analytical Challenges

The following limitations were identified:

  • The Budget 2016 change in the exemption amount for the Funeral and Burial Program may impact future demand. The evaluation used existing intake forecasts, interviews and statistical analysis to estimate future needs for the Program.
  • The evaluation team did not speak directly with program applicants. Where available, the evaluation team used existing public opinion research as well as internal and external studies. Interviews were conducted with representatives who deal directly with Veterans and their families to gauge the needs and views of Program recipients.
  • Limited detailed statistical data was available for the Honours and Awards Program for the evaluation scope period. This lack was partially mitigated by reviewing published data and conducting file reviews.
  • There is no Performance Measurement StrategyFootnote 13 (PMS) specific to the Honours and Awards Program; rather, it is included within the Ceremonies and Events PMS. Outcomes and economy will be assessed to a greater extent during an upcoming evaluation of the overall program.

The above limitations should be considered when reading the evaluation findings.

 
Date modified: