2.0 Scope and Methodology

2.0 Scope and Methodology

The evaluation was conducted in accordance with the directive and standards specified in Treasury Board of Canada’s 2016 Policy on Results. It covers the time frame from April 1, 2012 to March 31, 2016, and was conducted between September 2016 and June 2017. The evaluation covered four components: Commemorative Ceremonies and Events; Commemorative Partnership Program; Learning and Public Information; and Memorials.

Treasury Board of Canada Secretariat’s 2016 Directive on Results outlines standards to be considered for evaluations of federal Programs: continued need; alignment with Government priorities; alignment with federal roles and responsibilities; performance and efficiency; and economy. Table 2 details how the evaluation covered those issues.

Table 2 – Evaluation Scope
Relevance
  • Continued need for all programs/initiatives to ensure they are evolving to meet expectations of Canadians/Veterans/stakeholders.
  • Alignment with Government priorities and Federal roles and responsibilities for all Programs.
Performance
  • Mechanisms and tools in place to measure performance.
  • Achievement of immediate, intermediate and ultimate outcomes.
  • Unintended impacts associated with the programs/initiatives (where applicable).
  • Adequacy of processes, including identification of duplication/overlap.
Efficiency and Economy
  • Demonstration of efficiency and economy

The 2017 major ceremonies and events associated with the 100th anniversary of the Battle of Vimy Ridge fell outside the period of the evaluation. However, the evaluation team attended national Vimy events in Ottawa during April 2017 to observe/assess how national events are contributing to the public acknowledgement of Veterans and those who died in service.

Construction of a new Visitor Education Centre at the Canadian National Vimy Memorial site was completed in April 2017, outside the scope of the evaluation. The evaluation team did not examine the previous centre that operated during the period of the evaluation as it was scheduled to be decommissioned.

In the upcoming Overarching Evaluation of Commemoration, the overall governance of all commemoration activities will be assessed.

2.1 Multiple Lines of Evidence

The research methodology incorporated multiple lines of evidence, ensuring reliability of collected information and reported results. The lines of evidence used to evaluate each component’s relevance, performance, efficiency, and economy are shown in Table 3.

Table 3 – Sources of Information Reviewed During the Program Evaluation
Methodology Source
Non-departmental document reviews Key non-departmental documents such as: Parliamentary reports, Federal Budget documents, Speeches from the Throne, and evaluations of commemoration programs in other federal departments.
Interviews and/or work observation Approximately 50 interviews were conducted with VAC senior management; VAC staff involved in the delivery of commemorative programs; VAC partners and external departments delivering similar programs; and other subject matter experts. Regional and national ceremonies/events were observed as were internal work flows/procedures and external print production.
Departmental documentation and secondary research review The following types of departmental documents/information were reviewed to gain an understanding of the components, their authorities and requirements, and key issue areas: Memoranda of Understanding and their annexes; previous audits and evaluations; policies; business processes; records of decisions; strategic documents; planning documents, performance reports; research papers; survey results; third-party contracts; invoices; media articles; and complaints.
Statistical analysis Financial, demographic, gender (Gender Based Analysis +), and operational data collected by VAC for fiscal years 2012-13 to 2015-16 was examined. Financial and operational data for like-Programs delivered by external departments was also examined.
Site visits Site visits were conducted in Ottawa to observe and speak with VAC staff, program delivery partners, and stakeholders. Site visits also took place to observe national memorials in the National Capital Region (NCR) and to observe Veterans’ Week 2016 and Vimy 100 commemorative events in the NCR.
Survey A survey was completed by 67 VAC staff who self-identified as Canadian Armed Forces (CAF) Veterans or still-serving members. The survey (referenced in Annex A) was used to gain feedback on Canada Remembers Program components to determine relevance and effectiveness. The survey response rate was close to 60% (67 responses out of 120 contacted).
File Review A file review was completed using a judgemental sample of 40 Veterans whose deaths occurred during the period of the evaluation and within two years of completion of active duty. The review was conducted to determine whether eligible Veterans were being memorialized in the "Books of Remembrance".

2.2 Limitations and Analytical Challenges

The following limitations and challenges were identified:

  • The evaluation team did not speak directly with Commemorative Partnership Program applicants nor participants in commemoration events. Where available, the evaluation team used existing public opinion research and recipient feedback forms. The team also observed commemoration events in Ottawa during Veterans’ Week 2016 and Vimy 100 in April 2017.
  • Limited detailed statistical data was available for the Commemorative Ceremonies and Events component. To mitigate this limitation, the evaluation team utilized interviews and observations to the extent possible.
  • The evaluation team did not directly observe overseas events, memorials, or student guides. To the extent possible, the evaluation team observed memorials through photographs and online tours (Canadian National Vimy Memorial Google-tour). Interviews, internal documents, and media coverage were used to form the basis for analysis of the student guides and overseas events.

The above limitations should be considered when reviewing the evaluation findings.

 
Date modified: