2.0 Scope and Methodology

2.0 Scope and Methodology

The overarching evaluation assesses Commemoration from a strategic perspective, providing information on the linkages between programs, and capturing items that the previous individual evaluations (Evaluation of Commemorative Benefits and Services and Evaluation of Public Recognition and Awareness) were not able to fully assess and report on.

The evaluation was conducted in accordance with the directive and standards specified in Treasury Board of Canada’s 2016 Policy on Results. It covers the time frame from April 1, 2012 to September 30, 2017, and was conducted between September 2017 and December 2017.

Table 1: Evaluation Themes
Planning and Funding
Long-term strategic planning
Program funding
Funding pressures
Commemorative Integrity
Canadian National Vimy Memorial, Beaumont-Hamel Newfoundland Memorial
Alignment
Opportunities for enhanced alignment internally - Departmental
Opportunities for enhanced alignment internally - Divisional
Management/Organizational Structure
Do the current structures facilitate efficient and effective decision making for Commemorative objectives/priorities?
Opportunities
International comparison
Best practices

2.1 Methodology

The research methodology incorporated multiple lines of evidence, ensuring reliability of collected information and reported results. The evaluation was formativeFootnote 2 in nature and relied on a mix of qualitative and quantitative lines of evidence, including evidence gathered in previous commemorative evaluations, as well as the Audit of Overseas Ceremonies and Events, as shown in Table 2.

Table 2: Sources of Information Reviewed During the Program Evaluation
Methodology Source
Non-departmental document reviews Key non-departmental documents such as: Parliamentary reports, Federal Budget documents, Speeches from the Throne, media articles, and evaluations of commemoration programs in other federal departments.
Interviews and/or work observation Approximately 20 interviews with key informants were conducted during the fieldwork phase of this evaluation. In addition, this report was informed by approximately 40 interviews conducted during the Evaluation of Public Recognition and Awareness and the Evaluation of Commemorative Benefits and Services. Interviews were conducted with VAC senior management; VAC staff involved in the delivery of commemorative programs; VAC partners and external departments delivering similar programs; and other subject matter experts. Direct observations were conducted of VAC memorial sites in Europe.
Departmental documentation and secondary research review The following types of departmental documents/information were reviewed to gain an understanding of the components, their authorities and requirements, and key issue areas: memoranda of understanding and their annexes; previous audits and evaluations; policies; business processes; records of decisions; strategic and business planning documents; performance reports; research papers; survey results; third-party contracts; invoices; social media; media reports; and correspondence.
Statistical analysis Financial and operational data collected by VAC for fiscal years 2012-13 to 2016-17 was examined.
Survey Results from the recent Survey of Commemorative Initiatives conducted during the Evaluation of Public Recognition and Awareness were examined. The survey was completed by 67 VAC staff who self-identified as Canadian Armed Forces (CAF) Veterans or still-serving members.

2.2 Limitations and Analytical Challenges

The following limitations and challenges were identified:

  • The evaluation team did not speak directly with Canadians. Where available, the evaluation team used existing public opinion research.

This limitation/challenge should be considered when reviewing the evaluation findings.