4.0 Performance

4.1 Performance Tools and Measurement

Performance measurement monitors the progress of programs towards their expected results.^{Footnote 14} A Performance Measurement Strategy is used to regularly measure key indicators and results. This information can be used to assess against expectations and to assist in measuring the effectiveness and success of a program. A Performance Measurement Plan and a program logic model are tools that support the Performance Measurement Strategy.

The evaluation team reviewed each component’s Performance Measurement Strategy, Performance Measurement Plan, and associated performance, confirming:

• Performance Measurement Strategies are approved/current;
• Performance Measurement Plans identifying the specific outputs, outcomes and associated performance indicators are in place; and
• Performance results are available, both through operational reporting and strong public opinion research.

During the evaluation period, a new Government of Canada Policy on Results came into effect (July 2016),^{Footnote 15} detailing the requirements for departments to clearly identify core responsibilities and associated departmental programs with these respective responsibilities. In support of the new policy, departments will develop Program Inventories and Performance Information Profiles to replace Performance Measurement Strategies. The Commemoration Division is currently in the process of developing these tools. Future evaluations will be informed by the Department’s updated performance measures/outcomes.

4.2 Effectiveness – Achievement of Expected Outcomes

Outcomes are the expected changes or differences that result from activities and outputs. Outcomes are described as immediate, intermediate, or ultimate based on the contribution/influence the component has on each outcome. As outlined by the Treasury Board Secretariat (TBS) guideline, Supporting Effective Evaluations: A Guide to Developing Performance Measurement Strategies^{Footnote 16}:

• immediate outcome(s) should equate to a “change in awareness, knowledge or skill;”
• intermediate outcome(s) should equate to a “change in the target population’s behavior;” and
• ultimate outcome(s) should equate to a “change of state in a target population.”

When the above outcomes are met, the evaluated components contribute to the Department’s Strategic Outcome #2: Canadians remember and demonstrate their recognition of all who served in Canada’s efforts during war, military conflict, and peace.

This section of the report assesses the extent to which outcomes are being achieved across the four components evaluated.

Immediate Outcome #1: Commemorative Ceremonies and Events
Increased awareness of ceremonies and events that mark Canadian military milestones.

The evaluation finds that social media is being used effectively to increase awareness of national and overseas events for significant commemorative milestones. The Commemoration Division, in collaboration with the social media team in VAC’s Communications Division, has undertaken significant efforts to increase awareness through various social media platforms, including Facebook, Twitter, Instagram, and YouTube. Examples of these efforts include:

• advertising/promoting upcoming major events;
• live video streaming commemorative events (e.g., Facebook Live stream of Vimy 100 event);
• youth initiatives (e.g., youth overseas delegation “takeover” of VAC Instagram account); and
• still image social media posts during and post events.

A social media review completed by the evaluation team finds that VAC efforts are reaching a significant number of Canadians and youth (close to one million Canadians through Facebook alone). In addition, a review of posts on VAC social media pages found that overall comments were positive and that VAC sites are a useful channel for Canadians to show their appreciation for Veterans.

Though social media efforts are reaching a significant number of Canadians, and promotion of overseas and national events is quite prominent, promotion of regional and community commemorative events in Canada requires improvement.

According to the Attitudes Towards Remembrance and Veterans’ Week Survey 2016, the number of Canadians and their families who did not participate in Veterans’ Week activities due to a lack of awareness has been increasing. Further results are reflected in Table 8.

Table 8 – Attitudes Towards Remembrance and Veterans’ Week Survey Results: Non-participation due to Knowledge/Awareness

Results	2012 Survey	2014 Survey	2016 Survey
Percentage of Canadians and their families who did not participate in Veterans’ weeks that indicated they did not participate due to a lack of knowledge/awareness.	18%	24%	26%

The VAC National Survey 2017^{Footnote 17} asked Veterans and survivors to identify what VAC could do to reach more Canadians through remembrance initiatives. Responses were provided by 604 Veterans/survivors. An analysis of the results shows that over 60% of responses indicated a need for greater advertising/information sharing regarding remembrance initiatives. Over 100 of the respondents specifically pointed to the need for VAC to “focus on providing relevant information to Veterans (e.g. upcoming events).”

The primary tool used by VAC to promote regional and community commemorative events in Canada is the online calendar of events.^{Footnote 18} Both VAC’s website and VAC’s National Contact Centre Network (NCCN) refer Canadians, including Veterans, looking to participate in commemorative events/activities to VAC’s online calendar of events. During the period of the evaluation, the annual number of events posted to the calendar of events decreased 33% from 2012-13 to 2014-15.^{Footnote 19} Calendar of events page views show a similar downward trend, decreasing by 25%, from 71,164 visits in 2012-13 to 53,585 in 2015-16.

In addition, the evaluation team reviewed the calendar during the period leading up to Veterans’ Week 2016 and observed that there was a lack of information with respect to regional and community events. This observation held true for events funded in part by VAC (Commemorative Partnership Program) and events being led/supported by VAC. It is estimated that VAC led, partnered, supported and/or participated in close to 700 events per year during 2014-15 and 2015-16. The evaluation team finds there is an opportunity for the department to better inform Veterans and the Canadian public of events in their communities and to highlight VAC’s support of commemorative initiatives.

A review of VAC’s Canada Remembers Facebook page identified limited promotion of VAC’s calendar of events. As this Facebook page has close to one million followers, this is a missed opportunity to increase awareness of national, regional, and community events that recognize the contributions/sacrifices of Canada’s Veterans.

The evaluation team observed that efforts were increased to update and promote the calendar of events via social media in support of Vimy 100 events occurring in April 2017. These efforts were as a result of a temporary dedicated resource. The evaluation team was informed that no changes in processes have occurred that would ensure this approach would continue for future national, regional, or community events.

The evaluation team interviewed Canadian Heritage staff in relation to their Celebration and Commemoration Canada Program (which also offers grants and contributions to Canadian community organizations). The team was informed that processes exist at Canadian Heritage to ensure adequate coverage/inclusion of events being funded across Canada. These processes included internal partnerships with their Information Technology division to have calendars/maps updated based on information obtained through the program’s application process.

Based on this information, the evaluation team finds that improvements are required to increase information provided to Canadians, including Veterans, pertaining to community and regional commemorative events in Canada.

Recommendation #1

It is recommended that the Director General, Commemoration, build on current practices to enhance awareness of commemorative events in Canada.

Management Response:

Management agrees with this recommendation.

Management Action Plan:

Corrective action to be taken	OPI (Office of Primary Interest)	Target Date
The Director General, Commemoration, will implement measures, such as increased utilization and promotion of the calendar of events and social media, in order to enhance awareness of commemorative events in Canada.	Director General, Commemoration	August 2018

Immediate Outcome #2: Commemorative Ceremonies and Events
Increased appreciation for the efforts of Canadian Veterans and those who died in service to Canada.

Increased appreciation and interest for the efforts of Canadian Veterans and those who died in service to Canada is evident in the growing following of VAC social media accounts (further details in Table 12 of report). Through social media, VAC engages Canadians and provides a forum for the public to show and share their appreciation and interest in the efforts of Canadian Veterans. The evaluation team confirmed that comments on the Canada Remembers Facebook page are positive, respectful, and appreciative of Canadian Veterans and their sacrifices. VAC-produced content carried through social media, such as Facebook Live events, consistently draws interest from the Canadian public (as previously discussed in section 3.1.1).

Increased appreciation for the efforts of Canadian Veterans from the Canadian public is also evident in the results of the Attitudes Towards Remembrance and Veterans’ Week Surveys conducted during the period of the evaluation (Table 9).

Table 9 – Canadian Appreciation for Veterans and Their Service

Percentage of Canadians who report that ...	2012 Survey	2014 Survey	2016 Survey
They agree/completely agree that Canada’s Veterans have made major contributions to the development of our country	86%	83%	89%
They agree/completely agree that Canada’s Veterans should be recognized for their service to Canada	91%	93%	96%
They agree/completely agree that they make an effort to demonstrate their appreciation for Veterans	66%	66%	73%

Source: Attitudes Towards Remembrance and Veterans’ Week Surveys 2012, 2014, and 2016

Immediate Outcome #3: Commemorative Partnership Program
Organizations lead remembrance activities.

The evaluation found that the Commemorative Partnership Program is supporting organizations to lead remembrance activities. During the period of the evaluation, approximately 95% of requested projects received funding (an average of 227 projects were approved annually). A breakdown of annual funds available for use, the annual funds provided to organizations, and the number of projects funded annually is reflected in Table 10.

Table 10 – Commemorative Partnership Program (CPP) and Community War Memorial Program (CWMP)^{Footnote 20} Expenditures 2012-13 to 2015-16

CPP and CWMP Grants and Contributions (Gs and Cs)	2012-13	2013-14	2014-15	2015-16
CPP Grant – funds available for use	N/A	N/A	$750,000*	$750,000
CPP Grant – actual expenditures	N/A	N/A	$654,266	$653,113
CPP Grant variance (funds available minus actual expenditures)	N/A	N/A	$95,734	$96,887
Number of projects funded through CPP Grants	N/A	N/A	181	167
CPP Contributions – funds available for use	$2,310,000	$2,910,001	$2,548,000	$3,066,326
CPP Contributions – actual expenditures	$2,136,169	$2,140,321	$1,648,821	$3,066,326**
CPP Contributions variance (funds available minus actual expenditures)	$173,831	$769,680	$899,179	$0
Number of projects funded through CPP Contributions	142	185	58	69
CWMP Contributions – funds available for use	$1,500,000	$1,250,000	$1,252,402	$286,674
CWMP Contributions – actual expenditures	$753,989	$747,598	$724,166	$278,366
CWMP Contributions variance (funds available minus actual expenditures)	$746,011	$502,402	$528,236	$8,308
Number of projects funded through CWMP Contributions	26	33	31	14
Total: Funds available for CPP and CWMP Gs and Cs	$3,810,000	$4,160,001	$4,550,402	$4,103,001
Total: Actual Expenditures for CPP and CWMP Gs and Cs	$2,890,158	$2,887,919	$3,027,253	$3,997,805
Total: Variance all CPP and CWMP Gs and Cs (funds available minus expenditures)	$919,842	$1,272,082	$1,523,149	$105,195
Total: Number of projects funded through CPP and CWMP Gs and Cs	168	218	270	250
Percentage of funds utilized	75.9%	69.5%	66.6%	97.5%

Source: VAC Finance Division, and the Public Accounts of Canada, with analysis by Evaluation Team

*New Commemorative Partnership Grant established in 2014-15 for low-risk projects valued at less than $10,000

**Contributions under CPP expenditures for 2015-16 include $1,500,000 for 2017 Invictus Games.

Overall results show that, on average, between 2011-12 and 2015-16:

• $4,155,851 in annual funding was available to fund projects associated with remembrance activities.
• $3,200,784 in annual funding was utilized for funding projects associated with remembrance activities.

While utilization of Grants and Contributions resources for CPP and CWMP ranged from 66.6% to 75.9% in the first three years covered by this evaluation, 97.5% of funds were utilized in 2015-16. Through interviews, observations, and analysis of recent financial data, the evaluation team finds that a high rate of funds utilization will continue.

A review of VAC social media sites completed by the evaluation team found that VAC promotes the Commemorative Partnership Program on the VAC Services and Benefits Facebook page. However, limited promotion is occurring on the Canada Remembers Facebook page. This is a missed opportunity to target the large following that the Canada Remembers page has generated (approximately one million followers).

Immediate Outcome #4: Learning and Public Information
Increased access to learning and historical resources, interpretive programs and promotional products about Canada’s military history and remembrance.

On an annual basis, over four million print products are distributed to Canadians through the Learning and Public Information component. As identified in Table 11, the overall volume of demand has increased by approximately 5.2% between 2012-13 and 2015-16, the period of the evaluation.

Table 11 – Learning Product Demand Fiscal Years 2012-13 to 2015-16

Type of Product	2012-13	2013-14	2014-15	2015-16
Veterans’ Week Materials	3,704,533	3,718,496	4,041,812	4,011,212
Veterans’ Week Sample Packs*	132,000	132,000	132,000	132,000
Learning Resources	9,168	10,928	10,407	8,009
Booklets	17,793	16,640	8,891	17,657
Historical Sheets	94,238	109,875	73,387	69,949
Promotional Products (pins, poppy seeds, etc.)	343,449	243,572	301,930	287,265
Total	4,301,314	4,231,511	4,568,427	4,526,092

Source: VAC Commemoration Division
*22,000 sample packs containing six learning products each were distributed annually during the evaluation period.

A statistical analysis of learning resources ordered by geographical location in 2016 indicated that all provinces/territories with the exception of Ontario and Quebec represented a proportion of the learning resources orders within +/-3% of their portion of the Canadian population. The learning resources ordered from Ontario were proportionately 7% higher than its population size, whereas learning resources orders from Quebec were proportionately 14% lower than its population size. Interviews with VAC Education Officers confirmed that efforts are being made to enhance the reach of the learning resources within Quebec.

Learning resources are promoted on VAC’s website and through social media. In addition, an e-mail distribution list is maintained for the purposes of informing educators of upcoming product releases. Each year, following Veterans’ Week, VAC requests that instructors who received learning resources fill out a voluntary online evaluation form to provide the department with information to be used for performance measurement and continuous program improvement purposes. The evaluation team reviewed comments from educators and also analyzed responses to questions on the evaluation form. The evaluation concludes that Veterans’ Week materials are providing increased access to learning and historical resources about Canada’s military history and remembrance. During the period 2012-13 through 2015-16, the completed Veterans’ Week Online Evaluation forms identified the following:

• 98.5% of educators agreed that “The student materials were well-received and appropriate for the learning styles in my classroom.”
• 99% of educators reported that “I would encourage my students to take this material home to share with their families.”

A review of text comments provided by educators through the online evaluation form found overall positive responses; no negative trends or concerns were identified. Evidence indicates that suggestions and recommendations made by educators are noted by VAC Education Officers and taken into consideration when revising material for future years.

Though access to learning resources has increased, and no issues were identified with the quality of VAC print materials, areas for improvement have been identified.

Through interviews and observation, it was identified that the VAC website does not currently have a highly visible, easily accessible section through which to navigate the learning resources (e.g., learning portal). Simplifying the navigation of online learning resources could further increase access to this component.

In addition, the evaluation team was informed through interviews, and was able to confirm through documentation reviews and observation, that issues often arise in relation to the public online ordering website and inventory tracking for learning resources. VAC learning resources are printed and shipped under contract by a third party. The contracted company is responsible for maintaining an online ordering site and for providing up-to-date inventory/statistic reports. Through consultation with Commemoration staff and the print contract project authority, the evaluation team was advised that efforts are underway to correct these concerns.

The evaluation finds that VAC has increased access to learning and historical resources, interpretive programs and promotional products about Canada’s military history and remembrance through the use of VAC’s social media pages/accounts. As referenced in Table 12, VAC has capitalized on a worldwide trend for utilizing social media for promotion. This has resulted in the media reach increasing dramatically over the course of the evaluation period.

Table 12 – Social Media Reach

Social Media Initiative	March 2013	March 2014	March 2015	March 2016	Change
“Likes” Canada Remembers Facebook Page	680,727	773,133	904,527	987,378	+45%
“Followers” VAC Twitter Account	6,949	18,027	29,747	37,984	+447%
“Subscribers” VAC’s YouTube Channel	845	1,121	2,398	3,447	+308%
“Video Views” VAC YouTube Channel	277,744	342,098	2,487,783	3,676,472	+1,223%

Source: VAC Commemoration Division

With respect to interpretation, VAC’s student guides at the Canadian National Vimy Memorial and the Beaumont-Hamel Newfoundland Memorial conducted 6,436 tours for close to 65,000 visitors in 2016. In addition, in April 2017 (outside of evaluation scope), VAC opened a new Visitor Education Centre at the Canadian National Vimy Memorial to coincide with the 100^th anniversary of the Battle of Vimy Ridge.

Immediate Outcome #5: Memorials
Properly maintained and presented VAC memorials, cemeteries and grave markers on behalf of all Canadians.^{Footnote 21}

VAC has responsibility for 14 memorials located in France, Belgium, and the United Kingdom. These memorials are maintained through a contract and memorandum of understanding with the Commonwealth War Graves Commission valued at $1,975,000 annually.

The evaluation team obtained and reviewed memorial maintenance plans for each memorial site. The plans are used by the Commonwealth War Graves Commission to assess the state of maintenance at each memorial and to determine what activities should be undertaken as part of their annual work agreement. The annual work agreements were found to be thorough, with cost breakdowns where applicable.

Through interviews and documentation review, the evaluation team identified that although work agreements are being approved on an annual basis, VAC does not have formalized review practices in place to ensure maintenance is occurring in accordance with these agreements. The evaluation team finds that review requirements should be implemented to ensure adequate oversight of maintenance and presentation work. As indicated in Section 2.2 – Limitations and Analytical Challenges, on-site observation of memorials in Europe did not occur.

Recommendation #2

It is recommended that the Director General, Commemoration, review and update current practices and procedures to ensure the appropriate maintenance and monitoring of memorial sites in Europe occurs in accordance with annual plans.

Management Response:

Management agrees with this recommendation.

Management Action Plan:

Corrective action to be taken	OPI (Office of Primary Interest)	Target Date
The Director General, Commemoration, will review and update current practices, and implement measures to ensure memorial site maintenance work is completed in accordance with the priorities and expectations outlined in maintenance plans.	Director General, Commemoration	April 2018

Through in-person observations of the Books of Remembrance and the Memorial Chamber, the evaluation team found that these memorials are also being properly maintained. Interviews identified that care is taken to ensure the Books of Remembrance are properly handled. Searches were carried out to test the functionality of the online Books of Remembrance and the Canadian Virtual War Memorial; no issues were identified. A concern relating to the processes associated with entries to the physical Books of Remembrance is highlighted in Section 4.3.4 – Memorials.

Interviews in relation to memorials in Europe highlighted that no formal processes exist to collect visitor feedback at memorial sites. Interviews with Parks Canada identified that National Historic Sites in Canada use tools to collect visitor experience feedback. A recommendation associated with visitor experience is included in Section 4.3.3 Learning and Public Information.

In addition, the evaluation team was informed that the forestry plan for the Canadian National Vimy Memorial has not been fully acted upon, and that further planning and decisions are required regarding tunnel preservation. Further assessment of VAC’s memorials will occur as part of the upcoming overarching evaluation on Commemoration.

Opportunities for Improvement:

Key informant interviews identified that there are opportunities to enhance interpretive information at memorial sites in Europe. It was also brought to the evaluation team’s attention that electronic visitor counters at the Vimy Memorial have not been operational since September 2016, jeopardizing the accuracy of visitation statistics.

The evaluation team reviewed departmental responsibilities in relation to the Government of Canada Remembrance Policy. The policy states that VAC is responsible to “Ensure maintenance, preservation and presentation of Canada’s international monuments and memorials and battlefields.” The evaluation team reviewed the relevant Performance Measurement Strategy and Performance Measurement Plan and found that the “preservation” component of this responsibility is not referenced. The evaluation team was informed by Commemoration staff that, in support of the 2016 Policy on Results, a Performance Information Profile for the Memorial component is being developed, and that “preservation” related activities and measures will be included.

Intermediate Outcome #1: Commemorative Ceremonies and Events
Veterans, those who died in service, and Canadian military milestones are publically acknowledged regionally, nationally and internationally.

VAC-led and supported regional, national, and international events contribute to the public acknowledgement of Veterans, those who died in service, and Canadian military milestones.

Interviews highlighted that, each year, six commemorative events are organized by Commemoration staff located at the Canadian National Vimy Memorial and the Beaumont-Hamel Newfoundland Memorial in France. These events recognize anniversaries of military significance. In addition, overseas events marking major anniversaries (e.g., 100^th and 75^th anniversaries) are organized by Commemoration staff located at the VAC Head Office in Charlottetown, Prince Edward Island. Major anniversaries that occurred during the evaluation period are highlighted in Table 13.

Table 13 – List of Major Overseas Events 2012-16

Year	Overseas Commemoration Event
2012	70th anniversary of the Dieppe Raid 95th anniversary of the Battle of Vimy Ridge
2013	60th anniversary of the Korean War Armistice
2014	70th anniversary of D-Day and the Battle of Normandy
2015	70th anniversary of the Liberation of the Netherlands
2016	100th anniversary of the Battles of the Somme and Beaumont-Hamel

Source: VAC Commemoration Division

The 2017 major ceremonies and events associated with the 100^th anniversary of the Battle of Vimy Ridge fell outside the period of the evaluation. However, the evaluation team observed national events in Ottawa during April 2017 and confirmed that attendance at such events in-person and on-line contributes to the public acknowledgement of Veterans and those who died in service (see Section 3.1.1 Continued Need for the Programs, Table 5 – Facebook Live Participation in Events Observed by Evaluators for figures on ceremony reach).

The evaluation team obtained manual tracking statistics from VAC’s domestic events team for fiscal years 2014-15 and 2015-16. The documentation indicates that:

• on average, VAC led, partnered, supported, and/or participated in 678 events per year;
• 94.5% of events were regional/community-based events; and
• 5.5% of events were national in scope.

Statistics for further years were unavailable, therefore, a trend analysis could not be conducted. As recommended in Section 5.1 of the evaluation, improvements are required to documentation/tracking associated with ceremonies and events.

Intermediate Outcome #2: Commemorative Partnership Program
Canadians participate in remembrance.

The number of Commemorative Partnership Program projects, which provide opportunities for Canadians to participate in remembrance, and the funding provided are shown in Table 14. As identified, the number of funded projects increased from 168 in 2012-13 to 250 in 2015-16 (a 49% increase).

Table 14 – Commemorative Partnership Program Expenditures

Expenditure Area	2012-13	2013-14	2014-15	2015-16
Commemorative Partnerships Contributions	$2,136,169	$2,140,321	$1,648,821	$3,066,326*
Commemorative Partnerships Grants	N/A	N/A	$654,266	$653,113
Community War Memorial Contributions	$753,989	$747,598	$724,166	$278,366
Total Program Funds Utilized – Grants and Contributions	$2,890,158	$2,887,919	$3,027,253	$3,997,805
Projects Funded	168	218	270	250

Source: VAC Finance Division and Public Account of Canada

*2015-16 Contribution Payment expenditures include a one-time payment of $1,500,000 in support of the 2017 Invictus Games.

The evaluation team analysed data collected from program evaluation forms completed by funding recipients. Over the four-year period covered by the evaluation, 98% of funding recipients identified that their project was engaging to Canadians. In fiscal year 2016-17, the program area began using a new tool that, among other things, compiles data on program reach (e.g., the number of attendees at events, the number of people reached by learning resources, etc.). This information will better support future evaluations to assess the overall reach of the program.

The evaluation team observed a decision-making committee meeting in support of the Community Engagement Partnership element of this Program. The evaluation team found that key factors including geographical reach, diversity and youth impacts were considered during the decision-making process.

Intermediate Outcome #3: Learning and Public Information
Canadians, including youth, have increased knowledge and understanding about the sacrifices and achievements of Canadian Veterans and those who died in service.

The evaluation finds that the public information portion of the Learning and Public Information component is achieving its intermediate outcome. This component is responsible for developing learning resources for Canadians and youth such as Veterans’ Week materials, online and social media content, and Heroes Remember videos (videos of personal recollections of those who served), and for contributing to student programs such as the student guides in France and Encounters with Canada remembrance-themed weeks.^{Footnote 22}

Veterans’ Week online evaluation forms completed by educators who ordered VAC learning resources during the period of the evaluation indicate that:

• 99% of teachers agree that Veterans’ Week learning resources “…contributed to student’s understanding and appreciation of remembrance.”; and
• 99% of teachers agreed that “The learning material increased youth knowledge and understanding of the sacrifices and achievements of Canada’s Veterans and those who died in service.”

Further evidence that VAC initiatives foster increased knowledge and understanding about the sacrifices and achievements of Canadian Veterans is revealed in student evaluation forms completed by youth participating in Historica Canada’s Encounters with Canada Program.^{Footnote 23} VAC collaborates with Encounters with Canada/Historica Canada to offer a one-week remembrance-themed program to youth during Veterans’ Week each year.^{Footnote 24} Of the youth that participated in this program between 2014 and 2016:

• 87% reported that “As a result of the Canada Remembers theme week, I am likely to tell my friends about what I now know about Remembrance.”
• 97% reported that “After my experiences this week, I better understand the sacrifices and achievements of those who have served Canada during times of war and peace.”

The impact of the work of student guides on visitors could not be fully assessed due to a lack of visitor experience feedback. Additional performance feedback is required in order to assess the impact of guided tours and interpretation at the Canadian National Vimy Memorial and the Beaumont-Hamel Newfoundland Memorial. This finding is further reviewed in Section 4.3.3 Learning and Public Information of this evaluation.

Intermediate Outcome #4: Memorials
Canadians support properly maintained VAC memorials, cemeteries and grave markers.

VAC measures this outcome through its Attitudes Towards Remembrance and Veterans’ Week Survey. The survey asked participants if it is important that “VAC honour deceased Canadian Veterans and war dead by maintaining memorials, cemeteries and grave markers.” The results, as identified in Table 15, confirm that the vast majority of Canadians surveyed agree it is important for VAC to do so.

Table 15 – Canadian Support for the Maintenance of VAC Memorials

Percentage of Canadians who report that …	2012 Survey	2014 Survey	2016 Survey
It is important that VAC honour deceased Canadian Veterans and war dead by maintaining memorials, cemeteries and grave markers.	88%	89%	91%

Source: Attitudes Towards Remembrance and Veterans’ Week Surveys 2012, 2014, and 2016.

Public support for Veterans and Canada’s war dead is also evident in a November 2015 poll released by the Historica-Dominion Institute (now Historica Canada), which measured Canadians’ attitudes on remembrance and found that:

• Most (91%) Canadians ‘agree’ that Canada should do more to honour its Veterans; and
• Nine in ten (90%) ‘agree’ that Canadians should do more to honour those who fought and those who have died in war.^{Footnote 25}

Further support for VAC memorials is evident in visitation rates at Canadian memorials in Europe. As Table 16 shows, there are high volumes of visitation to the Canadian National Vimy Memorial and the Beaumont-Hamel Newfoundland Memorial. Visitation fluctuates on a yearly basis and can be influenced by major military milestones. VAC staff estimate that 15%-18% of visitors are Canadians.

Table 16 – Visitor Statistics Major Memorials

Memorials	2013	2014	2015	2016
Canadian National Vimy Memorial	687,389	775,759	790,878	460,096*
Beaumont-Hamel Newfoundland Memorial	133,457	223,279	179,045	181,425
Total	820,846	999,038	969,923	N/A

Source: VAC Commemoration Division
*Data available until end of August only due to on-site automated counter failure

Canadians’ support for memorials is also evident through the high volume of online visits to the Canadian Virtual War Memorial and the online Books of Remembrance. Requests for copies of pages from the Books of Remembrance received by the Office of the Sergeant at Arms of the House of Commons have also increased over the period of the evaluation ^{Footnote 26} (see Table 17).

Table 17 – Online Memorial Statistics

Channel	2012-13	2013-14	2014-15	2015-16
Number of visits to Canadian Virtual War Memorial	206,995	1,411,518	1,539,133	1,564,735
Number of visits to online Books of Remembrance	58,946	108,596	356,112	216,348
Number of print requests for pages from the Books of Remembrance	1,215	N/A*	4,126	3,893

Source: VAC Commemoration Division
*Page print request volumes unavailable for 2013-14.

Ultimate Outcome: All Programs
Canadian Veterans and those who died in service are honoured and the memory of their achievements and sacrifices is preserved.

The evaluation concludes that the components are contributing to the ultimate outcome of ensuring Canadian Veterans and those who died in service are honoured and the memory of their achievements and sacrifices is preserved. The components evaluated complement other Commemoration components such as the Funeral and Burial Program, Cemetery and Grave Maintenance, and Honours and Awards.

Public opinion research is the main line of measurements used by VAC to determine the ultimate outcome is being achieved. In VAC’s 2010 National Client Survey, the vast majority (88%) of VAC program recipients reported that they are satisfied with how the Department is delivering its remembrance program.

In addition, both the Attitudes Towards Remembrance and Veterans’ Week Survey 2016 and the Veterans Affairs Canada National Survey 2017 reveal that 76% of respondents feel VAC’s remembrance program “…effectively honours Veterans and those who died in service, and preserves the memory of their achievements and sacrifices”, exceeding the 70% target. A further breakdown of the Veterans Affairs Canada National Survey 2017 results is referenced in Table 18. As indicated, 85% of War Service Veteran respondents were satisfied with VAC’s commemorative initiatives, however, CAF Veterans, especially female Veterans were less likely to be satisfied.

Table 18 – 2017 VAC National Survey - Satisfaction with commemorative initiatives

Percentage of respondents who reported that they are “satisfied with the way VAC’s commemorative initiatives honour those who served our country and preserve the memory of their sacrifices.”	Male	Female	Total
War Service Veterans (n=200)	86%	80%	85%
Canadian Armed Forces Veterans (n=707)	72%	65%	72%

4.3 Does VAC have adequate/appropriate processes in place to ensure the Programs are administered as intended?

4.3.1 Commemorative Ceremonies and Events

Planning and post-mortem documents for overseas events were shared with the evaluation team. A review of the documents indicated that opportunities for further process improvements (lessons learned) are considered. Although documentation is compiled for overseas events, it is not always completed in a consistent and/or formal manner.

The evaluation team found that process changes are required in order to determine if the domestic ceremonies and events portion of this component is being administered as intended. The evaluation team observed domestic events during Veterans’ Week 2016. Although the events were found to be contributing to the components’ outcomes (raising awareness and recognizing Veterans’ achievements/sacrifices), follow-up reports documenting resources used, opportunities for improvement, and lessons learned were not consistently completed for all events.

The evaluation team would like to acknowledge that interviews also highlighted that there are challenges regarding administrative requirements and that there are no consistent tools available to track success. There is an opportunity for the program area to utilize a “final activity report” tool similar to the report currently required for funding recipients through the Commemorative Partnership Program. It is important that the activity report highlight reach of the event (in-person and online participation), resources utilized for the event, successes of the event, and any opportunities for improvement that should be considered in the future. A recommendation in this regard is identified in Section 5.1 Commemorative Ceremonies and Events.

As previously stated in Section 4.2 Achievement of Expected Outcomes, Immediate Outcome #1: Commemorative Ceremonies and Events, the evaluation team found that process improvements are required for promoting domestic events.

Opportunity for Improvement:

Key informant interviews highlighted that additional clarity is required with respect to roles and responsibilities for individuals (VAC Commemoration/Communication staff) organizing/supporting these events.

4.3.2 Commemorative Partnership Program

Through key informant interviews and workload observation, the evaluation team found that processes are in place to ensure the Program is administered as intended. The team determined that decisions on funding proposals less than $10,000 (grants) for the Community Engagement element of this Program, are approved at the Assistant Deputy Minister level upon recommendation by the program officer, manager, director, and director general. Decisions for projects receiving more than $10,000 are recommended by a decision-making committee prior to final sign-off by the Assistant Deputy Minister. All funding proposals through the Community War Memorial element of this Program are recommended by a decision-making committee prior to final approval at the Assistant Deputy Minister level.

Analysis of the approved projects during 2015-16 revealed that 65% of the projects were funded through a grant. Although these projects were determined to be low risk and were of low dollar value (less than $10,000), approval was still required at the Assistant Deputy Minister level. The evaluation team was informed through interviews with Canadian Heritage’s Celebration and Commemoration Program, that signing authority is delegated to the Director General level for projects funded through the grant portion of their program.

Opportunity for Improvement:

Though current service standards are being met (Table 22 – Service Standards Commemorative Partnership Program), removing decision layers would further reduce wait times for funding recipients and improve program efficiency and cost effectiveness.

Duplication/Overlap:

Processes are in place to ensure that funding provided through the Commemorative Partnership Program is not duplicated between VAC and other government departments (e.g., Canadian Heritage).

4.3.3 Learning and Public Information

Through key informant interviews and observation, the evaluation team confirmed that:

• during the development of learning resources, VAC’s Education Officers ensure Canada’s cultural diversity is appropriately reflected and that learning resources take into consideration identity factors such as gender, race, ethnicity, religion, age, and geography (Gender Based Analysis +);
• processes exist between the Department of National Defence and VAC to ensure the historical accuracy of VAC learning resources and to ensure appropriate historical training is provided to VAC student guides at the Canadian National Vimy Memorial and the Beaumont-Hamel Newfoundland Memorial; and
• effective processes are in place for posting content to social media that define the roles of key players such as VAC’s Communication Division.

Through interviews, the evaluation team determined that social media processes that determine workflows between Commemoration and Communications staff are working effectively. The evaluation team also reviewed the social media plan prepared by Commemoration Education Officers and Communication’s social media team and finds that comprehensive efforts are being made to ensure social media activities are effectively recognizing Veterans, including anniversaries and other milestones.

As previously noted in Section 4.2 Effectiveness – Achievement of Expected Outcomes, Intermediate Outcome #3: Learning and Public Information, the impact of the services provided by student guides on visitors could not be fully assessed due to a lack of visitor experience feedback. Interviews with employees currently involved with VAC’s student guide services as well as interviews with former student guides identified that new processes are required to ensure visitor experience information is captured/monitored. A document review of the Performance Measurement Strategy associated with the component confirmed that visitor experience information is not being collected in a meaningful way (currently paper-based guest book). Visitor feedback is essential to answer questions such as:

• Are VAC guides achieving the correct balance between interpretation and guiding?^{Footnote 27}
• Do visitors feel the tours are well organized and readily available?
• What elements are meeting visitor expectations and what elements need improvement?

The evaluation team interviewed employees from other federal organizations with similar Programs, including Parks Canada, Canadian Heritage and the Parliament of Canada (Parliamentary Guide Program). The evaluation confirms that these organizations are using or adopting updated processes/tools to collect visitor experience feedback (e.g. Parks Canada uses visitor surveys, Canadian Heritage outfits guides in the National Capital Region with tablets to record visitor experiences, etc.).

The evaluation team concludes that VAC should model the assessment of offerings at the Canadian National Vimy Memorial and the Beaumont-Hamel Newfoundland Memorial in France on best practices in use by other federal organizations.

Recommendation #3

It is recommended that the Director General, Commemoration, develop and implement measures to capture visitor experience feedback associated with Veterans Affairs Canada’s memorial sites in Europe.

Management Response:

Management agrees with this recommendation.

Management Action Plan:

Corrective action to be taken	OPI (Office of Primary Interest)	Target Date
The Director General, Commemoration, will develop and implement an approach to capture feedback from visitors to Veterans Affairs Canada’s memorial sites in Europe.	Director General, Commemoration	December 2018

Opportunities for Improvement:

Key informant interviews highlighted that the recruitment/hiring/training process associated with student guides is resource intensive and could be impacting the efficiency of the initiative. Currently, groups of students are hired for four-month terms (there are three four-month terms annually). Consideration is being given to proceeding with two six-month terms per year to help reduce resources needed for recruitment, hiring, and training. Longer work terms would also provide the benefit of allowing guides to gain additional experience and to further grow into their roles.

Duplication/Overlap:

Interviews with partners from external government departments, including Canadian Heritage and the Department of National Defence, highlighted positive/productive working relationships with VAC staff contributing to the Canada Remembers Program components evaluated. Roles between departments are defined and complementary; there is no evidence of duplication of effort.

4.3.4 Memorials

VAC’s 14 memorials in Europe, which include monuments and surrounding terrain, are maintained through a contract and memorandum of understanding with the Commonwealth War Graves Commission. Through interviews, and a review of the memorandum of understanding, the evaluation team confirms that appropriate processes exist with the Commonwealth War Graves Commission in relation to the maintenance work on these memorials. The evaluation team obtained and reviewed maintenance plans prepared for each memorial. These plans were updated and approved in 2014. The evaluation team found that VAC/Commonwealth War Graves Commission is utilizing these plans when assessing annual maintenance requirements.

Through interviews and observation, the evaluation team has determined that appropriate processes are in place to ensure commemoration banners are erected along the Confederation Boulevard ceremonial route in Ottawa. Roles and responsibilities are clear between VAC and Program partners (Canadian Heritage administers the banner initiative) and processes are in place for banner storage after use.

The evaluation found that processes for determining inclusion in the Books of Remembrance require improvement. For actions arising from a Matter-of-Right (death attributable) decision, the department does not have standards against which to measure.^{Footnote 28} Current processes are contributing to delays and, as revealed in a file review, in certain circumstances, Books of Remembrance entries are being missed. The evaluation team noted the missing entries to the work area responsible for correction in future updates. This finding is directly linked to recommended improvements identified in the recent VAC Evaluation of Commemorative Benefits and Services – Recommendation #3.^{Footnote 29} The evaluation team received agreement from program staff that processes associated with the Books of Remembrance are being included as part of the actions required to address the findings in the Evaluation of Commemorative Benefits and Services.