4.1 Performance Tools and Measurement
Performance Measurement Strategies are in place and performance results are being collected; however, administrative expenditure breakdowns are incomplete for both the Funeral and Burial Program and the Cemetery and Grave Maintenance Program.
Performance measurement monitors the progress of programs towards their expected resultsFootnote 29. A Performance Measurement Strategy (PMS) is used to regularly measure key indicators and results. This information can be used to assess against expectations and to assist in measuring the effectiveness and success of a program. A program Performance Measurement Plan (PMP) and a program logic model are tools that support the PMS.
The evaluation team reviewed the PMS, PMP and associated performance reporting for both the Funeral and Burial Program and the Cemetery and Grave Maintenance Program, confirming:
- Performance Measurement Strategies are approved/current;
- Performance Measurement Plans identifying the specific outputs, outcomes and associated performance indicators are in place;
- Performance results are available, both through operational reporting and strong public opinion research;
- Data/expenditure information pertaining to administrative costs is incomplete, specifically in relation to:
- Resources required to support decision-making associated with Matter-of -Right decisions for the Funeral and Burial Program; and
- VAC and CWGC resources required to contract cemetery maintenance.
The Honours and Awards Program is part of the PMS for the broader Ceremonies and Events Program, therefore the strategy and outcomes associated with this program will be assessed in a separate evaluation scheduled for 2017-18.
Outside of the scope of this evaluation, a new Government of Canada Policy on Results came into effect in July 2016,Footnote 30 detailing the requirements for departments to clearly identify core responsibilities and associated departmental programs with these respective responsibilities. In support of the new policy, departments will develop Program Inventories and Performance Information Profiles to replace Performance Measurement Strategies. Future evaluations will assess the appropriateness of the Department’s updated performance measures/outcomes.
4.2 Achievement of Expected Outcomes
Program outcomes are the changes or differences that result from program activities and outputs. Outcomes are described as immediate, intermediate, or ultimate based on the contribution/influence the program has on each outcome. As outlined by the Treasury Board Secretariat (TBS) guideline, Supporting Effective Evaluations: A Guide to Developing Performance Measurement StrategiesFootnote 31:
- immediate outcome(s) should equate to a “change in awareness, knowledge or skill;”
- intermediate outcome(s) should equate to a “change in the target population’s behavior;” and
- ultimate outcome(s) should equate to a “change of state in a target population.”
When the above outcomes are met, the Program contributes to the Department’s Strategic Outcome #2: Canadians remember and demonstrate their recognition of all who served in Canada’s efforts during war, military conflict, and peace.
This section of the evaluation assesses the extent to which Programs are being achieved.
Immediate Outcome #1: Funeral and Burial Program
The service provider has the funding and tools it needs to financially assist the estates of eligible Veterans.
The Service Provider (Last Post Fund) has the funding and tools needed to financially assist the estates of eligible Veterans; however, access to additional demographic information on potential applicants would enhance the Program’s reach.
VAC and the Last Post Fund have a Memorandum of Understanding (MOU) to ensure the service provider has the funding available to financially assist the estates of eligible Veterans. It states that the Minister of Veterans Affairs shall “approve and advance funds to the Corporation on a quarterly basis for reasonable administrative and operating expenses incurred by the Corporation in accordance with an approved annual budget for the provision and administration of last sickness, funeral, burial, cremation and grave marking benefits for Veterans and other eligible persons, in accordance with Part I of the Regulations (Veterans Burial Regulations).” Annual forecasts are prepared to ensure that the funding allocated to the Last Post Fund is sufficient to meet the requirements of the Program. The funds provided to the Last Post Fund for the past four fiscal years are shown in Table 7.
| Expenditure Type | 2012-13 | 2013-14 | 2014-15 | 2015-16 |
|---|---|---|---|---|
| Funds advanced to the Last Post Fund | $7,949,251 | $8,461,534 | $8,547,329 | $9,050,000 |
Source: VAC Finance Division
The evaluation team observed and confirmed through interviews that the Last Post Fund has the tools available to assist the estates of eligible Veterans. It has appropriate templates and systems (telephone and database) to support the determination of eligibility and the processing of payments.
The evaluation team identified that limited access to contact information for the survivors or power-of-attorneys/next-of-kin of deceased Veterans is impeding the Last Post Fund’s ability to make contact with potential recipients. If the Veteran was in receipt of VAC benefits or services at the time of death, the Last Post Fund is notified with the intention that information packages be distributed. However, in certain circumstances, lack of access to contact information results in the Last Post Fund not connecting with the Veteran’s estate and therefore no application is received. Each year, the estates of approximately 3,000 Veterans in receipt of VAC benefits and services at the time of death have no confirmed contact with the Last Post Fund, a portion of which can be attributed to lack of current contact information. It is the evaluation team’s opinion that providing the Last Post Fund with access to all current contact information within VAC’s database would increase the Program’s ability to reach additional estates of potential program recipients.
Immediate Outcome #2: Funeral and Burial Program
Estates of eligible Veterans have access to financial assistance for funeral and burial services.
Procedure and process improvements are required to ensure that the estates of deceased Veterans who were in receipt of VAC services associated with low income, homelessness, or disability benefit conditions commonly attributed to service-related deaths receive targeted follow-up for potential eligibility.
The Funeral and Burial Program provides access to financial assistance for funeral and burial services to approximately 1,200 Veterans’ estates annually, representing close to 6% of Veteran deaths that occur each year.
Program eligibility is determined through either a Means Test or a Matter-of-Right decision. A Means Test is performed by the Last Post Fund through a financial assessment of a Veteran’s estate, taking into account marital status, number of dependents and net assets.Footnote 32 When a Veteran’s death is related to military service due to a condition for which VAC provided a disability benefit (must be determined by a VAC medical authority), assistance may be granted as a Matter-of-Right (no Means Test). Approximately 11% of Veterans’ estates are being assessed for eligibility. Table 8 breaks down eligibility decisions by type and reach.
| Funeral and Burial Program | 2012-13 | 2013-14 | 2014-15 | 2015-16 |
|---|---|---|---|---|
| Favorable eligibility through financial Means Test | 1,050 | 1,050 | 1,078 | 986 |
| Favorable eligibility through Matter-of-Right | 177 | 163 | 108 | 188 |
| Total recipients | 1,227 | 1,213 | 1,186 | 1,174 |
| Unfavorable decisions | 1,569 | 1,077 | 1,165 | 1,065 |
| Total decisions | 2,796 | 2,290 | 2,351 | 2,239 |
| Estimated Veteran deaths | 21,300 | 21,700 | 20,300 | 20,100 |
| Overall % of Veteran’s estates that received an eligibility decision | 13.1% | 10.6% | 11.6% | 11.1% |
| Overall % of Veteran’s estates that received financial assistance | 5.8% | 5.6% | 5.8% | 5.8% |
Source: VAC Finance Division/Last Post Fund
Although the overall intake to the Program has remained relatively consistent over the past four years, the evaluation found that there is a risk that the Program is not fully reaching Veterans.
To assess the Program’s reach from a financial need perspective, the evaluation team used data analysis and file reviews to determine the extent to which funeral and burial eligibility was reaching the estates of Veterans who had an increased likelihood of requiring assistance due to low income and/or from being homeless. Over the four-year period, 119 Veterans died while receiving VAC services and benefits that supported these areas of need. The analysis showed:
- 45% of the estates associated with these Veterans received eligibility decisions for the Funeral and Burial Program; of those 83% were deemed to have financial insufficiency and received access to funds for funeral and burial services.
- 55% of the estates did not receive eligibility decisions, either because they were not contacted for potential eligibility or there was no response to the information sent out by the Last Post Fund (address/contact information available to the LPF may have been incomplete, therefore it cannot be confirmed that packages were ever received).
Through interviews, the evaluation team confirmed that no mechanisms were in place to monitor and/or provide additional assistance/follow-up to this group. Therefore, the evaluation concluded that VAC should implement proactive procedures and processes to better support these targeted groups.
With respect to the Program reach associated with Veterans likely to be eligible through a Matter-of-Right decision, the evaluation team was informed there is potential that cases are being missed and that certain medical conditions represent a large portion of favorable Matter-of-Right decisions.
A demographic analysis identified that Veterans who were single/widowed were less likely to receive eligibility through a Matter-of-Right decision. Analysis showed that 1 in 34 Disability Benefit recipients who were married/common law at their time of death received Matter-of-Right eligibility while only 1 in 124 Disability Benefit recipients who were single/widowed at the time of death received Matter-of-Right eligibility.
A Gender Based AnalysisFootnote 33 of this finding determined that the estates of female Veterans are negatively impacted. During the evaluation period, female Disability Benefit recipients who passed away were more likely to be single/widowed than males, thus were less likely to receive a Matter-of-Right eligibility decision.
An analysis of medical conditions associated with Veterans who received favorable Matter-of-Right decisions during the evaluation period indicated the most common are: Amyotrophic lateral sclerosis (ALS), Heart Disease, Coronary Artery Disease, Carcinoma of Lung/Rectum, Chronic Obstructive Pulmonary Disease (COPD), Pulmonary Fibrosis, Mesothelioma and respiratory cancers. The evaluation team found that more than 50% of Veterans in receipt of VAC benefits for one or more of these conditions did not receive a Matter-of-Right eligibility decision. A subsequent file review on these cases found a medical professional’s determination on eligibility should be pursued in cases where the death information is recorded on VAC’s Client Service Delivery Network (CSDN).
Recommendation 1:
It is recommended that measures be implemented to increase access to funeral and burial financial assistance for:
- Veterans receiving VAC benefits and/or services for low income and/or homelessness (Means Tested); and
- Single/Widowed Veterans and Veterans in receipt of disability benefits for certain conditions (Matter-of-Right).
Management Response:
Management agrees with this recommendation.
Management Action Plan:
| Corrective Action to be Taken | Office of Primary Interest (OPI) | Target Date |
|---|---|---|
| The Director General, Commemoration will implement measures (e.g. generating detailed exception reports and establishing follow-up actions) to help increase access to funeral and burial Means Tested financial assistance for Veterans that were in receipt of benefits for low income and/or have been identified as homeless/having increased risk for homelessness. | Director General, Commemoration | June 2017 |
| The Director General, Commemoration will develop and release communication messaging to Field Operations which emphasizes/confirms the requirement to render Matter-of-Right decisions for single Veterans. | Director General, Commemoration | June 2017 |
| The Department will develop a referral process for decisions aimed at ensuring Matter-of-Right consideration for the survivors/estates of Veterans that were in receipt of disability benefits for certain conditions. | Assistant Deputy Minister (ADM), Strategic Policy and Commemoration in conjunction with ADM, Service Delivery | December 2017 |
Immediate Outcome #3: Cemetery and Grave Maintenance:
Properly maintained and presented VAC memorials, cemeteries and grave markers on behalf of all Canadians.
Additional funding is required to ensure all Veteran grave markers are properly maintained.
Due to different contract arrangements, the graves of the war dead of the First and Second World Wars and the graves of deceased Canadian Veterans were assessed separately.
Canadian War Dead of the First and Second World Wars
As part of a 1917 Royal Charter, Canada, along with other member countries entered into an agreement with the Imperial War Graves Commission (now known as Commonwealth War Graves Commission). This Charter provided the Commonwealth War Graves Commission (CWGC) with the mandate to mark and maintain the graves of Commonwealth servicemen and servicewomen who died during the First World War. The Charter was later updated to include war dead from the Second World War. Approximately 110,000 Canadians lost their lives during these wars and are buried in more than 70 countries throughout the world. The vast majority of Canadian war dead graves are found in France, Belgium and Italy. In keeping with the commitment of the 1917 Royal Charter, Canada through VAC, provides an annual grant to the CWGC for the maintenance of these graves. CWGC has become known as a worldwide authority in the area of military cemetery and grave marker maintenanceFootnote 34 and interviews and document reviews showed that CWGC maintains war dead graves to a high standard.
VAC provides a grant to CWGC, with the amount depending on the work required and applicable exchange rates. Table 9 shows the annual grants by fiscal year.
| Expenditure Type | 2012-13 | 2013-14 | 2014-15 | 2015-16 |
|---|---|---|---|---|
| Funds advanced to the CWGC | $9,391,002 | $9,714,969 | $10,913,493 | $12,040,538 |
Source: VAC Finance Division
The evaluation team observed grave markers at 18 cemeteries located in Prince Edward Island, Nova Scotia and Quebec. The majority of the 37 war dead grave markers which were observed were acceptable. In a few instances, improvements relating to cleaning and minor misalignment of the marker were required.
Deceased Canadian Veterans
VAC is also responsible for the grave maintenance of more than 200,000 Veterans who were buried by the Government of Canada. They are located in more than 6,000 cemeteries throughout the country.
Until 2003, VAC received approximately $5M per year for the upkeep of Veterans’ graves throughout the country. At that time, since VAC was unable to accurately account for graves requiring maintenance and their locations, this amount was reduced to $1M. The annual budget amount increased to $1.2M in 2009 and has since remained consistent. Of this amount, $700,000-$800,000 is allocated for contracts to maintain Veteran graves: the remaining funds are used to support cyclical inspections of graves, address CWGC administrative costs, and manage and maintain VAC’s two departmental cemeteries.
In 2004, VAC developed the Canada Remembers Grave Tracking (CRGT) system, a database to identify and track the condition of all the Veteran graves in Canada for which the Department is responsible. It then contracted the CWGC to locate, identify and provide condition reports on the identified markers. The database currently lists more than 200,000 Veteran grave markers with a corresponding report on their condition. More than 45,000 of these Veteran grave markers require maintenance, with a total of close to 60,000 repairs required (some grave markers require more than one type of repair).
Interviews with VAC and CWGC identified that there are not enough funds currently allocated to Veteran grave maintenance to address the known required repairs; i.e., the current funding level allows VAC to perform only a portion of the required maintenance. To manage the work, VAC produces an annual cemetery and grave maintenance plan which identifies and prioritizes required maintenance work in private cemeteries throughout Canada as well as in its two departmental cemeteries. VAC contracts with various suppliers, including the Canadian Agency of the CWGC to administer the work.
Additionally, as part of its contract with VAC, CWGC is responsible for providing an inspection report on all Veterans’ graves on a 12- year cycle, allowing for the inspection of approximately 17,000 Veteran graves per year. Table 10 shows the volumes of maintenance on an annual basis.
| Repairs per year | 2012-13 | 2013-14 | 2014-15 | 2015-16 | 4-year Average |
|---|---|---|---|---|---|
| # of repairs | 2,707 | 2,630 | 3,875 | 4,176 | 3,347 |
| # of Veteran grave markers repaired | 2,622 | 2,500 | 3,730 | 4,046 | 3,225 |
Source: VAC Commemoration Division
Repair costs differ significantly, from $75 for grave marker cleaning to $1,100 for a new upright marker. While the total volume of repairs increased over the past two years, the number of lower cost repairs (e.g., cleaning) increased and the number of higher-cost repairs (e.g., grave marker replacements) decreased.
Through graveyard visits, the evaluation team assessed various elements to determine issues/concerns that may be visible to the general public. Of the 103 Veteran grave markers appraised, 96% (99/103) were identified as needing improvement; however, 77% (76/99) of these required only minor repairs (cleaning, weeding, and potential alignment) and were still assessed as achieving the Program’s outcome of being properly maintained. Most of the 23 markers assessed as not achieving this outcome required major repairs such as replacement markers due to cracks/breaks, foundation issues and inscription legibility.
The overall number of outstanding repairs has decreased by close to 9% since 2013; however, higher-cost maintenance items such as legibility issues, grave marker replacements and foundation repairs have been increasing on an annual basis and should be considered as priority items. As observed, these higher- cost items have a significant impact on the achievement of the Program outcome. Table 11 shows outstanding repairs by year.
| Repair Type | 2013 | 2014 | 2016 | % Variance 2013-2016 |
|---|---|---|---|---|
| Replacement | 236 | 259 | 286 | 21.2% |
| Illegible | 72 | 108 | 159 | 120.8% |
| Alignment | 20,649 | 20,415 | 18,185 | -12% |
| Replacement Foundation | 1,198 | 1,249 | 1,463 | 22.1% |
| Grave Surfacing | 668 | 670 | 750 | 12.3% |
| Cleaning | 39,181 | 38,635 | 35,629 | -9.1% |
| Other | 574 | 609 | 591 | 3.0% |
| Total Repairs Required | 62,578 | 61,945 | 57,063 | -8.8% |
Source: VAC Commemoration Division
Based on the allocated funding, an average of 3,347 repairs can be completed each year (see Table 10). At this rate, it will take more than 17 years to complete the current outstanding grave marker repairs (57,063 outstanding repairs/3,347 repairs per year).
The average cost per unique maintenance item repair over the past four years was $225. This average encompasses lower costs repairs, such as $75 for grave marker cleaning, and high cost repairs, such as, $1,100 for a new upright marker. Based on the overall average of $225, the total cost to address the 57,063 outstanding repairs is approximately $12.8 million. Table 12 shows the breakdown and costs of completed repairs by year.
| Breakdowns | 2012-13 | 2013-14 | 2014-15 | 2015-16 | 4-year Average |
|---|---|---|---|---|---|
| # of repairs completed | 2,707 | 2,630 | 3,875 | 4,176 | 13,388 |
| Expenditures specific to grave/cemetery maintenance | $765,406 | $735,418 | $739,759 | $771,277 | $3,011,860 |
| Average contract expenditure per grave maintained | $292 | $294 | $198 | $191 | $225 |
Source: VAC Commemoration Division
Veterans Affairs Canada obtained feedback from Canadians through the 2014 Attitudes Towards Remembrance surveyFootnote 36. Results showed that 62% of Canadians are satisfied with how VAC recognizes and honours Veterans through the maintenance of memorials, cemeteries and grave markers.
The evaluation concludes that additional efforts are required to ensure Veteran graves are being properly maintained. Increased funding as well as a targeted work plan are required as to ensure priority is placed on:
- Reducing the backlog of outstanding repairs; and
- Fixing higher-cost items such as legibility issues, grave marker replacements and foundation replacements, as these items have been increasing and are contributing to the inability to adequately achieve the program’s immediate outcome.
Recommendation 2:
It is recommended that the Director General of Commemoration put measures in place to reduce the backlog of outstanding Veteran grave maintenance, with an increased focus on maintenance items such as inscription legibility, grave marker replacements and foundation repairs.
Management Response:
Management agrees with this recommendation.
Management Action Plan:
| Corrective Action to be Taken | Office of Primary Interest (OPI) | Target Date |
|---|---|---|
| To reduce the volume of outstanding repairs, current practices will be reviewed for possible efficiencies. A revised approach, such as prioritizing the repair of items that compromise the structural integrity of Veteran graves, will be integrated in the annual maintenance plan. | Director General, Commemoration | June 2017 |
| In support of the 2018-2019 maintenance plan, options will be explored for increased resource allocations to help address the backlog of maintenance items. | April 2018 | |
| Implementation of the 2018-2019 maintenance plan will begin. | April 2018 |
Intermediate Outcome #1: Funeral and Burial
Financial support is provided to eligible Veterans for funeral and burial expenses.
Financial support is provided to those currently eligible for the Program; however, a needs assessment associated with single Veterans should be completed to ensure current eligibility requirements are not impacting the achievement of the Program objectives.
Approximately 1,200 survivors and estates are receiving financial support through the Funeral and Burial Program each year (see Table 1).
The published service standard for processing Means Tested applications is: “a decision will be made within 30 calendar days of receiving all information in support of your application.” The performance results for this service standard have exceeded the 80% target, ranging from 87%-90% during the evaluation period.
As there are no service standards associated with payments for those accessing the program through a Matter-of-Right decision, the evaluation team recommends that process improvements and standards be established. Additional details on these findings and the associated recommendation can be found in Section 5.
The team found that the low number of issues and complaints received from estates with respect to the process and/or amount of funds received are primarily related to eligibility requirements established through the Veterans Burial Regulations.
With respect to eligibility when there is a survivor, Budget 2016 enhancements increased survivor estate exemption amounts from $12,000 to more than $35,000. This change will help ensure an increased number of survivors are able to receive financial assistance. This increase does not pertain to the estates of single/widowed Veterans.
The primary area of concern, as identified through staff interviews, is that estates of single/widowed Veterans are challenged to meet eligibility through the Means Test, which requires that all net assets of the estate as a whole - i.e., all liquid assets (e.g., savings, life insurance, CPP Death Benefit and monthly income) and fixed assets (e.g., Veteran’s residence, car and household effects) - be taken into consideration. In order for favourable eligibility to be granted for these estates, the deceased Veteran would have to have had debts that exceeded the total asset amounts.
From April 2012 to March 2016, 41% of single Veterans’ estates that applied for the Funeral and Burial Program through Means Testing were granted eligibility and received an average payment of $5,400. During the same period, 53% of the applications from the estates of Veterans that had a survivor/dependent were granted eligibility and received an average payment of $6,100. The favourable rate and average payment for applications from survivors/dependents is expected to increase in the future as a result of Budget 2016 enhancements.
Upon further review, the evaluation team is of the opinion that the estates of single Veterans whose primary means were low-value fixed assets were at risk of not having a dignified funeral and burial. In these instances, single Veterans’ estates were deemed to have available assets to either fully or partially contribute to the cost of funeral and burial due to fixed assets such as low-market value cars and low-valued property/real estate. During the evaluation period, 79 estates were either declined or received a reduced reimbursement due to a fixed asset valued at $5,000 or less being considered as available funds.
Although there is no evidence to suggest these Veterans did not receive a dignified funeral and burial (no follow-up with specific applicants), it is possible that the expectation that a Veterans’ estate would have to liquidate low value items to obtain funds for a funeral and burial could have negative impacts on the decisions made by the estate. This situation is highlighted by the evaluation team as a potential unintended outcome of the Program. It would be valuable for the Program area to undertake a needs review associated with deceased single Veterans in order to ensure current eligibility requirements are not impacting their ability to receive a dignified funeral and burial.
A Gender Based Analysis of program recipients assessed through a Means Test found that the estates of female Veterans deemed eligible represented 7% of recipients, higher than the proportion (4%) of female Veteran deaths.Footnote 37 This discrepancy could be attributed to females having a lower average income than males, especially in the 65 and over age group.Footnote 38
Annex A provides details on the Means Testing criteria differences between Veterans who had a survivor/dependents at time of death versus Veterans who were single/widowed with no dependents at time of death.
Intermediate Outcome #2: Cemetery and Grave Maintenance
Canadians support properly maintained VAC memorials, cemeteries and grave markers.
Canadians agree it is important that VAC honour deceased Canadian Veterans and war dead by maintaining memorials, cemeteries and grave markers.
VAC measured this outcome through its 2014 “Attitudes Towards Remembrance Survey." The survey asked participants if it is important that “VAC honour deceased Canadian Veterans and war dead by maintaining memorials, cemeteries and grave markers.”Footnote 39 The results showed that the vast majority of Canadians (89%) agreed it is important for VAC to do so.
Canadians’ support for Veterans and war dead is also evident though a November 2015 poll released by the Historica-Dominion Institute,Footnote 40 which measured Canadians’ attitudes on remembrance and found that:
- Most (91%) Canadians ‘agree’ that Canada should do more to honour its Veterans; and
- Nine in ten (90%) ‘agree’ that Canadians should do more to honour those who fought and those who have died in war.
Media coverage and news releases relating to the subject of Veterans’ graves over the past four years provide evidence of Canadian support for properly maintained cemeteries and grave markers. Numerous articles from family members and community members identifying required repairs reference how it should be the Government’s responsibility to repair Veterans’ graves.
Ultimate Outcome Funeral and Burial Program and Cemetery and Grave Maintenance Program
Canadian Veterans and those who died in service are honoured and the memory of their achievements and sacrifices is preserved.
The ultimate outcome is being achieved.
The evaluation concludes that the objectives of both the Funeral and Burial Program and Cemetery and Grave Maintenance Program contribute to the ultimate outcome of preserving the memory, achievements and sacrifices of Canadian Veterans and those who died in service. To achieve this outcome, these Programs complement other Canada Remembers programs, including Public Education and Awareness, Ceremonies and Events, Partnerships and Collaborations and Memorial Maintenance.
Public opinion research conducted by VAC shows that this ultimate outcome is being achieved. In VAC’s 2010 National Client SurveyFootnote 41, the vast majority (88%) of VAC program recipients reported that they are satisfied with how the Department is delivering its remembrance programs.
In addition, in both 2012 and 2014, VAC conducted Veterans’ Week surveysFootnote 42 to gain Canadians’ opinions in relation to commemoration, finding that 75% and 73%, respectively, felt VAC’s remembrance programming effectively honours Veterans and those who died in service, and preserves the memory of their achievements and sacrifices. The target was 70%. Further detail on the survey can be found in Annex B.