4.1 Effectiveness and Performance Measurement
In July 2016, the Policy on Results was implemented, replacing the previous Policy on Evaluation. Under this new policy, additional emphasis was placed on monitoring and reporting of performance measurement for federal government departments, including requirements for reporting and consultation with Treasury Board Secretariat on program performance outcomes, indicators, and outputs.
Performance measurement is a key function that aides in effective program management. Performance measurement is generally described as the regular measurement of indicators and outputs established to track progress towards achieving the intended outcomes of a program. This information is used to assess the effectiveness and efficiency of programs and to inform day-to-day decision making in program management.
<Currently, a Performance Information ProfileFootnote 7, is in place for the Program which outlines outcomes, outputs, accompanying indicators and targets for the Program. Appendix B shows a summary of the outcomes listed in the Performance Information Profile. The evaluation found that there is an opportunity for greater consistency in the collection and use of performance measurement information to help guide program management decisions made by the Rehabilitation, Financial Well-Being and Business Intelligence Unit.
4.2 Outcomes
The Program is meeting its ultimate outcomes:
- Veterans are physically and mentally well.
- Veterans and their families are financially secure
- Veterans are satisfied with the services they receive
The 2017 VAC National SurveyFootnote 8 identified that 92% of War Service Veterans were satisfied or very satisfied with their overall well-being. In addition, 89% indicated they were satisfied or very satisfied with their financial situation and 94% were satisfied or very satisfied with the services they receive from VAC. This is important because War Service Veterans are the target group for the Program and these numbers are well above identified targets in the Performance Information Profile.
Key informant interviews and the file review provided insight that access to other VAC programs through eligibility for the Program may actually be more important to the Veterans than the actual monthly payment itself. This includes access to Health Care Benefits (Treatment Benefits), the Veterans Independence Program, the Long-Term Care Program and the VAC Assistance Fund. Eligibility for these programs enhances the ability of the recipient to meet their basic needs.
By providing eligible recipients with access to additional income to support their basic needs and serving as a gateway to other programs and services, the Program contributes to the achievement of these ultimate outcomes. However, it is important to recognize that other health factors, regional variances in cost of living, family composition, and spending habits may also influence the ultimate outcome.
The Evaluation team was unable to fully assess the Program’s intermediate outcome:
- Eligible Veterans and other clients’ basic needs are met
2017 VAC National Survey results discussed above relates to War Service Veterans satisfaction with their current financial situation.
The Performance Information Profile identifies an additional performance measurement outcome for this indicator: a target of 5% or less WVA recipients below the Low Income MeasureFootnote 9(LIM). Through consultation with the Program area, the evaluation team was informed that the Department will use data collected through the 2019 VAC National Survey and conduct secondary analysis to calculate the LIM for WVA recipients. However, the measurement of this performance indicator has limitations in that the recipients are a small sub-population of the overall War Service Veteran population. Therefore, 2019 VAC National Survey results may not be statistically significant. Moreover, the survey sample includes Veterans and survivors; data will not be collected from orphan recipients. The Program area will work with VAC Statistics and the Business Intelligence Unit to develop a more comprehensive method of measurement for this outcome indicator.
In addition, the Program area intends to review the current performance information profile for the Program and seek the necessary approvals for any identified updates.
The Program is meeting its immediate outcome:
- Veterans and their families have timely access to War Veterans Allowance funding to meet basic needs
The performance target for this service standard is that 80% of decisions will be rendered within four weeks. Program data confirms that, on average, 83% of decisions were made within four weeks over the entire review period as outlined in Table 4.
| Year | Total Decisions | Decisions within Standard | Percentage |
|---|---|---|---|
| 2014-15 | 448 | 402 | 90% |
| 2015-16 | 460 | 390 | 85% |
| 2016-17 | 381 | 275 | 72% |
| 2017-18 | 365 | 292 | 80% |
| 2018-19* | 163 | 157 | 96% |
| Overall | 1,817 | 1,516 | 83% |
* first six months of fiscal year 2018-19
Although, the percentage of decisions rendered dipped below the standard in 2016-17, key informant interviews identified that the training of new analysts contributed to the recent improvement. Achieving 96% of decisions rendered within the standard in the first six months of fiscal 2018-19 demonstrates the positive impact of the training of the new analysts.
4.3 Economy
Between fiscal year 2014-15 and fiscal year 2017-18, Program expenditures decreased by 28% and the number of Program recipients decreased by 32%. As of March 2018, there were 1,895 recipients and annual Program expenditures of $6.1M compared to the expenditure forecast of $6.4M. The average annual payment per recipient was $3,219.
The decreasing trend is forecasted to continue for both the Program recipient base and the Program expenditures into the foreseeable future. By 2022-23, it is forecasted that there will be 1,030 recipients receiving $4.2M.
Veterans, whose receipt of OAS/GIS places them above the income threshold for a payment, also known as Near Recipients, may also be eligible for VAC Health Care Benefits and Services. As of March 31, 2018, 4,196 Veterans who were considered a near recipient and therefore having access to important benefits and services to meet their health needs.
Resource Utilization
In 2017-18, VAC salary costs allocated to the Program activities were less than 0.2% of all VAC program administration costs. This figure is comparable to the level of Program expenditures, which is about 0.1% of total VAC Program expenditures. Further analysis of administrative costs was not undertaken due to the low risk and low materiality of the Program.
4.4 Efficiency
The 2014 evaluation of the Program found it to be stable and efficient. Key informant interviews, as well as a review of documents, indicate that few changes have occurred since the last evaluation. The Program continues to be delivered by a small, centralized team.
In addition, the improvement on decisions rendered within the service standard point to the positive effect of training of new analysts.
VAC Policy also has a longer term goal of reducing the current number of policies associated with the Program (13) down to one overarching policy to simplify direction associated with the Program.
4.5 Unfavourable Decisions
Program data, key informant interviews and a review of documents identified that there is a high rate of unfavourable decisions among applications received for the Program. A file review of a random sample of 182 applicants with unfavourable decisions between April 1, 2017, and September 30, 2018, provided additional information around the reasons behind the unfavourable decisions. Table 5 shows a detailed breakdown.
| Reasons | Number unfavourable within sample | Percentage unfavourable within sample |
|---|---|---|
| Unfavourable – ServiceFootnote 10 | 54 | 30% |
| Unfavourable – Excess IncomeFootnote 11 | 111 | 61% |
| Unfavourable – OtherFootnote 12 | 17 | 9% |
| Total | 182 | 100% |
Through the file review, the Evaluation team found that a number of applications for the Program originated as a result of individuals seeking access to other VAC programs such as the Health Care Benefits (Treatment Benefits) Program, the Veterans Independence Program and the Long-Term Care Program.
The file review found that 30% of applications in the sample received unfavourable decisions as it was found that the Veteran associated with the application had an ineligible service period. The review also found that 61% of applications in the sample received unfavourable decisions as a result of the applicant having income above the threshold. Table 6 shows a detailed breakdown of the unfavourable decisions in the sample and their proximity to the income threshold.
| War Veterans Allowance | Total | < $200 per month | $201 - $500 per month | > $500 per month | Average per month |
|---|---|---|---|---|---|
| Veteran | 72 | 7 (10%)* | 9 (13%) | 56 (78%) | $1,117 |
| Survivor/Orphan | 39 | 13 (33%) | 17 (44%) | 9 (23%) | $498 |
* Percentages may not add due to rounding.
On average, Veterans, who applied for the Program and who received unfavourable decisions for income in excess of the threshold, were $1,117 away from qualifying as a near recipient. By comparison, survivors and orphans were, on average, $498 away from qualifying for a monthly WVA payment. Further analysis reveals that 77% of the survivor and orphan group were within $500 per month of qualifying while 33% were within $200 per month of qualifying.
The findings of this file review can be used by the Policy Directorate in support of future policy renewal.
In addition, during the review period, only thirteen files were received by the Veterans Review and Appeal Board on appeal and of these, only five were overturned.