To assess performance, the evaluation team examined the Performance Measurement Strategy (PMS), the achievement of outcomes, and Program efficiency and economy.
4.1 Performance Tools and Measures
There is a Performance Measurement Strategy in place for the Financial Benefits Program and performance targets are defined; however, some modifications are suggested to strengthen measurement.
Data was being reported against many, but not all, of the indicators identified in the PMS. The unreported data was subsequently provided to the evaluation team.
The evaluation team noted some data limitations in the original PMS. For example, the reported numbers of ELB applications received, pending and adjudicated were misstated. This discrepancy is being investigated.
With respect to PIA, the evaluation identified inconsistent practices among Area Office staff when entering the receipt of applications in the Client Service Delivery Network (CSDN)Footnote 20. This inconsistency leads to inaccurate reporting of processing times and additional administration. Further, inconsistent tracking of application documentation could lead to misplaced applications. In January 2016, PIA processing procedures were revised to address this issue.
The evaluation team suggests the PMS be updated to add performance measures; i.e., Program cost per unit, cost per unit trends as compared to other programs (cost of inputs/number of outputs), and reporting on TPI decisions and reviews.
The PMS and Performance Measurement Plan (PMP)Footnote 21 was updated in July 2015, however, given this revision was completed outside the evaluation scope period, it was not assessed.
4.2 Achievement of Expected Outcomes
Outcomes – described as immediate, intermediate or ultimate – are the changes or differences that result from the program outputs. The following sections address each outcome in turn.
Immediate Outcome: Eligible Veterans and other Program recipients have access to income replacement or income support.
Access to income replacement and income support benefits is provided to eligible Veterans and other eligible Program recipients.
Program use has been increasing annually. The ELB is accessed the most, followed by PIA, including the PIAS. The favourable decision rates for ELB and PIA are high, indicating that most Veterans who apply are able to access the Program. Table 4 shows the number of decisions and favourable decision rates for the ELB, PIA and PIAS benefits in 2014-15.
| Benefit | Decisions | Favourable Decision Rates |
|---|---|---|
| Earnings Loss Benefit | 2,009 | 98% |
| Permanent Impairment Allowance Permanent Impairment Allowance Supplement |
768 704 |
82% 61% |
Source: VAC Finance Division.
The lower favourable rate for PIAS, as shown in Table 4, may be due to the application form, which is also used for PIA. Because an applicant need only check a box on the form to apply for the Supplement, most do so even if they clearly do not meet the eligibility criteria (out of a sample of 67 applications, 84% applied for both benefits at the same time and 49% of these were denied).
In 2014-15, VAC made 42 SRB decisions, of which 50% (21) were denials, largely due to lack of approval for the Rehabilitation Program (and therefore, eligibility for ELB), not being deemed TPI, or being under 65 years of age. This high denial rate could indicate that applicants are not fully aware of the requirements for SRB, and eligibility criteria are not clearly outlined on the application form. Improvements to communications including application forms are further discussed in Subsection 4.3.
Veterans must be deemed TPI prior to age 65 to be eligible for SRB. In an analysis of 34 files, 10 Veterans were deemed TPI after age 65 (median age at which they were deemed TPI as 65.4 years), and therefore not eligible for SRB, although they were participating in Rehabilitation prior to age 65. The average SRB amount to which they would have been entitled, if they had been deemed TPI earlier, was $1,183.
Recommendation 1
It is recommended that the Director General, Service Delivery and Program Management, implement a monitoring system for Financial Benefit Program recipients likely to be deemed totally and permanently incapacitated to determine TPI eligibility before they turn 65.
Management Response
Management agrees with this recommendation. A CSDN system change will be required in order to ensure that Veterans participating in the Rehabilitation Program who are soon to turn 65 years of age can be proactively identified.
Management Action Plan
| Corrective action to be taken | OPI (Office of Primary Interest) | Target Date |
|---|---|---|
| 1.1 Submit Request for Change form to National Information Technology Service Desk seeking implementation of functionality of a CSDN system-generated work item that will be sent to the relevant AO to request consideration of a TPI designation when a Veteran participating in Rehabilitation Program turns 64 years of age. | Health Care, Rehabilitation and Income Support Programs Directorate | April 2016 - Completed |
| 1.2 Given current obligations and expected future commitments, the Information Technology, Information Management and Administration groups expect that they can fully implement the above functionality by April 2017. | April 2017 |
The timeliness of decisions also impacts the immediate outcome. The time from the date a complete and signedFootnote 22 application is received by VAC to the date of the decision falls within the service standards for ELB and PIA. The overall turnaround time (i.e., the turnaround time from the Veteran’s perspective) is the period from the first day the Veteran submits an application to the date a decision is rendered, including delays incurred while VAC waits for supporting documentation to be submitted. The overall turnaround time exceeds the standards. Table 5 shows the timeliness of decisions as determined from a file review.
| Benefit | Service StandardFootnote 23 | Completed Application to Decision Time |
Initial Application to Decision TimeFootnote 24 |
|---|---|---|---|
| Earnings Loss Benefit | 28 days | 21 days | 76 days |
| Permanent Impairment Allowance | 84 days | 72 days | 132 days |
| Permanent Impairment Allowance Supplement | None | 69 days | 107 days |
Source: VAC Performance Measurement Plan and VAC Audit and Evaluation Division.
Although high program recipient rates indicate that the immediate outcome is being met, shortened decision timelines would improve access. Efficiency is discussed more fully in Section 4.3.
Intermediate Outcome: Eligible Veterans and other eligible Program recipients are able to fund basic needs.
Eligible Veterans and other Financial Benefits Program recipients, especially those with smaller household sizes, are generally able to fund basic needs.
The ELB ensures that participants in the Rehabilitation Program receive a minimum income of $42,426 (in 2015). In addition to ELB, Program recipients may qualify for additional assistance, such as PIA and PIAS.
A file review of 130 ELB applicants in 2014-15 indicates that their monthly military salary ranged from $2,700 to $21,072. Their median salary upon release was $59,040 annually. Applying the 75% ELB rate to that amount equates to approximately $45,000 which exceeds the guaranteed ELB minimum annual amount of $42,426.
One method of determining the adequacy of Program benefits is to compare them to Canadian median income by family type, keeping in mind that ELB, PIA and the PIAS are individually based and are unrelated to family type.
The minimum ELB payable falls below the average total income for all census familiesFootnote 25 and couple familiesFootnote 26 and exceeds the average total income for lone parent familiesFootnote 27 and non-household census families.Footnote 28 This trend continues for those in receipt of minimum ELB and PIA and those in receipt of minimum ELB, PIA and PIAS, with the gap getting smaller with respect to all census families and couple families and getting larger with respect to lone parent and non-household census families. Chart 6 illustrates the comparison.
Chart 6 – Median Total Annual Income by Census Family Type Compared to Minimum Annual Financial Benefits Amounts, 2015
Chart 6 – Median Total Annual Income by Census Family Type Compared to Minimum Annual Financial Benefits Amounts, 2015
| Economic Family Type | Median Census Family Income | Minimum ELB | Minimum ELB + PIA | Minimum ELB + PIA + PIAS |
|---|---|---|---|---|
| All | $78,921 | $42,426 | $49,446 | $62,346 |
| Couples | $86,684 | $42,426 | $49,446 | $62,346 |
| Lone Parents | $41,631 | $42,426 | $49,446 | $62,346 |
| Non-household | $23,506 | $42,426 | $49,446 | $62,346 |
* Note: The amounts of financial benefits (ELB, PIA, and/or PIAS) provided in Chart 6 above do not reflect additional income potentially earned by the householdFootnote 29.
Source: Statistics Canada and VAC Audit and Evaluation Division.
While studying the trend, it is important to note that, in 2014-15, 21% of those who received a payment for ELB were also in receipt of PIA while 17% of them were in receipt of PIAS. The majority (79%) of individuals in receipt of ELB were not receiving payment for another financial benefit in 2014-15.
Another measure of benefit adequacy is a comparison with low income measures. Statistics Canada measures of low income in Canada: Low Income Measure (LIM)Footnote 30, Low Income Cut-offs (LICOs), and the Market Basket Measure (MBM).Footnote 31 The LIM, used in VAC’s Life after Service Studies (LASS)Footnote 32, is slightly higher than the other two. As with the annual income levels by census type discussed above, LIM reflects family size, not individual income as elaborated on in the note below Chart 6.
The minimum ELB payable is higher than LIM for smaller families (less than four members) and lower than LIM for larger families (four or more members). According to the 2011 Census, the average number of children at home per family is 1.1; therefore, it is expected that most program recipients would have a household size of 2 to 3, meaning that the ELB payable exceeds the LIM for most. Table 6 compares LIM and ELB payable.
| Measure | Before Tax, by Household Size | |||
|---|---|---|---|---|
| 1 | 2 | 3 | 4 | |
| ELB (minimum 2015) | $42,426 | $42,426 | $42,426 | $42,426 |
| LIM (adjusted to 2015Footnote 33) | $24,140 | $34,139 | $41,813 | $48,281 |
| Difference between ELB and LIM | $18,286 | $8,287 | $613 | ($5,855) |
Source: VAC Research Directorate and VAC Audit and Evaluation Division.
One of the performance measures identified in the PMP specifies that those in receipt of ELB will not have an individual total annual income more than 5% lower than LIM; however, since ELB is based on individual income and LIM on household income, using LIM as a measure may need to be reviewed.
VAC processed 40 CFIS applications during 2014-15. Thirty-five percent (35%) of the decisions (14 out of 40) were denials due to the applicant’s household income exceeding the allowable ceiling. CFIS rates are below LIM for all household sizes, with before-tax amounts ranging from $6,646 (household size of 1) to $13,573 (household size of 4). Table 7 compares LIM to CFIS amounts (after tax) payable.
| Measure | Before Tax, by Household Size | |||
|---|---|---|---|---|
| 1 | 2 | 3 | 4 | |
| CFIS (2015) | $17,494 | $26,533 | $30,621 | $34,708 |
| LIM adjusted to 2015Footnote 34 | $20,992 | $29,687 | $36,359 | $41,983 |
| Difference between CFIS and LIM | ($3,498) | ($3,154) | ($5,738) | ($7,275) |
Source: VAC Audit and Evaluation Division.
At the current level, the amount of CFIS payable does not appear to be sufficient to meet basic needs (food, shelter, clothing, health care and transportation) of program recipients. Since 35% of applicants for CFIS are denied, mostly due to having income in excess of the allowable amount, the rates should be reviewed to determine if they should be increased. This statement is consistent with research which notes that VAC should consider the rates of CFIS as they are below the social adequacy of benefits.Footnote 35
Recommendation 2
It is recommended that the Director General, Policy and Research propose adjustments to the rates for the Canadian Forces Income Support so that they are sufficient to meet a person’s basic needs.
Management Response
Management agrees with this recommendation.
Management Action Plan
| Corrective action to be taken | OPI (Office of Primary Interest) | Target Date |
|---|---|---|
| The Department is always striving to find ways to improve programs and services. Over the past year, the focus has been on implementing the changes recommended by the parliamentary committee focused on Veterans' issues. The Department recognizes the importance of financial supports for Veterans. As such, a comprehensive analysis of existing financial benefits is about to begin. The issue of the adequacy of the Canadian Forces Income Support rate will be considered as part of that analysis. |
Policy and Research Division | December 2016 |
Ultimate Outcome: Basic needs of eligible Veterans and other eligible Program recipients are met.
The basic needs of eligible Veterans and other eligible recipients are generally met by the Financial Benefits Program.
To determine achievement of the Program’s ultimate outcome, the evaluation team used a combination of survey data and interviews.
A review of data from the Rehabilitation Intake and Program Completion Survey Results of those completing the Rehabilitation Program indicates a high level of satisfaction that their income was sufficient to meet their basic living expenses or better. These results are illustrated in Table 8.
| Time Period | 2010-11 | 2011-12 | 2012-13 | 2013-14 |
|---|---|---|---|---|
| Number of respondents entering program | 421 | 368 | 334 | 107 |
| Income reported sufficient at Rehabilitation Program entry | 84% | 82% | 89% | 82% |
| Number of respondents completed the program | 147 | 329 | 308 | 273 |
| Income reported sufficient at Rehabilitation program completion | 94% | 88% | 90% | 92% |
| Income reported sufficient during Rehabilitation Program, for those recently completed | 83% | 87% | 88% | 94% |
Source: VAC Performance Measurement Plan.
Interestingly, survey results do not support the previously-noted findings that financial benefits are insufficient for certain family types. This discrepancy could be because interviewees were responding based on all financial benefits, not each benefit individually.
Despite the positive response, it is important to note the survey participation rate of 33% (697 out of 2,110), is considered below adequateFootnote 36 and may not be representative of the population. In addition, the survey does not take into consideration those that are still participating in the Rehabilitation Program.
Interviews with Case Managers and Veteran Service Team Managers who have direct interaction with Program recipients also indicate that financial needs are being met. Of the 33 interviewees, 30 (91%) agreed that the financial benefits (ELB, PIA, PIAS) were sufficient to meet the needs of individuals participating in the Rehabilitation Program. The three interviewees who answered “no” assessed the benefits individually; i.e., ELB alone is not enough, CFIS alone is not enough.
Unintended Outcome
Veterans who complete the Rehabilitation Program and are able to earn a minimum of 66 2/3 percent of their pre-release salary are considered “suitably and gainfully employed”. However, a Veteran who remains in the Rehabilitation Program continues to be eligible for an ELB, which is 75% of their pre-release salaryFootnote 37. Therefore, the discrepancy between the income of what a Veteran might receive after completing the Rehabilitation Program and what he or she is guaranteed while in Rehabilitation, not to mention potential eligibility for additional benefits (i.e., PIAS, EEL, SRB), can cause a disincentive to work and an incentive to stay in the program.
The majority (91%) of the 36 Case Managers and Veteran Service Team Managers interviewed identified that the Financial Benefits Program can be a disincentive to complete the Rehabilitation Program.
Research shows a fine balance between incenting and disincenting a person to return to work, arising from the amount of wage replacement benefit received. A 2014 literature reviewFootnote 38, conducted by VAC’s Research Directorate, revealed the following:
- Eight out of 11 studies reported that benefit levels had a significant negative association with employment.Footnote 39
- The likelihood of employment was reduced only at payment levels of more than $800 per month.Footnote 40
- The most robust study had a small but significant negative association. While there was evidence indicating that benefit level was negatively associated with employment, there was insignificant evidence of a high enough quality to determine the extent of that effect.Footnote 41
- High, but not all, levels of disability compensation from the United States Veterans Affairs create disincentives for employment.Footnote 42
- Participants who were receiving disability benefits worked fewer hours in compensated work therapy each week, earned less income, had a higher vocational rehabilitation dropout rate, and were less likely to be competitively employed at discharge.Footnote 43
Because the implications, from both a material aspect and a social aspect, are great if the Program is acting as a disincentive for Veterans to get well or return to work, it is recommended that VAC review and address the unintended impact that Financial Benefits is having on the Rehabilitation Program completion rates.
Recommendation 3
It is recommended that the Director General, Policy and Research review the relationship between Financial Benefits and the Rehabilitation Program completion rates and propose adjustments which support the Veteran outcomes.
Management Response
Management agrees with this recommendation.
Management Action Plan
| Corrective action to be taken | OPI (Office of Primary Interest) | Target Date |
|---|---|---|
| The Department is always striving to find ways to improve programs and services. Over the past year, the focus has been on implementing the changes recommended by the parliamentary committee focused on Veterans' issues. The Department recognizes the importance of financial supports for Veterans. As such, a comprehensive analysis of existing financial benefits is about to begin. The financial benefits associated with the Rehabilitation Program will be considered as part of that analysis. |
Policy and Research Division | December 2016 |
4.3 Demonstration of Efficiency and Economy
Overall, the Program is administered in an efficient and economical manner; however, there are opportunities for improvement.
Ratio of administrative costs to program costs
One indication of a program’s efficiency and economy is the ratio of administrative costs to program costs. VAC’s Finance Division provided the administrative costs which are based on an attribution model developed by the program areas and the Finance Division. From 2012-13 to 2014-15, Program costs increased while administrative costs decreased, resulting in a decreasing ratio and higher efficiency. Program cost increases are due to changes implemented since 2011 (as described in Section 3.1) as well as the year-over-year increase in Program recipients. Administrative costs decreased due to reduced VAC Departmental budgetary commitments. Table 9 shows administrative and program costs for fiscal years 2012-13 to 2014-15.
| 2012-13 | 2013-14 | 2014-15 | |
|---|---|---|---|
| Administrative Costs (000s) | $4,268 | $4,071 | $3,740 |
| Program Costs (000s) | $105,328 | $149,714 | $186,254 |
| Administrative to Program Cost ratio | 4% | 3% | 2% |
Source: VAC Finance Division and Audit and Evaluation Division.
PIA processing
The evaluation team found PIA-related processes to be inefficient. At the time of the evaluation, such processes involved several areas within the Department. Case Managers in Area Offices submitted PIA applications, after gathering supporting medical data and creating a docket in CSDN. In 2013-14, there were 208 Case Managers, each of which could be involved in gathering information and submitting PIA applications. With 768 PIA decisions made in 2014-15, and assuming the number of Case Managers remains at 208, each Case Manager would have been involved in approximately four applications, which does not allow them to become proficient in preparing applications. Through observation and file review, the evaluation team noted that varied business practices were being used in the AOs, indicating uncertainty among Case Managers with respect to PIA application process requirements. Case Managers themselves indicated that the PIA application process was confusing and called it a “paper exercise.” Twenty-seven out of 30 (90%) key informant interviewees cited bottlenecks and issues with the timeliness of PIA decisions, primarily due to large caseloads but also because of the time required to gather supporting information.
Since the evaluation period, some of the PIA processing procedures have been moved from the AOs to HO. This change was too recent for the evaluation team to gauge the effectiveness and consistency in preparation and processing of the PIA applications. The situation should be monitored to ensure there is compliance with the procedures and that the change has achieved its desired result: less administration for Case Managers and more efficient processing of PIA.
Recommendation 4
It is recommended that the Director, Health Care, Rehabilitation and Income Support Programs, conduct quality assurance review/activities regarding the Permanent Impairment Allowance process to ensure the new procedures are being followed and are achieving the desired result.
Management Response
Management agrees with this recommendation. The Performance Measurement Strategy for the Program includes the regular measurement of compliance performance through file review, with review topics identified by Program Management. A Program Level Compliance Review of the PIA is planned for the 2017-2018 fiscal year. This will allow sufficient time for stabilization of the PIA following the recent expansion of the PIA, effective 01 April 2015, as well as changes to business process, effective 18 January 2016.
Management Action Plan
| Corrective action to be taken | OPI (Office of Primary Interest) | Target Date |
|---|---|---|
| 4.1 Create a Terms of Reference for a Program Level Compliance Review | Health Care, Rehabilitation and Income Support Program Directorate |
31 March 2017 |
| 4.2 Complete the Compliance Review Report | 31 March 2018 | |
| 4.3 Implement Corrective Action Plan, if any. | 31 October 2018 |
CFIS and SRB processing
CSDN has not been updated to allow for automated processing of SRB and CFIS applications. Rather, the Centralized Processing Centre uses spreadsheets to manually calculate payable amounts for both programs. Desktop procedures and business processes have not been documented for CFIS application processing; therefore, decisions are made on a case-by-case basis leading to potential inconsistencies in decisions. Applications are tracked and paid through CSDN using the functionality established for War Veterans AllowanceFootnote 44 applications, but analysts must use workarounds within CSDN to ensure proper payments are issued (i.e., they must use a test environment to ensure calculations will work in production). SRB applications are tracked in CSDN but payment requests are not incorporated into the system and are sent to Finance for payment. The turnaround times reported for both SRB and CFIS are generated from CSDN but have to be manually verified and corrected.
The manual processes for these benefits are labour intensive and open to human error. Further, as CSDN is the Department’s system of record, all information relating to the Program should be included in it.
Recommendation 5
It is recommended that the Director General, Service Delivery and Program Management, in collaboration with the Director General, Information Technology, Information Management and Administration, incorporate Supplementary Retirement Benefit and Canadian Forces Income Support processing into the Client Service Delivery Network.
Management Response
Management agrees with this recommendation.
Management Action Plan
| Corrective action to be taken | OPI (Office of Primary Interest) | Target Date |
|---|---|---|
| Given current obligations and expected future commitments, the Information Technology, Management and Administration groups expect that they can fully integrate SRB and CFIS processing into the CSDN by 30 April 2018. | Health Care, Rehabilitation and Income Support Programs Directorate | 30 April 2018 |
Communication
With respect to communication with Veterans, the evaluation team found that in some cases, letters were not clear, attachments were not readable and application forms did not provide appropriate guidance for completion. The VAC Veteran-Centric Communications Task ForceFootnote 45 , in consultation with stakeholders, is reviewing how VAC communicates with Veterans through letters, forms and applications. Each and every form and letter used to communicate with Veterans must be understandable, relevant to the process, and used with the appropriate frequency. The task force is also analyzing the amount of assistance required by the Veteran/family to complete forms as well as the time given to the Veterans return the forms.
Recommendation 6
It is recommended that the Director General, Service Delivery and Program Management, further review letters, attachments and application forms sent to Veterans in regards to the Program.
Management Response
Management agrees with this recommendation.
The VAC Veteran-Centric Communications Task Force has conducted a review of frequently used forms and applications, including some of those associated with the Financial Benefits Program. They will be continuing with a review of letters and other correspondence in 2016, and making recommendations for improvement. Further, as the Program letters are converted into Adobe LiveCycle format, they are reviewed by program analysts within the Directorate to ensure they meet plain language requirements and are Veteran-centric. The Directorate also liaises on an ongoing basis with other Program areas and Service Delivery staff to identify communications needing improvement.
Management Action Plan
| Corrective action to be taken | OPI (Office of Primary Interest) | Target Date |
|---|---|---|
| As part of these ongoing processes, the Directorate will: 6.1 Identify the ten Financial Benefits communication documents most frequently sent to potential and present program recipients and consult with users in CPC and Disability Adjudication to identify other documents requiring review. |
Health Care, Rehabilitation and Income Support Programs Directorate; and Information Technology, Information Management and Administration Directorate |
September 2016 |
| 6.2 Review each document in order to assess adherence to plain language requirements and adjust wording as necessary. Request that any modified communication documents be added to CSDN build. | December 2017 |