4.0 Performance

4.1 Effectiveness

4.1.1 Client satisfaction and achievement of outcomes

Ratings from the 2020 VAC National Client Survey indicate a high level of satisfaction with VIP; 90% indicate they were satisfied with VIP. This is the highest satisfaction rating across the VAC suite of programs examined in the survey. VAC key informants confirmed they receive few complaints from VIP recipients. See Table 2 below for VIP performance information profile (PIP) outcomes.

These positive impacts were echoed in interviews and surveys with VAC key informants and VIP decision-makers who were of the view that VIP is effective in providing supports for a clean and safe environment, improving quality of life, reducing the burden of care for family members as well as providing care with dignity including palliative care at home.

*LASS (entire Veteran population, not only those receiving services from VAC and excludes survivors and primary caregivers) ** NCS (VAC clients including survivors and PCGs)

4.1.2 Performance measurement development

The PIP is a TBS-mandated performance management tool that is to be periodically reviewed and updated to remain relevant and support the performance monitoring of the program.^{Footnote 28} Within the PIP there is also a complementary logic model, a systematic illustration of how a program works and how outcomes are achieved.^{Footnote 29} Major components of the PIP for VIP including a logic model have been developed and are in place stemming from the 2016 evaluation recommendations. The PIP was last updated in 2018 and its logic model in 2013.

For VIP, the PIP has eight performance outcomes divided into two immediate outcomes, one intermediate outcome and five ultimate outcomes (see Table 2 above). Taking an aggregate view of these eight outcomes it was observed that three are directly tied to home care, two to recipients’ overall service experience with VAC and three to the wider Canadian Veteran population. Currently, the ultimate outcomes for VIP focus on the well-being of the entire Veteran population. While some system-wide targets on the entire Veteran population are important and align to outcomes tied to VAC’s Departmental Results Framework, more outcomes directly related to the population VIP serves would strengthen the mix of measurement outcomes. Five of the outcomes mentioned only Veterans and in three, the terminology speaks to other recipients, VIP recipients or Veterans and their families.

The outcomes do not directly refer to survivors or PCGs despite them making up 36% of VIP clients. Strengthening the inclusivity of VIP performance measurement outcomes would clarify performance of the program for key client populations, the 21,000 survivors and PCGs who are 96% women and 80% who are over 80 years of age. At the same time, key Performance Indicators which are each linked to a specific outcome do include all three VIP client groups: Veterans, Survivors and PCGs.

A user-friendly dashboard was also created and presented at the January 10, 2019, meeting of the Performance Measurement and Evaluation Committee.^{Footnote 30} Its development was a strong step forward in the development of performance measurement tools and regular reporting on performance measures would be recommended. The next step would be to regularly gather, analyse and report on the data through the dashboard as well as integrate any finding into the program delivery.

“I’m thrilled to have housekeeping support. I don't know what I'd do if I did not have it.”

“Without VIP I couldn't stay in my own home, and I want to stay in my own home as long as I can; especially during this pandemic it’s helped me to feel safer in my home.”

~ Surveyed VIP clients

It is an opportune time to review the PIP as the Department and Statistics Canada will no longer be undertaking the Life After Service Study (LASS), the last cycle being in 2019, which is linked to three ultimate performance outcomes for VIP. The last cycle of LASS was in 2019. The Department is working with Statistics Canada on the Canadian Community Health Survey and this survey has three key benefits; it includes all VIP client groups; Veterans, Survivors and Primary Caregivers, will happen every two years rather than every three years, and includes the most elderly as LASS was limited to Veterans who released in 1998 or later. This will enable more current performance monitoring and responsive programming.

The evaluation found VIP is effective in supporting Veterans to live independently in their home and community. Results from the 2020 VAC National Survey found the majority (91%) of VIP recipients rely on these supports to help them remain at home or in their community. Similarly, the survey of clients conducted as part of this evaluation found most recipients agree that they receive the VIP benefits and services they need to be able to live in their own home and community (81%).

As mentioned, within the continuum of care, programs such as VIP are intended to support staying at home, ultimately leading to delayed entry into long term care. Care at home meets the needs of individuals who prefer to remain at home and addresses system-level health infrastructure and care costs. The Canadian Institute for Health Information estimates that one in nine long term care residents could reasonably have remained in their home with adequate support.^{Footnote 31} For VIP, 59% of recipients surveyed for the evaluation agreed the VIP has helped them to delay moving to a care residence or facility.

Analysis of VAC administrative data indicated VIP recipients entering Long Term Care (under the Intermediate Care Component) were on average four years older than non-VIP recipients entering Long Term Care (under the Intermediate Care Component (see Table 3). The same analysis conducted in 2011, indicated that VIP recipients entering Long Term Care were on average two years older than non-VIP recipients. This is consistent with population data in Ontario where it was found that the average age to nursing homes increased over time from 83 years in 2000 to 84.3 years in 2015.^{Footnote 32}

4.1.3 GBA plus considerations

While there are currently data limitations to examine program outcomes using a GBA plus lens, the evaluation found five out of the six key informant groups interviewed (Senior Management, VIP Managers and staff, Other Managers, Subject Matter Experts and representatives from Veteran organizations) raised concerns related to how VIP serves different identity groups including those with intersecting identities such as older and younger clients, survivors and PCGs who are primarily women as well as Indigenous and racialized populations.

Specifically, concerns were raised about how historical, outdated gender-based assumptions may continue to impact access to VIP or the determination of needs for VIP. Historically, gendered assumptions about the division of household responsibilities informed the department’s assessment of the contribution of client relatives to care (i.e., spouses were assumed to contribute to housekeeping when they were women and grounds maintenance when they were men). Additionally, there are some concerns from VAC key informants on whether VIP benefits and services entitlement for PCGs, which are a continuation of the Veteran’s benefits, might continue to reflect these assumptions.^{Footnote 33} Related to these historical assumptions is the view from five of the six key informant groups about a reliance on spouses to provide home care with four respondents pointing specifically to a gap in terms of remunerating spouses.

Most surveyed VIP decision-makers felt VIP benefits are well-designed to consider the needs of older Veterans (79% indicated VIP worked well) and are less effective for other groups with about half of respondents indicating the program design worked well for survivors, PCGs and women. For example, a key informant group raised that a spouse may not be entitled to a VIP support if it was not in place when the Veteran was living in the home. Only a third of respondents indicated the design of VIP worked well to consider the needs of younger Veterans.

“The follow-up with Veterans and clients to re-assess their needs could use improvement. Every 3 years is not sufficient . . . It needs to be yearly.”

“I’d like to stress again how important the annual VIP follow-up is. Over my long career as a Veteran Service Agent. . . I find that the yearly annual follow-up is better than every three years, I’ve had multiple incidents where a Veteran required more assistance and the Veteran Service Agent was never notified, and the client's needs weren't addressed, and the family are upset with the department.”

~ VIP decision-makers

With respect to other identity groups, the 2020 National Client Survey found Indigenous VIP recipients were less likely to be satisfied with VIP overall compared to non-Indigenous recipients (64% vs 71% who strongly agree they are satisfied). Overall, the 2020 survey results also indicated VAC clients who are Indigenous or are a visible minority are consistently less satisfied across the VAC suite of programs and have lower health and well-being ratings.

The evaluation found that the department is undertaking actions to further collaborate with Indigenous groups, conduct research/analysis, and implement policies and practices that will help inform what factors and issues may be leading to lower satisfaction and well-being ratings among Indigenous Veterans. This work includes:

• Ongoing departmental efforts in response to the Government of Canada Standing Committee 2019 report titled INDIGENOUS VETERANS: FROM MEMORIES OF INJUSTICE TO LASTING RECOGNITION. VAC’s official response to the standing committee report identifies the work being undertaken to address the recommendations, which includes improving communications and collaboration with Indigenous Veterans’ Associations.
• Finalizing a GBA+ Policy which articulates departmental roles and responsibilities, and guiding principles with the purpose of ensuring GBA Plus is applied in the development, implementation and evaluation of all initiatives and across all work areas to better meet the needs diverse Veteran and employee populations, while helping to ensure the breadth of their experiences are reflected in programs, services, policies, research, legislation and communications.
• Launching the 2022 National Client Survey, which will collect data regarding the satisfaction and well-being ratings, inclusive of a small sampling of Indigenous Veterans, and conduct an analysis of these results to help identify potential areas for improvement.
• Using the census data to better understand the Indigenous Veteran population and the reach of VAC’s programs/services to this population.

Recommendation 2:

The Director General, Policy and Research in collaboration with the Director General, Service Delivery Program Management, determine if there are unintended gender-related impacts occurring relating to VIP services eligibility for primary caregivers.

4.2 Efficiency

4.2.1 Program delivery efficiency

To determine the efficiency of delivery of VIP, administrative cost estimates were examined. These administrative costs are associated with program delivery and include items such as salaries, overhead, employee benefits, and contract administration costs.^{Footnote 34} During the period under study, the allocation model used to distribute administrative costs across programs indicates administrative costs have trended upward during the period under study from a low of 7.3% in 2016-17 to 9.1% in 2020-21 (see Table 4).

During the period of study, VIP administrative costs saw a steady increase with the exception of a dip between 2017/18 and 2018/19, when the Department implemented a new allocation model to assign program administrative costs. Aside from this, the administrative cost increases have been attributed to the departmental onboarding of additional staff beginning in 17/18 and relative allocation to VIP.

Additionally, to continue to provide services efficiently during the COVID-19 pandemic, VAC Service Delivery Program Management removed the need for signatures on VIP contribution arrangements and added some flexibilities to services for Veterans in long term care who chose to leave the nursing homes to be cared for at home.

Source: VAC Financial Data, VAC Facts and Figures *Excludes VIP Intermediate Care ** 2020-21 expenditures are forecasted

Surveyed VIP decision-makers rated the delivery of VIP program as moderately to very efficient (79%). Surveyed VIP decision-makers selected some factors they felt promoted the efficient delivery of VIP: knowledgeable and/or experienced VIP staff (88% selected this factor) and appropriate decision-making authorities (74%). The introduction of the grant determination tool was noted in key informant interviews and the decision-maker survey as contributing to efficiency with greater automation in the assessment and use of a grant instead of reimbursement through claims. The option for online application to the program was also noted in interviews as an efficiency.

Conversely, specific factors that were viewed as inhibiting the efficiency of program delivery included lack of capacity and support within field operations. Surveyed VIP decision-makers, in particular, noted turnover among Veteran Service Agents detracted from efficient delivery (58% selected this factor) and in a related question, about a third (31%) did not agree that there is adequate capacity in their office to respond to applications and requests in a timely fashion. Lack of clarity in policy guidance (51%) and lack of easy access to knowledge/support (26%) were also identified as detracting from efficiency.

“Policy and Guidelines need to be updated and consistently applied across the country.”

“Better training and more consistent guidelines and messaging across the country, particularly in regard to Mental Health eligibility and also to VIP Personal Care amounts.”

~ VIP decision-makers

Finally, both program key informants and VIP decision-makers noted burdensome approval processes as compounding inefficiencies. Based on interviews and survey responses, examples of burdensome approval processes include:

• often multiple levels of approval for small rate exceedances and outdated rate ceilings, especially for personal care, and
• no cost-of-living increase for grants which triggers rate exceedance and re-assessment processes.

To improve efficiency, suggestions often mirrored the factors that respondents identified as inhibiting efficiency. Surveyed VIP decision-makers most commonly recommended clearer and streamlined policies, business processes and tools (21%) to improve both efficiency and consistency in delivery. This sentiment included, for instance, suggestions to ensure up-to-date policies are easily accessible by Veteran Service Agents, improve policies and tools for needs assessment (e.g., refine the grant determination tool, improve functional guidance and tools for assessing personal care and for mental health needs), clarify policies related to the role of health professionals and include scenarios or examples as part of the functional guidance. In 2020, to begin to address this, VAC completed a pilot to improve the clarity and ease-of-use of VIP functional guidance documents.

Other suggestions included:

• Increasing the number of Veteran Service Agents and training and mentoring of Veteran Service Agents were raised in the survey and focus groups with VIP decision-makers as a way to increase efficiency, consistency and to improve turnaround times for clients.
• A number of key informants and surveyed VIP decision-makers felt Veteran Service Agents should be granted increased discretion and system’s permission to approve VIP supports for amounts over the maximum allowable benefits. Related to this, regular review of rate maximums was recommended by a few decision makers to avoid time consuming layered approval processes required to obtain rate exceedances.
• Some key informants and VIP decision makers suggested streamlining eligibility for the program. For example, incorporating an age-based criterion to automate approvals to reduce administrative burden for older Veterans who can be assumed to have needs for housekeeping and grounds maintenance without a formal assessment of capabilities.
• Program Managers also indicated that VIP clients occasionally request modern food delivery supports such as meal kit services which are not clearly covered by the access to nutrition policy.

4.2.2 Program tools and resources

The grant determination tool was introduced in 2013 to replace receipt-based reimbursement for expenses incurred by Veterans and other clients for housekeeping and grounds maintenance. The grant determination tool ^{Footnote 35} is used to determine the need and amount paid for services two times per year to approved recipients. According to key informants, the introduction of the grant determination tool made a significant contribution to the efficient administration of VIP. It reduced the number of internal approvals required for claims and reportedly led to more consistency in decision-making for this benefit.

With the grant determination tool, the department accepted the risk of an honour-based approach with respect to the use of funds by recipients. Such self-directed approaches to care have also been found in other settings, such as in the US, to be effective and the flexibility and control is appreciated by clients.^{Footnote 36}

In the 2016 evaluation, it was also recommended to refine the grant determination tool, however, the management response deemed the grant determination tool to be working as intended. In this current evaluation, some surveyed VIP decision-makers still saw a need to review the grant determination tool for a variety of reasons:

• to account for client needs that may stem from mental health,
• to increase the number of gradations to more precisely estimate support, and
• to better distinguish between routine and non-routine housekeeping and maintenance tasks in the assessment.

Although 60% of clients who were surveyed for this evaluation indicated the housekeeping/grounds maintenance grant meets their needs, nearly 55% of clients felt the grant should be higher. Some clients and VIP decision-makers pointed out that while the maximum rate for VIP benefits is adjusted annually to account for inflation, recipients’ benefit arrangements, which means the amount of the grant, is not automatically adjusted annually for inflation. Additionally, service provider rate increases need to be addressed through a request for reassessment.

“As Veterans age or deal with the impact of their disabilities, they often require personal care above what can be provided by family to remain in their homes. Currently, personal care is considered a VIP service and subject to an annual maximum.

Frequently, due to the low dollar threshold for personal care, exceeding rates are applied and higher levels of Departmental sign off are required which is burdensome and can cause delays for Veterans.

There is an opportunity to re-position personal care as a stand-alone benefit with analysis to establish a new annual maximum contribution for personal care that would ease current challenges within the program.”

~ Key informant interview respondent

‘There were mixed views among key informants on the applicability of the grant model to other VIP components. Some key informants considered the VIP access to nutrition element a possible candidate for administration through a grant, but this view was not uniformly supported.

The program processes that received the most positive ratings from VIP decision-makers were consultations with other divisions to render decisions (59%); assessment of the eligibility of Veterans for VIP (56%); and qualification of Veterans under the frail designation (55%).

An aspect of the program that was flagged as not working well by VIP decision makers is assessing the capabilities or contribution of clients’ relatives (33%).^{Footnote 37}

A theme evident in the evaluation data collected was the increase in the complexity of some VIP cases and the need to better support Veteran Service Agents to address these complex cases. Cases involving personal care, for instance, are difficult to assess and case management services for VIP complex cases appear to be insufficient (20% of VIP decision-makers indicated this aspect of the program is not working well). Some key informants and VIP decision-makers suggested the need for a more rigorous tool to assess complex personal care needs and guidance to address the increasing amount of requests to exceed the maximum rate for personal care. They also suggested a review of the personal care rate and explained how this rate or the amount given for personal care can be limited by the other VIP services that a Veteran receives. Currently, personal care is under an umbrella of VIP home care services. Together, these items have a maximum rate. As a result, personal care options can be constrained by the amounts given for other services. Giving personal care it its own maximum rate unrelated to other home care supports would alleviate this issue of exceeding maximum rate requests.

4.2.3 Communications

Awareness of VIP is fostered through the VAC website, social media, and communications by Veterans organizations. The evaluation found few concerns around awareness of the program, especially for the grounds maintenance and housekeeping grant. In the client survey conducted for the evaluation, 80% of respondents agreed VIP information is easy to understand and 67% feel they understand the types of VIP benefits and services that are available to them. Whereas 28% suggested the program could be improved by increasing awareness through increased communication regarding the types of VIP services that are available.

With regards to communications, most VIP client survey respondents (80%) reported VAC employees are knowledgeable about VIP and 70% of VIP clients surveyed for the evaluation agree that contracted Health Claims Processor employees are knowledgeable about VIP program eligibility, benefits and services. Overall, VIP program management view the arrangement with the Health Claims Processor as working well. However, some VIP decision-makers in the area offices expressed concerns about lack of clarity of roles and responsibilities between Veteran Service Agents and the contractor’s analysts as well as the level of system documentation required by the Health Claims Processor.

In 2015, follow-up with Veterans and other clients who are VIP recipients was decreased from a mandatory annual follow-up to once every three years. This reduction was in response to criticism resulting from a media article related to follow-ups conducted by another organization that also provided benefits to Veterans. Current VAC follow-up is perceived to be inadequate by key informants and by respondents of the VIP decision-maker survey to understand and respond to changes in recipients’ needs, particularly for vulnerable and older clients. Many interviewees and VIP decision-makers suggested more frequent client follow-up, and this has been recommended in other reviews of VIP (e.g., by the Office of the Veterans Ombudsman).

Programs in other jurisdictions have more frequent follow-up or follow-up geared to the nature of the benefit (fixed term, short-term, long-term) such as in New Zealand. The client survey conducted for the evaluation indicated 36% of respondents reported a change in needs in the last year. However, less than half (47%) confirmed they had reported this change in need to VAC. The review of administrative data show that in 2019-20, 16% of files had been reassessed.

4.3 Unintended Impacts

The evaluation team noted several unintended impacts surrounding VIP:

• Assessing the needs of clients has evolved to become more reactive since the early nineties when there were home visits and the advent of over-the-phone follow-ups happening every three years. In the evaluation, staff expressed that these factors are of particular concern for elderly clients especially given assessment tools, though a flexible mix, can lack specificity on the impacts of the aging process.
• Despite having funds for support services through VIP, it is becoming harder for some VIP clients to access and retain these services, in particular for those in more remote communities. This is, in part, due to various pressures on the Canadian health care infrastructure including the repercussions of the COVID-19 pandemic. At the same time, VAC’s role is delineated to be financial and not the cultivation or maintenance of the health care and support service infrastructure on which VIP is dependent for effective service delivery outcomes.

Recommendation 3:

The Director General, Service Delivery and Program Management, work in collaboration with the Director General, Policy and Research to review and enhance the tools and processes to assess client needs. Priority areas include:

• Explore the development of a tool/instrument and associated process to support evidence-based and consistent assessment of personal care needs;
• Identify vulnerable clients and undertake more frequent follow-up with them clients; and
• Assess the precision of the Grant Determination Tool and its relative adaptability in view of evolving client needs, and the adequacy of the process for rate adjustments.