4.1 Are the tools available for Case Management effective and efficient in achieving their intended results?
The evaluation finds that changes are required in order to improve the effectiveness and efficiency of VAC’s case management tools.
During the period of the evaluation, VAC’s Case Management Support Services Directorate and VAC’s Research Division developed and piloted a new screening tool which would replace the Regina Risk Indicator Tool ((Section 4.1(a)) as well as the Regina Risk Indicator Tool – Reestablishment (Section 4.1(b)). The evaluation team participated in the assessment of this new tool and conducted a comparison analysis between the current risk tools and the new screening tool. Analysis available at the time of writing of this report is reflected in section 4.1(c) – Risk Comparison Analysis.
In addition to the assessment of the new screening tool, Sections 4.1(a) and 4.1(b) include activities the evaluation team completed in order to identify challenges and/or opportunities for improvement that should be considered for the new screening tool.
4.1(a) Regina Risk Indicator Tool
The July 2018 Case Manager Survey found that 35% of Case Managers agree/strongly agree that the RRIT is appropriately identifying Veterans at risk with managing independently in their community and identifying the potential need for Case Management Services. Through the survey, Case Managers identified risk areas that they felt were under-represented in the RRIT, which included mental health and family/social support; and risk areas that they felt were over-represented in the tool, which included aides to daily living and information on hospital stays.
The evaluation assessed whether the RRIT is appropriately identifying at-risk Veterans. During the evaluation period, 6,481 Veterans received an “At-Risk/ High Risk” RRIT score and were referred for Case Manager consideration.Footnote 18 Subsequently, of the 6,481 Veterans referred, 870 (13%) ended up receiving Case Management Services. This analysis indicates that the RRIT is not triaging efficiently as referrals are being sent to Case Managers for Veterans that did not require this level of support.
Interviewees noted that one of the challenges associated with the RRIT is that often elderly Veterans can score “high risk” but not require a referral to a Case Manager because their needs are being met appropriately by a Veteran Service agent and/or a field nursing services officer. According to procedures, if a Veteran is deemed as high risk on the RRIT, the employee conducting the RRIT does not have the ability to override the referral to the Case Manager.
The findings from this section on the RRIT link to Recommendation #2 (page 29).
4.1(b) Regina Risk Indicator Tool - Reestablishment
The July 2018 Case Manager Survey found that 46% of Case Managers agree/strongly agree that the RRIT-R is appropriately identifying Veterans who are at-risk for unsuccessful re-establishment to civilian life, and who have a potential need for Case Management Services. Through the survey, Case Managers identified risk areas that they felt were under-represented in the RRIT-R, which included mental health, family/social support, and addictions. They also identified risk areas they felt were over-represented in the tool, which included aids to daily living and information on hospital stays.
Veterans who score Moderate/At-Risk/High Risk on the RRIT-R are referred to a Case Manager for consideration for Case Management. The evaluation team completed data analysis which identified that during the evaluation period, 12,893 Veterans received a “Moderate/At-Risk/High Risk” RRIT-R score and would have been referred for Case Manager considerationFootnote 19. It was found that 11,237 (87%) of these Veterans ended up receiving Case Management Services. Of this group, 91% were in the Rehabilitation and Vocational Assistance Program. It is difficult to determine if the receipt of Case Management Services was a result of the Veteran receiving a moderate/at-risk/high RRIT-R score, or because they were eligible for the Rehabilitation and Vocational Assistance Program.
During interviews, Veteran Service team members and health professionals employees were divided on whether the RRIT-R is an effective risk screening tool.
4.1(c) Risk and referral comparative analysis
In support of assessing VACs new screening tool, a pilot was completed in order to compare the new screening tool results to the results of existing risk tools (RRIT and RRIT-R). During the period from March 2018 to June 2018, 166 Veterans that had recent RRIT/RRIT-Rs completed were contacted and assessed using the new screening tool questions. A comparative analysis of the existing risk tools to the new screening tool was completed by the evaluation team in order to analyse the impacts on Case Management Services.
Comparative analysis of 166 Veterans that had a RRIT/RRIT-R and a screening completed using the new tool found that the overall volume of Veterans who may receive a “referral for Case Management Consideration” will remain relatively the same (-1% change). Further analysis is reflected in Table 10 below.
| Key indicators | % of Referrals based on existing RRIT/RRIT-R (High Risk + At Risk + RRIT-R Moderate Risk) | % based on new Screening Tool (High Risk Scores) | % Variance |
|---|---|---|---|
| Overall (n-166) | 28% | 27% | -1% |
| Veterans < 65 years old (n-88) | 41% | 40% | -1% |
| Veterans 65-84 years old (n-46) | 11% | 9% | -2% |
| Veterans 85+ years old (n-32) | 19% | 19% | 0% |
| Veterans with Fair/Poor SR MH (n-78) | 51% | 54% | 3% |
| Veterans - No one to count on (n-21) | 38% | 62% | 24% |
| Veterans - Possible alcohol dependence (n-21) | 62% | 52% | -10% |
| Veterans - Addiction/recent addiction (n-10) | 100% | 70% | -30% |
| Male Veterans (n-146) | 28% | 26% | -2% |
| Female Veterans (n-19) | 32% | 37% | 5% |
Although the overall volume of Veterans being referred will remain relatively the same, the composition of Veterans being referred will change significantlyFootnote 20.
- Of those that will be referred, it is estimated that 56% are the same Veterans as those that would have been referred under the RRIT/RRIT-R.
- Approximately 44% of high risk scores will be for Veterans who would not have been referred through the RRIT/RRIT-R.
- Approximately 47% of the Veterans who were referred previously through the RRIT/RRIT-R will not be considered as high risk based on the new screening tool.
In comparing the results of this analysis to the feedback received through the July 2018 Case Manager Survey, the evaluation team found that the new tool provides potential enhancements for the following groupings of clients:
- Mental Health and Family/Social Supports - Two of the risk areas that Case Managers identified as being “under-weighted/scored” in the current RRIT/RRIT-R are mental health and family/social support. Referrals for Veterans with these types of risks will increase based on the new screening tool.
- Aides to daily living and Hospital stays - Two of the risk areas that Case Managers identified as being “over-weighted/scored” in the current RRIT/RRIT-R are aides to daily living and information on hospital stays. The new screening tool has reduced emphasis on these areas.
The evaluation team found that there was a potential gap in the new tool for one of the risk areas that Case Managers identified as being “under-weighted/scored” in the current RRIT/RRIT-R (addictions). Referrals for Veterans with addictions related risks may decrease based on the new screening tool.
The findings from this section on the risk and referral comparative analysis between existing risk tools and the new screening tool is linked to Recommendation #2.
4.1(d) Case Needs and Complexity Indicator (CNCI) Tool
The July 2018 Case Manager Survey found that only 16% of Case Managers agree/strongly agree that "The CNCI helps me to identify the time and effort required for me to work effectively on my assigned caseload.” When Case Managers were provided the opportunity to comment on the CNCI, 114 Case Managers provided significant, primarily critical, textual comments relating to the tool. The main concerns included:
- The CNCI is too subjective and results can vary between Case Managers;
- The CNCI is not accurate in identifying the amount of time/effort a Case Manager requires to work effectively on his/her assigned caseload;
- There is no value added in completing a CNCI; and
- It is taking time away from Veterans/caseload.
These concerns from Case Managers were reiterated further during site visits. Interviewees were critical of the CNCI, noting that: the tool is too subjective and that it is not valuable for Case Managers to be spending their time on it. Interviewees also stated that the tool is not being used as intended (for caseload allocation based on the Intensity Factor Indicator toolFootnote 21).
In support of assessing whether the CNCI is an efficient use of case management resources, the evaluation team completed data analysis and obtained feedback through the July 2018 Case Manager Survey. Data analysis found that during the evaluation period from April 2014 to March 2018, Case Managers completed the CNCI more than 114,000 times. The majority of the CNCIs completed (84%) were associated with the requirementFootnote 22 for Case Managers to complete a CNCI every 90 days. In analysing the results for CNCIs completed to meet the “90 day requirement,” it was found the overall CNCI score range (Low, Moderate or High) remained the same in 78% of cases when the previous CNCI was completed less than 90 days prior. Feedback received through the July 2018 Case Manager Survey found that 65% of Case Managers did not feel there was a need for the CNCI to be completed every 90 days.
The findings from this section on the CNCI are linked to Recommendation #2.
4.1(e) Case Plan Tool
This section of the evaluation report provides the findings associated with activities completed in assessing whether the Case PlanFootnote 23 tool is an effective and efficient tool that supports Case Managers in the documentation and monitoring of their case management practices and activities.
The July 2018 Case Manager Survey found:
- 51% of Case Managers agree/strongly agree that "The Case Plan tool in the CSDN is an effective platform to document my case management practices and activities.”
- 40% of Case Managers agree/strongly agree that "The Case Plan tool in the CSDN supports the efficient documentation of my Case Management Practices and activities.”
- 50% of Case Managers agree/strongly agree that "The Case Plan tool in the CSDN strengthens my ability to monitor Case Management practices and activities.”
In addition, Case Managers provided detailed comments relating to each of the 10 sectionsFootnote 24 within the Case Plan Tool. The evaluation team reviewed these comments and summarized the primary themes as follows:
- The Case Plan Tool is too administratively burdensome;
- Numerous sections are repetitive with information already captured in other sections of the Case Plan Tool or VAC’s Case Manager Assessment;
- The tool needs to be streamlined and simplified - Categories/Standard lists/Drop-down menu;
- The tool should directly align with (integrate/auto populate) other Case Management documentation (RRIT’s, CNCI, Area Counsellor Assessment, Rehabilitation Record of Decision, and Summary of Assessment);
- There should be direct links between Desired OutcomesFootnote 25 , Actions StepsFootnote 26 , and ResourcesFootnote 27 ;
- Certain sections within the tool should be linked for the client to access through My VAC Account; and
- There should be improved usability features, such as: ability to save in draft, edit functionality, spellcheck, and reading/viewing.
Similar concerns to those identified in the July 2018 Case Manager Survey were noted to the evaluation team during site visits. Primary concerns identified by interviewees included: too much repetition within the tool, a lack of linkages/alignment within CP tool and also with other case management tools, more efficient to have drop down/pre-populated options, and that there are challenges with usability/viewing/scrolling.
The evaluation team also observed challenges with the Case Plan Tool first-hand by sitting with a Case Manager and through participation in the file review. Some of the challenges observed by the evaluation team included: the usability of the tool, duplication of effort, and a lack of alignment/linkages between key elements of the tool (i.e., Desired Outcomes, Action Steps, and Resources).
In addition to the concerns relating to how the Case Plan Tool supports the Case Manager’s practices and activities, the evaluation team also encountered challenges with respect to reporting/performance results available through the Case Plan Tool. The evaluation team found there was incomplete/insufficient data to support outcomes measurement for case management recipients. Data obtained through the Case Plan Tool is able to reflect the overall volume of desired outcomes/action steps/resources completed for a case managed client. However, the tool does not provide any links to the types of domains in which case management is making a difference (i.e., Health, Mental Health, Employment, Social Integration, etc.). Therefore, in order to determine what outcomes are being worked on/achieved for Veterans receiving case management support, the evaluation team had to undertake a manual file review (results associated with the file review are reflected in section 6.4, page 37). In order for subject matter experts completing the file review to gain enough information regarding a Veteran’s needs and outcomes, they needed access to an average of 6 different sources within VACs system (the Case Plan Tool + 5 different forms/tools outside of the Case Plan Tool).
The file review results also identified there are opportunities to better align case management tools. The results of the review indicated that in instances when a need for improvement in a particular domain was identified through initial assessments/screenings; these needs were not always addressed accordingly within the desired outcomes for the Veteran. In this situation, if the tools were aligned/linked appropriately, it would help Case Managers in ensuring that needs are better reflected and monitored within the Case Plan.
The findings from this section on the Case Plan Tool are linked to Recommendation #2.
4.1(f) Administrative Burden
Through the evaluation, one of the concerns that the evaluation team was informed of during site-visits and through comments on the July 2018 Case Manager survey was the high levels of administrative-related burden on Case Managers.
Through the survey, Case Managers reported that they should be spending significantly less time documenting than what is occurring. The survey results are reflected in Table 11 below. The results show the majority of Case Managers reported spending > 50% of their time documenting and one third of Case Managers reported spending > 70% of their time documenting. When asked how much time they think should be allocated to documenting, the majority reported in the 21% to 50% range, a significant reduction from the current practice.
| % of Time | Overall, what portion of your time do you spend documenting your case management practices and activities? (% of Case Manager Responses) | Overall, what portion of your time should be taken to document all of your case management practices and activities? (% of Case Manager Responses) |
|---|---|---|
| Over 70% of your time | 32.7% | 5.8% |
| 61-70% of your time | 21.5% | 4.9% |
| 51-60% of your time | 18.3% | 8.1% |
| 41-50% of your time | 10.3% | 23.3% |
| 31-40% of your time | 10.3% | 22.4% |
| 21-30% of your time | 4.9% | 22.0% |
| 11-20% of your time | < 1% | 11.2% |
| Less than 10% of your time | < 1% | 2.2% |
Recommendation #2
It is recommended that the Director General, Service Delivery and Program Management improve the effectiveness and efficiency of case management tools by:
- Updating processes to:
- Have Case Managers complete a CNCI at entry/graduation from Case Management and when their professional judgment deems there is a change in need/complexity level, thus eliminating the requirement to complete it every 90 days;
- Increasing efforts to:
- Monitor the effectiveness of the new screening tool by developing/implementing performance metrics and quality management processes to assess/measure that clients are being triaged to the appropriate level of service.
- Implementing system/tool/process improvements (over the medium-long term, next 1-3 years) to:
- Join the case plan tool directly to any assessments completed which identify the needs associated with a case managed client (i.e., health, mental health, social integration, employment, etc.);
- Establish domains/categories that the case plan desired outcomes, actions steps, and resources can be assigned to;
- Link action steps and resources to the desired outcomes/goals they are associated with;
- Streamline and simplify the level of effort required within the Case Plan Tool by: eliminating possible overlap/duplication, creating standard lists and drop down options where appropriate, and using key fields to automatically generate resource authorizations;
- Integrate the ongoing assessment of complexity and need directly within the Case Plan, thus eliminating the CNCI tool;
- Provide options for information to be updated/shared through My VAC Account (where appropriate); and
- Improve usability features (viewing, reading, editing, spell-check, etc.)
Management Response:
Management agrees with this recommendation.
| Corrective Actions to be taken | Office of Primary Interest (OPI) | Target Completion Date |
|---|---|---|
| The Director General, Service Delivery and Program Management, work in collaboration with the Director General, Field Operations, to improve the effectiveness and efficiency of case management tools by: | ||
| 1. Implementing system and process changes to eliminate the need for a CM to complete a CNCI every 90 days. | Director General, Service Delivery and Program Management Division and Director General, Field Operations Division | June 2019 |
| 2. Developing a quality management framework for the new screening tool to ensure that Veterans are being triaged to the appropriate level of service, and implementing ongoing performance metrics to monitor the effectiveness of the new screening tool. | April 2020 | |
3. i) Leveraging the development of the client plan as part of Pension for Life to update and finalize new case plan/client plan requirements that will:
|
April 2020 | |
| 3. ii) Establishing an implementation strategy for the new case plan/client plan requirements, and if required, identifying/pursuing additional resource requirements. | ||
| 3. iii) Fully implementing a new case plan/client plan. | April 2021 | |