4.0 Findings

4.1 To what extent are the outputs delivered by the Treatment Benefits Program Management Unit in line with the original intent of the unit, and linked to program objectives?

4.1.1 Program Objectives

As highlighted in the Program’s logic model, the overall program objective is: “to provide funding for health care benefits so that eligible Veterans’ and other program recipients’ health care needs are met.” The Program objectives are documented in various sources: VAC Departmental Plan 2017–18, Performance Measurement Strategy/Performance Information Profile, VAC external website, and an internal Treatment Benefit Program Roles and Responsibilities document. The objectives listed within the documents are described and detailed at varying levels and the documents are not easily accessible to Head Office and/or Area Office staff.

4.1.2 Program Management Unit Roles and Responsibilities

When asking interviewees from both VAC Head Office and Field Operations what the expected roles and responsibilities for the Treatment Benefits Unit would be, the top three categories could be classified as:

1. Providing clear guidance and direction to field staff;
2. Providing timely and thorough communications; and
3. Overseeing and reviewing program components to improve efficiency and effectiveness (e.g. business processes and benefit grids).

The Treatment Benefits Unit provided a copy of their roles and responsibilities for the Program which was created in response to a recommendation within VAC’s 2009 Programs of Choice Analysis Audit. The document highlights the overall program intent/objective as well as roles and responsibilities of various sections of VAC in the delivery of the Program. According to the document, the Treatment Benefits Unit is responsible for:

• setting program objectives and providing strategic direction for the program;
• maintaining the benefit grids, including working with VAC systems and the third-party health claims processor to ensure that the products, services and prices on the grids are a fair representation of current market conditions and adequately address the needs of VAC clients while adhering to VAC regulations and policy;
• identifying program trends and responding to program related concerns;
• participating with the Federal Healthcare Partnership (FHP) in negotiating various provider agreements with provider associations from across Canada^{Footnote 20};
• investigating, interpreting and resolving systemic issues within the Treatment Benefits Program;
• participating on the Formulary Review Committee (FRC)^{Footnote 21}, Benefit Review Committee (BRC)^{Footnote 22} and the Treatment Benefits Integrated Network (TBIN)^{Footnote 23} to ensure that the Treatment Benefits program continues to adapt to the changing conditions of delivering health care benefits and services in Canada; and
• working in conjunction with the Policy and Research Division to ensure VAC policies are current and reflective of the goals of the Treatment Benefits Program and VAC.

The document has not been updated since the completion of the action plan from the audit recommendation in approximately 2010 and it is currently out-of-date. As stated in section 3.2, there have been numerous program-related changes since 2010, including modifications to the program management structure and program delivery structure. In its current form, the document does not clearly define the objectives and outputs of the Treatment Benefits Unit or the role of key internal and external partners (e.g., VAC Health Care Professionals Division, VAC Contract Administration Directorate, and the third-party health claims processor).

Further supporting the need for consistency and availability of program objectives, of the field staff interviewed, many had limited knowledge of the Program roles and responsibilities performed by the third-party health claims processor and by Head Office. The 2016 VAC Service Delivery Review^{Footnote 24} had a similar finding and an associated recommendation^{Footnote 25} to address this concern. As of February 2018, these actions were still outstanding. An internal departmental review in 2016 also proposed developing a strategic overview for each VAC program area to clarify program intent, direction and performance expectations.

4.1.3 Program Management Priorities

The priorities for the Treatment Benefits Unit are informally documented and discussed but are not formally distributed. The priorities identified include:

• Benefits Review Committee;
• Drug Formulary Review Committee;
• Multi-Disciplinary Clinics (POC 5);
• Health Related Travel (POC 2);
• Benefit Grids; and
• Business Processes/Guidelines.

While the Treatment Benefits Unit priorities were found to align with the roles and responsibilities of the unit, they are not aligned with the majority of work completed by unit staff (as discussed further in the following sections).

4.1.4 Program Management Outputs

The intended outputs of the Treatment Benefits Unit, as self-identified by unit staff, were found to be linked to the overall program objective. Staff identified the following outputs relating to Treatment Benefits:

• Enquiry responses/guidance on individual cases and general topics (VAC field staff and contract staff, Issues Resolution Officers^{Footnote 26}, Ombudsman’s Office, VAC senior management, Access to Information, media, and parliamentary questions);
• Business guidelines, rules and directives;
• Forms and letters;
• Benefit grid updates;
• CSDN updates;
• Memorandums of Understanding, Agreements, and drug Product Listing Agreements;
• Committee/meeting packages, attendance, and records of decision;
• Ad-hoc initiatives and reviews (including departmental working group membership);
• Training packages for VAC field staff and third-party health claims processing staff;
• Stakeholder presentations and meetings; and
• Action plans responding to internal and external audit and evaluation reports.

The evaluation team found that the outputs produced by the Treatment Benefits Unit support the activities and outputs in the program’s logic model (benefit authorizations, appeal decisions, payments, health care identification card processing) as seen in Appendix B.

According to interviews, observation, and self-reported activity data, the majority of staff time is spent responding to benefits-related enquiries. These enquiries range from simple clarifications to in-depth matters involving policy interpretation and consultations with other VAC staff. The level of effort expended on enquiries will be further elaborated on in section 4.2.3.

4.2 To what extent is the Treatment Benefits Program Management Unit optimizing its use of resources and producing the required outputs?

4.2.1 Resources (Inputs)

Inputs are the human, financial, or infrastructure resources needed to administer a program. The key inputs of the Treatment Benefits Unit are human and financial resources, regulations and policies, and the FHCPS and CSDN systems.

As mentioned in the introductory section 1.0, there have been a number of changes impacting the Program in recent years. As noted in Table 3, these changes include a loss (-3.4) of full-time equivalents (FTEs). Compounding the loss of these resources, there has been a significant departure of corporate knowledge within the Treatment Benefits Unit. In recent fiscal years, some indeterminate positions have been backfilled with non-permanent employees and/or employees new to the business area. As can be seen in Table 3, in 2013–14 100% of Treatment Benefits Unit FTEs were indeterminate, compared to 62% as of September 2017. New staff require training and time to learn the complex program benefit policies, processes, and intricacies.

Note: During the examination phase of the evaluation, the Treatment Benefits Unit lost an additional experienced analyst.

The great majority of Head Office interviewees noted that the Treatment Benefits Unit did not have sufficient resources to appropriately manage the program. Though not specific to only Treatment Benefits, a 2016 internal review indicated that there were too many staff enquiries and not enough functional direction expertise available.

4.2.2 Activities and Outputs

Activities are processes or operations that an organization completes using available inputs. Outputs are the direct products or services resulting from the activities.

According to interviews and self-reported tracking, the majority of the Treatment Benefits Unit resource effort is spent on operational items. Enquiries and escalations affecting Veterans are considered highest priority for the Treatment Benefits Unit. This focus is directly linked to departmental priorities of Veteran centricity, including the well-being of Veterans and service excellence, as well as program objectives.

When issues or risk areas are recognized Treatment Benefits Unit resources are assigned to work on these areas; however, the effort often cannot be fully dedicated as staff are also assisting in addressing enquiries.

Enquiry Escalation Process

Enquiries received by the Treatment Benefits Unit can be based on individual Veteran cases, provider specific issues, or related to general treatment benefit topics. Enquiries are received by the Treatment Benefits Unit from multiple avenues:

• SDAT system;
• Direct from field office and third-party health claims processing staff;
• VAC senior management;
• Issue Resolutions Officers; and
• Ombudsman’s office.

Interview results from third-party health claims processor staff and VAC field staff noted varying methods of enquiry escalation, and did not always include the Treatment Benefits Unit (e.g., local supervisors, Standards Training and Evaluation Officers^{Footnote 28}, SDAT, or direct to the Treatment Benefits Unit). There is limited documentation available for VAC field staff regarding when or how to escalate treatment benefit issues and requests for guidance. Current SDAT processes indicate that staff follow ‘local practices in place’ after which a supervisor submits a request through SDAT if they are unable to resolve the issue. The electronic SDAT system is used to submit, log, and track enquiries. The SDAT team distributes questions to the appropriate areas within VAC for action. (e.g., Treatment Benefits questions are directed to the Treatment Benefits Unit). SDAT enquiries received by the Treatment Benefits Unit are triaged by one central unit staff member. For a visual representation of the escalation process refer to Appendix F.

Interview results and program management activity tracking report that a number of enquiries (100-150 per month) are sent directly to Treatment Benefits Unit staff, including items considered of higher risk. The limited documented direction regarding the escalation process for treatment benefit issues can cause confusion for field staff leading to irregularities in the process and an inconsistent enquiry tracking method for the Treatment Benefits Unit.

Enquiries Received

The Program has seen an increase in the number of SDAT enquiries received in the last three fiscal years. There has been a 51% (226) increase in total SDAT enquiries between 2014–15 and 2017–18 YTD (January 2018). Estimates based on a review of 2016–17 data indicate that total 2017–18 SDAT enquiries are on track to match or exceed the previous fiscal year. The most recent year-over-year change in total SDAT enquiries (2015–16 to 2016–17) shows an increase of 40% (229), with enquiries from both VAC and the third-party health claims processor increasing. Table 4 highlights incidents logged from 2014–15 to 2017–18.

*The 2017–18 forecast data is based on data as of January 31, 2018 plus estimates for the last two months of 2017–18 based on previous fiscal year activity.

Based on interviews and an analysis of SDAT incident data from April 2014 to January 2018, there is an indication that the move in treatment authorizations from VAC Treatment Authorization Centres^{Footnote 29} to the third-party health claims processor during 2014–15 and 2015–16 has resulted in additional enquiries to the Treatment Benefits Unit. There has been an increasing trend in the number of SDAT enquiries received from April 2014 to March 2017^{Footnote 30} from the contractor (383 or 249%). There are a number factors that could be reasonably attributed to the rise in enquiries:

• Overall increase in treatment benefit authorizations (560,702 in 2014–15 to 673,474 in 2016–17, an increase of 20%);
• Experience/knowledge of new benefits/services takes time to build (plus the third-party health claims processor had to hire additional staff, which would be an added learning curve regarding Veterans and general VAC policies and processes);
• Business rules and benefit grids in need of review could be contributing to the increasing enquiries; and
• Third-party health claims processor does not have the same level of discretionary decision making that VAC staff do, and therefore if business rules are not clear to make decisions they are left escalating issues or enquiries to the Treatment Benefits Unit.

Interestingly, an internal review of the benefit grids completed in 2010 predicted that if (TAC) processing work shifted to the third-party health claims processor and VAC’s processes were not refined and streamlined, there was a risk that increasing volumes of work would be gradually pushed back to VAC. Based on the evidence reviewed, the evaluators are not able to definitively conclude if this prediction has been realized, however evaluation findings do indicate that the shift in work has contributed to additional work for the Treatment Benefits Unit.

Generally, third-party health claims processing staff and VAC field staff interviewed indicated that they have the support they need to deliver the Treatment Benefits Program and are able to resolve the majority of their requests locally without additional clarification or involvement from Head Office. As noted previously, there is no documented process for escalating issues/questions.

As many of the Treatment Benefits Unit Analysts do not have an extensive background in health care, it is important that consultations occur with subject matter experts. At present, there is limited documentation in place highlighting when Treatment Benefits Unit staff should consult with non-program area staff (e.g. health professionals or policy) and consultations are not typically documented within CSDN^{Footnote 31}. Based on interviews and a small review of SDAT enquiries (37 files), informal and ad-hoc consultations appear to be occurring with non-program area staff.

Activity Tracking Tool

To assist in gathering a full representation of workload, a new activity tracking tool was implemented in October 2017 for the Treatment Benefits Unit. The evaluation team had four months of information to assess as the evaluation progressed. During the four months (October 2017 to January 2018) 1,778 reported activities were recorded.

Enquiry-based work was the primary focus of capturing information, with little to no information recorded regarding strategic work. The evaluation team found limited evidence that the information gathered is used to help manage the Treatment Benefits Unit workload and/or to inform management decision-making.

The evaluation team determined that the current tracking tool could be enhanced to maximize electronic capabilities and to better inform trends and issues, as well as program management decision making. The tool would benefit from additional information being captured such as: enquiry responses, turnaround times, types of enquiries (e.g. by POC or health care provider type), and other work being completed by Treatment Benefits Unit staff (e.g. business process updates, BRC, and internal ad-hoc reviews).

4.2.3 Assessing Optimization of Resources

Due to the unavailability of past output information and organizational changes the evaluation was unable to accurately attain information regarding all Treatment Benefits Unit outputs for the evaluation scope period. During the evaluation examination phase, it was clarified that the new tracking tool was measuring enquiries received (not completed) by the Treatment Benefits Unit and that not all activities/outputs were being tracked. Therefore it is difficult to comment on improved production of outputs.

In order to facilitate knowledge growth and retention for the Treatment Benefits Unit, the Program Manager began moving from a POC specialist model to a model where unit analysts are more generalist in 2017. Interviews with Treatment Benefits Unit staff noted that weekly team meetings are also used as a forum for sharing information and case discussions.

Industry Comparators

The evaluation team identified the following federal government departments with similar federally funded health care programs:

• Canadian Armed Forces (CAF) [health coverage for CAF members];
• Royal Canadian Mounted Police (RCMP) [health coverage for RCMP members];
• Treasury Board of Canada [Public Service Health Care Plan (PSHCP) offers health plan for federal public servants and retirees, as well as a few other groups]; and
• Health Canada [Non-Insured Health Benefits Program (NIHB) for First Nations and Inuit.

Program delivery and management/organizational structures of the above programs were assessed through a document review, internet search and interviews with program staff. Each program has its own intricacies (e.g. CAF provides health care directly to CAF members and NIHB has more regional administration), with varying ‘client’ base size (e.g. NIHB and PSHCP serve a much larger population). In the end, there was no clear comparison structure for the management of the VAC Treatment Benefits Program.

The evaluation did not conduct comparisons with private insurance plans due to key differences in the program design and delivery structure, primarily that recipients of private plans co-pay portions of benefit coverage, eligibility is not linked to conditions (like pensioned conditions), coverage is the same for all members, and cost containment is a major driving factor.

Benefits Review Committee (BRC)

The BRC is an official VAC body which is meant to offer a forum for discussion, consultation, and recommendations regarding new and emerging health care interventions and benefits, and the related health professional groups. The committee membership includes many key subject matter experts within VAC (e.g. Health Professionals, Appeals, Case Management, Rehabilitation, Policy, etc.). Since January 2016, the entire BRC quorum has not formally met. One meeting was held in September 2017 but not all members were present.

In 2017, the Terms of Reference for the BRC were reviewed to align more closely with the newly enacted VAC drug Formulary Review Committee (FRC) Terms of Reference. The FRC was revamped in response to a recent OAG audit report, and interviews indicate that the committee is functioning more efficiently and effectively based on these changes. At the writing of the evaluation report, the BRC Terms of Reference had been in progress since July 2016 and were not yet finalized. There continues to be discussions between the Service Delivery and Program Management Division and the Health Professionals Division regarding membership composition and roles and responsibilities of members.

Interviewees agree that the committee is needed and there are a list of items awaiting discussion and review. This statement is further supported by the 2016 OAG Drug Audit and the 2017 internal Audit of VAC’s Governance, which both highlighted the importance of clear roles and responsibilities and documentation of decisions.

Based on a review of documentation and interview results, the evaluators determined that ongoing membership discussions, irregular meetings, and the lack of a finalized Terms of Reference are impeding efficient decision making for treatment benefits and services. The evaluation team also identified areas where the BRC could be further maximized including:

• Reviewing/driving treatment benefit trend and risk analysis;
• Acting as an advisory panel for special projects and ad-hoc reviews; and
• Incorporating a representative from VAC’s clientele group to ensure more stakeholder consultation is occurring.

4.2.4 Opportunities for Improvement

Due to the many staffing changes and insufficient tracking information, the evaluation is not able to definitively confirm that additional resources would fix the issues currently being experienced by the Treatment Benefits Unit or if the additional resources would be required on a long-term basis. Short-term additional staffing could allow the Treatment Benefits Unit to analyze potential issue/risk areas, better support program management, and improve program delivery. For example, updating business processes/rules and providing more attention to key areas such as benefit grid reviews, multi-disciplinary clinic issues, health related travel, updating drug product listing agreements, creating a repository of enquiry responses, and other potential areas of efficiency.

The Treatment Benefits Unit tracking tool is a good basis for a more in-depth product and, as such, there are improvements which could enhance its purpose and value. There are clear benefits of tracking activities/outputs of the Treatment Benefits Unit: justification of current resources and workload limitations, informing and aligning future resource needs, and monitoring of trend/issue areas that require further analysis.

Evaluation findings suggest a lack of awareness of roles and responsibilities regarding the Treatment Benefits Unit as well as inconsistencies in the current treatment benefit enquiry escalation process. Since there is limited documentation available for VAC field staff regarding when or how to escalate issues and requests for guidance, there is an opportunity to augment awareness and knowledge with field staff in these areas.

4.3 Are there any opportunities to improve efficiencies in the management of the Treatment Benefits Program?

4.3.1 Performance Measurement

In July 2016, the Policy on Results was enacted, replacing the previous Policy on Evaluation. Under the new policy suite, there is additional emphasis placed on monitoring and reporting of performance measurement for federal government departments, including requirements for reporting and consultation with Treasury Board Secretariat on program performance outcomes, indicators, and outputs.

Performance measurement is a key function that aides in effective program management. Performance measurement is generally described as the regular measurement of indicators and outputs established to track progress towards achieving the intended outcomes of a program. This information is used to assess the effectiveness and efficiency of programs and to inform day-to-day decision making in program management.

The 2014 VAC Health Care Benefits and Services Program Evaluation noted challenges regarding performance measurement. At the time of the current evaluation’s commencement, program performance measurement was two years in arrears. As such, there is no evidence of performance measurement being regularly collected, reported, or monitored by program management and used to support decisions made by the Treatment Benefits Unit.

4.3.2 Risk Management

Another important aspect of program management involves the identification, assessment, and prioritization of risks followed by an application of resources to mitigate the impact of negative events and to maximize possible opportunities. Risk management’s objective is to ensure that uncertainty does not hinder an organization from reaching its goals.

At present, informal priorities and risks are informally discussed by management and the Treatment Benefits Unit staff but they are not formally documented. Additionally, there is no evidence of formal trend or risk analysis being completed on the enquiries received by the Treatment Benefits Unit. It is expected that the BRC will provide a forum to discuss risks, however this role is not clear as the BRC’s Terms of Reference has not been finalized and an official BRC meeting did not take place during the evaluation project timeline (June 2017 to March 2018).

4.3.3 Data Analysis

The Program is large and there is much data available, however there has been limited analysis conducted in the last number of years. Ongoing monitoring and reporting of data would support program performance measurement and help identify issues/trends that could help improve the program delivery and management and therefore improve service for Veterans.

Though the evaluation did not assess authorization or transaction data at the POC or recipient type level, the evaluation team did note some areas of concern regarding consistency, integrity, and use of data to inform decision making. For example:

• Until requested by the evaluation team, the program performance snapshot was not generated for two years;
• Interviews with Treatment Benefits Unit staff supported that there has been little use of program data to manage the Program in the last few years; and
• Once analyzed by the evaluation team, the program performance snapshot data indicated that VAC was processing 65-75% of transactions, which is an error, the third-party claims processor actually processes nearly all Program transactions. The error had been ongoing for a number of years, indicating that this information is not regularly reviewed.

4.3.4 Stakeholder Engagement

Collaborating with stakeholder groups enables program managers to identify needs, issues and perspectives. Stakeholders can include: program recipients, internal and external partners, support teams, providers, and other organizations/groups affected by a program. As previously stated in section 1.0, results from the 2017 VAC National Veteran Survey indicate that program recipients are highly satisfied with the Program.

According to interviews with Treatment Benefits Unit staff and third-party health claims processing staff, some collaboration/discussions occur with various health care provider associations/groups. Some interviews noted a need for more focus on provider issue management. In fact, prior to 2014 the Treatment Benefits Unit had a staff member dedicated to this function. As discussed in section 4.2.3, there is also potential to include stakeholder representation on the BRC.

4.3.5 Lessons Learned

Lessons learned are experiences (positive or negative) gained from previous cases/projects. The purpose of identifying, documenting and sharing lessons learned is to apply knowledge derived from experience. The Treatment Benefits Unit meets weekly to discuss ongoing cases and to share information among the team. There is limited evidence that previously answered enquiries are used to inform future enquiries on similar topics. Ad-hoc and informal consultations among the Treatment Benefits Unit staff happen but there is no formal database to track previously completed enquiries. A concerted effort needs to be placed on gathering and analyzing this information as there has also been a significant turnover in staff in the Treatment Benefits Unit and this may assist in training and educating staff, and ensuring consistency in enquiry responses.

4.3.6 Other Report Findings

The 2016 Spring Reports of the Auditor General of Canada Report 4—Drug Benefits—Veterans Affairs Canada noted the following gaps in relation to POC 10, but which could be applied to the entire Program:

• Adequate process for making evidence-based decisions related to benefits
• Monitoring for trends of exceptional decisions (non-benefit grid items routinely approved)
• Trend utilization that could inform program management; and
• Applying cost-effective strategies.

Similarly, an internal departmental review of the benefit grids in 2010 found that there was no clear process for determining the parameters contained within the grids and that processes for assessing and making changes should be standard, well-understood, and hold up to financial and legal scrutiny. The review also recommended that a regular data reporting regime be established to help identify trends and issues.

4.3.7 Efficiency Initiatives Underway/Realized

Although it was not within the scope of the evaluation to assess whether efficiencies regarding the delivery of the program were being realized, the evaluators did note that several efficiency seeking initiatives occurred since the last evaluation examination phase (autumn 2012) or were underway:

• In 2013, a project intended to streamline business processes regarding benefit/service authorizations was completed. The project close-out report indicates that subsequent pre-authorizations were greatly reduced (over 80% of claims are being processed without VAC interaction) and internal and external websites were enhanced for easier navigation of information. The evaluation team did not find sufficient evidence to validate that the authorization reductions occurred as reported. This further supports the evaluation report’s finding regarding the need to improve data/trend analysis function for the program.
• In an effort to optimize client service and delegate authorities to decision makers, VAC area office staff received additional delegated authorities for the Treatment Benefits Program in 2014. The evaluation team did not assess whether the intended goals were achieved.
• As part of the new FHCPS contract, authorization work previously completed by various VAC Treatment Authorization Centres was consolidated with the third-party health claims processor in 2014–15 in order to gain processing consistencies and efficiencies. The efficiency of the FHCPS contract was outside of scope, however there is an increasing amount of treatment benefit enquiries from the third-party health claims processor, as well business rules and benefit grids in need of review, suggesting that there are opportunities for improvement.
• In 2012, a mapping exercise was completed to ensure appropriate linking of medical conditions to benefits. The exercise resulted in a ‘stop-light’ tool in 2015 which is used as a reference guide for decision making/consultation. Interviews with third-party health claims processing staff indicate the tool is helpful.
• During the evaluation the Program launched a review of Health Related Travel claims processing. The goal of the review is to streamline processes to reduce administrative burden, reduce error rates, and meet the needs of Veterans. The review was too early in the process for the evaluation to comment on results.
• During the evaluation reporting phase, the Treatment Benefits Unit was also undertaking a review of 20+ program-related letters and forms for the purposes of clarifying information, streamlining processes, and enhancing user experiences. These letters/forms were considered priority due to their high usage through FHCPS.