The evaluation was conducted in accordance with VAC’s 2017-2022 Risk-Based Audit and Evaluation Plan, and in compliance with the directive and standards specified in the Treasury Board of Canada’s 2016 Policy on Results.
The evaluation focused on an assessment of operational efficiency of the Treatment Benefits program management function (as described in section 3.2). It should be noted that the management of cannabis for medical purposes was considered to be out of scope as it is managed by a separate unit. Additionally, the evaluation did not assess the delivery of the Program itself or the FHCPS contract.
The time period covered by the evaluation was April 1, 2014 to September 30, 2017. Appendix C outlines the evaluation issues/questions assessed.
In support of developing the scope for the evaluation, a risk/calibration assessment was completed as informed by preliminary interviews, a document review, and data analysis. Based on the risk assessment results, as well as the identified need by the program area, it was agreed that the evaluation would focus on an assessment of operational efficiency of the program management function.
2.1 Multiple Lines of Evidence
The evaluation was formativeFootnote 15 in nature and was a process and utilization-focused evaluationFootnote 16.
The research methodology incorporated multiple lines of evidence, ensuring reliability of collected information and reported results. The lines of evidence used to evaluate the operational efficiency of the Program’s management function are shown in Table 2.
| Methodology | Source |
|---|---|
| Departmental Documentation and Secondary Research Review | The following Departmental documents/information were reviewed to understand the Program objectives/intent, their authorities and requirements, complexity, context and any key issue areas: planning documents, previous audits and evaluations, strategic documents, performance reports, research papers, and survey results.
Various program management unit documents were reviewed to understand the governance structure, workload, and key issue areas: policies, business processes, organizational chart, records of decisions/meeting minutes, and performance reports. |
| Non-Departmental Document Reviews | Various non-departmental documents such as: program literature from other federal department and other countries, and published journals and/or articles relating to process evaluations and program management were reviewed. Parliament reports, Budget Speeches, Speeches from the Throne were also reviewed for context purposes. |
| Interviews | Over 70 interviews were conducted with VAC senior management, VAC staff involved in the management and operations of the programs (including field staff), and other subject matter experts. Interviews were also conducted with staff from the third-party health claims processor who play a role in the delivery of the program. Interviews with other federal and provincial government department program management units were conducted to understand resource models and potential best practices. |
| Data Analysis | Financial and operational data collected for fiscal years 2014–15 to 2017–18 was analyzed, where available. |
| Work Observation / Site Visits | Site visits and work observations with the third-party health claims processor took place to observe processes and practices in place regarding the Treatment Benefits Program since the adoption of the new FHCPS contract. |
| Logic Model | A logic model with program management inputs, activities and outputs was created to assist in understanding the workload, priorities and roles and responsibilities of the Treatment Benefits Program Management Unit and unit staff. |
| File Review | A small review of 37 enquiries received by the Treatment Benefits Program Management Unit was conducted to better understand the types of enquiries incoming, the length of time required to respond, and if there were any key trends or issues. |
| Process Map | A map of the escalation enquiry process was created to assist in understanding the workflow steps and key parties involved in the process. |
2.2 Limitations and Considerations
The limitations and considerations noted below should be considered when reviewing the evaluation findings.
- Limited program performance measurement information has been collected and monitored for the past two years. The Program (as well as all other programs within VAC) is currently undergoing a process to revise all performance measurement strategies to align with the 2016 Policy on Results requirement for Performance Information ProfilesFootnote 17.
- There was limited management data available regarding outputs and activities completed by the Treatment Benefits Program Management Unit. To mitigate this risk, the evaluation worked closely with program staff to understand and document inputs, activities, and outputs of the unit. During the evaluation examination phase, it was clarified that the new tracking tool was measuring enquiries received (as opposed to completed enquiries) by the Treatment Benefits Program Management Unit and that not all activities/outputs were being tracked. Additionally the information was self-reported and manually tracked, therefore there is a risk of errors. The evaluation team considered conducting an activity-based assessment however due to the varying nature of the enquiries received it was difficult to obtain an adequate depiction of workload within the evaluation timeline.
- Significant Treatment Benefits Program Management Unit staff turnover occurred during the scope period of the evaluation which impacted workload as staff were learning new roles and responsibilities, and being trained by senior staff.
- Individual perceptions of workload may influence opinions from staff on efficiency. In order to mitigate personal bias the evaluation team conducted interviews with multiple staff members, including the manager, and reviewed a sample of enquiry files.