3.0 Audit Results

3.1 Data Integrity and Automation

The audit team identified a number of opportunities where the system could provide additional control to improve data integrity when processing first applications. Staff often manually transcribe key dates from one screen in the system to another screen. If the system automatically populated the dates, transcription errors^{Footnote 6} would be eliminated. In addition, some data is entered manually from drop down lists when the system could perform the function automatically. For example, an application is defined as favourable if one or more of the conditions being claimed on the application are deemed favourable. Staff must enter a favourable/unfavourable decision for each condition on one screen and then manually enter the decision for the overall application on another screen, which can lead to data entry errors. Staff who generate the reports on favourable rates have identified the inaccurate data field and have developed work arounds to ensure reporting is accurate; therefore, staff time is being used to manually enter information that is not being used.

My VAC Account ensures that applications submitted electronically through that system have all the key fields filled in. This ensures a more complete application and staff do not have to contact the applicant to gather additional information which is often missing from paper applications (such as a signature). However, once the application is received through My VAC Account, staff must manually create applications in CSDN and enter the information into CSDN from the application. More functionality in the system where fields could automatically populate from the My VAC Account system would reduce the likelihood of errors and free staff time to complete other tasks.

The audit looked at the integrity of key data fields used for generating performance information. The main data fields in CSDN used for reporting include the following:

• Application Form Received Date (AFRD) – the date VAC received the application;
• Service Standard Start Date (SSSD) – the date VAC received all the documentation to support the application (i.e., the date an application package was deemed complete);
• Decision Docket Decision Date (DDDD) – the date the adjudicator made a decision on the claimed conditions on the application.

The AFRD, SSSD and the DDDD are manually entered into the system and are determined by staff in accordance with a business process. The audit team observed VAC staff conducting data entry and noted control weakness for AFRD and SSSD. A file review to determine the accuracy of these dates found errors in the AFRD and the SSSD. The errors resulted in inaccurate calculations of turnaround times, however these inaccuracies did not significantly affect the accuracy of overall reported turnaround times. It is not always clear what the correct AFRD and SSSD should be and these dates are determined based largely on processing staff’s judgment. It would benefit data integrity if these dates were determined by the system thereby reducing manual entry and judgment errors. The audit team identified no significant errors in the DDDD.

There are some system controls over key date fields in CSDN (e.g. SSSD cannot be greater than the current date, SSSD must be greater than or equal to the date of application). The audit team tested these controls and found them to be working. While these controls are working, they do not prevent all errors. When data errors are noted by staff who generate reports, there is no official process by which these errors may be corrected, no one is tasked with identifying and correcting errors. Exception reports were generated in the past to identify errors but they were discontinued as the reports did not result in any corrections to the system data. The audit team identified instances of errors, for example future dated decisions and illogical age and military service combinations. These errors did not result in any errors in reporting and therefore were not material, as the issues are known to staff who generate reports. In addition, it is important to note that these errors had no impact on clients.

A review of claims pending pointed to applications which, according to policy, should have been withdrawn. Currently, when the Department is awaiting additional information from the applicant, a reminder is mailed to the applicant reminding the client that we are awaiting information from them. Follow-up calls may also occur. The policy allows for 90 days for the client to provide the information and if the documentation is not received within that timeframe, the application is to be withdrawn. As it stands, staff must identify these cases and manually withdraw them. The volume of applications pending in the cue which should be withdrawn is overstating the workload and contributing to the backlog. Withdrawing claims takes effort on the part of staff in terms of having to send letters, make phone calls and answer client’s questions. This process could be automated in the system. Also, consideration should be given to changing the terminology. The client can come back to the Department at any time with the additional information and from that perspective the application is not withdrawn, but rather is on hold awaiting information.

Recommendation 1

It is recommended that Director General, Centralized Operations Division, in collaboration with the Director General, Information Technology, Information Management, Administration and Privacy Division and the Director General, Service Delivery and Program Management, strengthen controls over key data fields in CSDN. (Essential)

Recommendation 2

It is recommended that the Director General, Information Technology, Information Management, Administration and Privacy Division in consultation with the Director General, Centralized Operations Division, establish the governance (i.e., processes and accountabilities) for ensuring integrity of data over disability benefits. (Essential)

3.2 Relevance of performance data

Reporting at VAC is done through the reporting system. Data is transferred from the live system (CSDN) to the reporting system approximately every two days and is two days in arrears. Raw data is compiled into report formats using a reporting application. A data dictionary exists which provides the name of the data field and a description of its contents, but it does not identify the location of the corresponding field in CSDN. As a result, given the complexities of the data, those running reports need to have a good understanding of the data they are querying.

Currently, reports are generated in three different parts of the Department for various purposes. The Statistics Directorate in the Finance Division under Assistant Deputy Minister, Chief Financial Officer and Corporate Services, is the official data reporting group who are responsible for public reporting and approving statistics prior to release. Service Delivery and Program Management Division under Assistant Deputy Minister Service Delivery generates reports for programs and workload statistics related to all VAC programs. Centralized Operations Division under Assistant Deputy Minister, Service Delivery, also generates workload and program reports which are specific to the programs administered or managed by that division.

The business goals of disability benefits are that Veterans receive timely adjudication of claims and Veterans feel that the disability benefits program is effectively managed to meet their current and future needs. A review of current performance data showed alignment with these business goals. Performance data consists primarily of turnaround times (TATs) and completion rates (% within 16 week service standard). Lack of additional performance data such as cost per unit and workload forecasts were identified as data gaps. Analysing changes in cost per unit would provide senior management with information to determine the cost-benefit outcome of program changes, for instance, increasing or decreasing staff, reallocating duties or streamlining processes. Workload forecasts would enable senior management to make informed decisions regarding future staffing and budget requirements. Currently, TATs are reported on a per application basis and this data is available for applications for one claimed condition. When multiple conditions are claimed on an application, the Department is unable to determine how long individual conditions take to adjudicate or which conditions have been referred for medical consultation. In 2016-17 the Department adjudicated 18,687 applications, representing 30,848 conditions, indicating that there are often multiple conditions on each application.

There are a large volume of reports issued weekly and some managers receive as many as fifteen reports per week. The reports do not provide a description of their content, the reason it was generated, or who is responsible for taking action. They also display different data but use the same terminology to label the data. For example, a report on completed applications generated by one unit draws the information at the point the decision is complete, as opposed to another unit reporting on completed applications drawn at the point the claim was paid. The reports would benefit from a description of the methodology used to create the report and what the report contains. The reports are issued in read-only format and are not interactive. Managers indicated they would like to be able to analyze the data in spreadsheet format rather than the current read-only format.

Recommendation 3

It is recommended that the Director General, Centralized Operations Division, in collaboration with the Director General, Service Delivery and Program Management and the Director General, Finance, improve reporting by providing managers with interactive reports and to clearly describe content, methodology and importance of reports. (Critical)

Managers further indicated they had the information they needed, but that the reports are not in real time, given the delay in uploading data from the live database to the reporting database. As a result, the reports do not identify workload assignment that occurred during the two day lapse while managers assign tasks in real time. To accomplish real time assignments, managers in the field use a series of filters and search parameters in CSDN in order to determine workload and assign priorities. This is a manual process and is specific to the geographic region for which the manager is responsible. Staff then employ a similar system of filters and search parameters to select the claims they will work on. Generally, staff select the oldest claim first, but if they are training on a particular type of claim, they will select it. Based on interviews and prior internal reports, filtering and searching to prioritize and select workload tasks can take from 1 to 2.5 hours per day per person. There is no national workload distribution tool that would provide current workloads, identify where the workload is building up and assign claims to staff based on their availability and which claims they are being trained on.

Recommendation 4

It is recommended that the Director General, Centralized Operations Division, in collaboration with the Director General, Service Delivery and Program Management, and the Director General, Information Technology and Information Management and Administration, develop and implement a national workload distribution tool. (Critical)

3.3 Audit Opinion

The objectives of the audit were to assess the adequacy of performance data and data integrity in regards to disability benefits. The audit team identified weaknesses in system controls, and although the data remained uncorrected, mitigating factors were established to ensure accurate reporting. The audit team further identified areas of inefficiency in terms of the withdrawal process, manual assignment and selection of workload, manual transcription of electronic data fields and additional work arounds required to ensure accurate reporting. Given that these issues are significant enough to compromise the sound management of the disability benefits programs the audit team determined the overall audit opinion to be “Requires Improvement”.