2.0 Scope and Methodology

2.0 Scope and Methodology

The evaluation was conducted in accordance with the directive and standards specified in Treasury Board of Canada’s 2016 Policy on Results.

2.1 Evaluation Scope and Questions

The evaluation covered the time period from April 2014 to March 2018.

In support of developing the scope for the evaluation, a risk/calibration assessment was completed as informed by preliminary interviews, a document review, and data analysis. Based on the risk assessment results, as well as the identified need by the service area, the evaluation objective and questions were determined and are outlined in Table 1.

Table 1: Evaluation Scope/Questions

Are VAC’s Case Management Services being provided to Veterans who require this level of support?

  1. Are there Veterans who should be getting Case Management Services that are not receiving them?
  2. Are Case Management Services currently being provided to Veterans who do not require this level of support?

Are the tools available for case management effective and efficient in achieving their intended results? (RRIT, RRIT-R, CNCI, Case Plan Tool)

Can VAC’s Case Management Services be enhanced by adopting practices and professional standards utilized in other federal government departments?

Do VAC’s Case Management Services support Veterans with complex needs, including women, men and gender-diverse individuals with many identity factors (GBA+Footnote 1) in addressing their needs?

Are VAC Case Management service standards/commitments:

  1. Being achieved?
  2. Appropriate?

Findings from this evaluation will be used to support decision making and implementation of departmental initiatives.

2.2 Multiple Lines of Evidence

The research methodology incorporated multiple lines of evidence, ensuring reliability of collected information and reported results. The evaluation uses a mix of qualitative and quantitative data sources, which are outlined below. The lines of evidence used for this evaluation are shown in Table 2.

Table 2: Sources of Information Reviewed During the Program Evaluation
Methodology Source
Document Review The following departmental documents/information were reviewed to understand the service objectives/intent, their authorities and requirements, complexity, context and any key issue areas: planning documents, memorandums of understanding and their annexes, policies, business processes, records of decisions, strategic documents, performance reports, research papers, and survey results.
Various non-departmental documents such as: literature from other federal departments were reviewed. Parliament reports, Federal Budget and Budget Speeches were also reviewed for context purposes.
Employee Survey A survey was distributed to VAC Case Managers to obtain feedback in the following areas:
  1. Tools (RRIT-R, RRIT, CNCI, Case Plan Tool)
  2. Caseload
  3. Reach
  4. Disengagement/Transition to another level of service
  5. Non-Case Manager Activities
Participatory Observation The evaluation team gathered information through participation on relevant project groups and/or was involved in analysis to support these group, including:
  • VAC’s proposed New Screening Tool / Risk Comparison Pilot
  • VAC’s Case Management Renewal working group
This information was used to inform evaluation questions relating to reach, tools and outcomes.
File Review A file review was completed to help assess reach, the achievement of Veteran outcomes, and the effectiveness/efficiency of case management tools. The evaluation team was supported by subject matter experts (SMEs) for the file review. The review was completed in two parts:
  1. Judgemental samplingFootnote 2 was used to randomly select 224 files to assess the reach of Case Management Services.
  2. A representative sample of 193 randomly selected files was reviewed to assess the outcomes achieved associated with Case Management Services. This sample size provided a confidence level of 95% (margin of error +/- 7%)
Interviews and/or Site Visits Over 80 interviews were conducted with VAC senior management, VAC staff involved in case management, and other subject matter experts. Interviews with other Government Department management units were conducted to understand resource models and potential best practices.
Statistical Analysis Financial, human resource, and operational data collected by VAC for fiscal years 2014 to 2018 was analysed.

2.3 Considerations, Strengths and Limitations

  • Case Management is not a program, thus there is no requirement for a Program Information ProfileFootnote 3 (Policy on Results). Without the requirement of a Program Information Profile, the associated outcomes for the service are not readily tracked and monitored. The evaluation team utilized available system data, a manual file review, and public opinion research information to assess performance.
  • VAC completed a National Survey of clients (Public Opinion Research) in 2017, however prior to this, the last survey conducted by the department was in 2010. The time gap between the surveys limited ability to complete trend analysis.
  • Departmental pilots and renewal projects that relate to VAC’s Case Management Services were occurring during the period of this evaluation. The evaluation team participated/assessed initiatives where able, however, as implementation will span over future years, the full impacts of these changes could not be evaluated at this time.
 
Date modified: