2.0 Scope and Methodology

2.0 Scope and Methodology

This evaluation was conducted in response to the Treasury Board of Canada’s requirement to ensure there is adequate evaluation coverage of risk and organizational spending. It was conducted in accordance with the directive and standards specified in their Policy on Results.

2.1 Evaluation Scope

The objective of this formativeFootnote 4 evaluation was to look for alignments between the existing ToD medical impairment criteria for the PSC, and the health-related assessment criteria of the APSC and CRB programs. These alignments could:

  • identify opportunities to be more efficient;
  • determine if unintended impacts are occurring among these VAC programs; and
  • inform/promote equitable program access for all VAC clients.

The evaluation will support program and policy decision making and inform the implementation of new departmental initiatives.

This evaluation focused on CAF Veterans assessed with more significant levels of disability and impairment. To explore the potential alignments of the program assessment criteria through data analysis, the evaluation team set a disability percentage of ≥40% assessment for a single disability condition as the evaluation threshold. This threshold was used for analytical purposes only to explore relationships, and does not suggest a standard for alignments based on assessment level. It is recognized that each program may benefit from examining alignments at a higher or lower threshold based on program intent and criteria.

2.2 Evaluation Questions

Three questions (see Table 1)—based on information collected from a planning phase staff survey in fall 2019 and preliminary staff interviews—guided the evaluation.

Table 1: Evaluation Questions
Recognition, Health, and Income Replacement Programs for Canadian Armed Forces Veterans with more significant disabilities/impairments
  1. Are there opportunities to streamline processes by using alignments between the Table of Disability medical impairment criteria for the Pain and Suffering Compensation, and the health-related assessment criteria for the Additional Pain and Suffering Compensation and Caregiver Recognition Benefit programs for Canadian Armed Forces Veterans with more significant disabilities and impairments?
  2. Are there unintended impacts occurring between VAC programs?
  3. Do current eligibility requirements provide equitable program access for all Veteran applicants, regardless of biological sex, gender identity, sexual orientation or other identity factors?

To focus on the evaluation objective, meet timeline targets, and comply with available resources, a number of areas were excluded from the scope of the evaluation. These include:

  • The delivery of specific program benefits/services (including VAC providers); Disability Benefits/Pain and Suffering Compensation and claims processing;
  • Case Management function;
  • Assessment of specific Treatment Benefit grids;
  • CAF Long Term Disability Insurance Plan (previously called SISIP);
  • CAF to VAC transition process;
  • Program relevance; and
  • Program logic models and progress towards achieving program outcomes.

2.3 Multiple Lines of Evidence

Multiple lines of evidence have been used to support the evaluation findings. The methods undertaken to support these lines of evidence are identified in Table 2.

Table 2: List of Methods
Methodology Source
Documentation Review and secondary research review
  • Departmental documentation/information has been reviewed to understand the program objectives/intent, authorities and requirements, complexity, context and any key issue areas. Documents included: departmental planning documents, policies, mandate letters, business processes, strategic documents, research papers, previous Audit and Evaluation reports, social media monitoring reports and National client survey results.
  • A comparative review of the 2006 Table of DisabilitiesFootnote 5 and program assessment criteria was conducted to explore potential alignments
Non-Departmental Government Document Review
  • Various non-departmental government documents were reviewed, including, Parliamentary reports, Budget Speeches/Plans, and Speeches from the Throne.
Consultations To support this evaluation, consultations with various stakeholders were conducted. These key consultations began early in the evaluation process and included:
  • Medical Advisory
  • Program Policy
  • Strategic Policy
  • The Office of Women and LGBTQ2 Veterans to highlight the scope of the evaluation and the relevance of one of the evaluation questions to their unit.
  • ADMs, DG’s and Programs Directors, presentations to Health Professionals, as well as presentations at Corporate Policy and Program Management Committee (CPPMC) and Performance Management Committee and Evaluation Committee (PMEC) as part of the Departmental Evaluation Plan.
Literature Review
  • During the course of the evaluation, a third party contractor conducted a research study on Veteran and non-Veteran specific disability benefits programs. This information was used to inform the evaluation in terms of how other countries/private organizations use alignments for similar programming. In addition, the evaluation team conducted an academic literature review to inform and contextualize the evaluation findings within existing research findings.
File Reviews (conducted by VAC employees and in adherence to privacy and personal information safeguard requirements)
  • Two File Reviews were completed which reviewed the relationship between a client’s disability condition and their relationship with the APSC/CIA program.
  • Each file review was conducted using stop and go sampling which pulls a random sample of a specific population and the evaluator looks at set number of clients until the evaluation team is satisfied that the hypothesis is validated.
  • For the purposes of these two file reviews, the evaluation team looked at a sample of 100 clients.
Statistical/Program Data
  • Comprehensive descriptive statistical analysis was conducted to support this evaluation using multiple program data sets. The available client data was pulled from VAC’s Client Service Delivery Network (CSDN) system as of March 31, 2019. This evaluation examined VAC client and program data as of March 31, 2019.
  • In addition to analyzing raw data from VAC’s CSDN system and the 2017 National Survey, published data was also utilized such as VAC’s Facts and Figures.
Survey
  • To help inform the scoping portion of the evaluation a planning phase staff survey was distributed to the following program decision makers: Veterans Service Agents, Case Managers, Veterans Service Team Managers, Field Nursing Services Officers, Field Occupational Therapy Service Officers, Disability Adjudicators, Benefits Program Officers, Benefits Program Officers (EBU) and Appeals Officers. This survey was used to garner feedback regarding eligibility determination, supporting program policies, processes, assessment tools, and applications. The survey was distributed to 1,080 employees and had 184 respondents.

2.4 Considerations, Strengths and Limitations

The evaluation team acknowledges the following with respect to the evaluation and findings:

  • On 1 April 2019, VAC implemented significant changes through the Pension for Life (PFL) initiative. PFL included the launch of new programming (PSC, APSC and the Income Replacement Benefit) and the implementation of a new client relationship management system called GCcase. APSC administrative client data was not available in GCcase for the evaluation period due to ongoing system development. To mitigate, the evaluation team used data from the program’s precursor, the Career Impact Allowance (CIA)Footnote 6, as a substitution to explore the relationship between the PSC and APSC.

    CIA program data was determined to be a suitable substitution for the APSC as:

    • Like the APSC program, the CIA had the same definitions of permanent and severe impairments laid out in the program regulations; and
    • The CIA measured the extent of the impairment using the same three grade levels now used by the APSC program.

    The key differences between the CIA and the APSC are the different program intents (economic vs. non-economic compensations) and that in order to be eligible for the CIA Program, the Veteran had to apply for and be participating in the Rehabilitation Program. As the CIA was an economic benefit, the department assessed the Veteran’s earning capacity and considered the number of years the Veteran had left to serve in the CAF as part of the grade level determination as well as any medical and physical impairments. It is important to note that earnings capacity and years left to serve are not considerations in the APSC grade level determination. Additionally, while participation in the Rehabilitation Program is not required to gain access to APSC, a barrier to re-establishment must be identified for APSC entitlement.

  • Any potential changes to the assessment process for any program can have significant cost implications that require approval from the Minister of Finance. As a result, any potential changes that are explored by the Department will require an accrual costing completed by the Office of the Chief Actuary and if new funding is required, any proposed changes must be submitted to the Department of Finance for consideration as part of the Federal Budget process.
  • The implementation of the GCcase system required considerable input from and impact on operational decision makers. Evaluation requests to operational staff were concise so as to minimize impact on their already busy workload.
  • At the same time as this evaluation, Strategic Policy began an initiative which was to consider the overall structure of VAC programming. As such, the efficacy of the general structure of VAC programming was not examined.