Appendix A – Evaluation Matrix

Appendix A – Evaluation Matrix
Evaluation Matrix
Questions Indicators Data Collection Method Source of Data
Issue: Relevance
1. What is the relationship between CTS and ETB and between these programs and other VAC programs in the continuum of supports for Veterans? Are these relationships appropriate/ working effectively to meet Veterans’ needs? 1.1 Descriptive information about the programs describe the linkage/relationship Document review Program documents, including policy, applicant guide, FAQs, deep dives, training materials, communications/outreach materials

Departmental documents including plans, reports, PIP
1.2 CTS/ETB participants’ take-up of other VAC programs, either before, during or after their CTS/ETB participation Data review CSDN database
Client database
1.3 CTS/ETB participants’ plans for next steps after program participation ends Survey CTS/ETB participants
1.4 Views on the nature, appropriateness and effectiveness of the relationship(s) between the two programs Interviews

Survey

Focus groups
VAC

CTS/ETB participants

VAC personnel (see notes)
2. To what extent are the eligibility criteria for CTS and ETB meeting Veterans’ needs? 2.1 Degree of alignment between eligibility criteria and expected program objectives/target beneficiaries Document review

Interviews
Program documents, policies, discussion papers

VAC
2.2 Evidence of barriers to access for target beneficiaries, considering GBA+ factors, rural/urban, region, age, rank, French/English Interviews

Survey

Focus groups
VAC

CTS/ETB participants

VAC
2.3 For ETB, eligibility criteria for educational institutions, programs and courses meets the needs of participants Survey

Data review
CTS/ETB participants

Client data base
3. To what extent do Veterans have unmet needs for career transition and/or accessing education or training? 3.1 Evidence of/views on unmet needs or gaps, considering GBA+ factors, rural/urban, region Interviews

Document review

Data review

Survey

Focus group
VAC

Program reports, recommendations from stakeholders/OVO, directives and operational guidelines; service delivery reports

CSDN database

CTS/ETB participants

VAC
Issue: Performance/Effectiveness
4. Are the programs achieving the expected/ desired reach? 4.1 # of applications for each program

4.2 # of formal program plans and short courses established

4.3 # favorable eligibility decisions for CTS, for Veterans, spouses, survivors (GBA+, regional/provincial)

4.4 Program utilization over time (i.e., # of clients applying versus # clients accessing the programs (specifically, enrolling education/course for ETB and accessing services for CTS), by type of client
Document review

Data review
Corporate reports

CSDN database
CRM database
Client database
4.5 Proportion of budget expended, by FY Data review Financial database
4.6 Awareness of the programs among front line staff Interviews
Focus groups
VAC
4.7 How participants become aware of the programs Survey CTS/ETB participants
5. Are there alternative program delivery approaches that could be considered that would increase outcomes for Veterans? 5.1 % clients satisfied with services they received from VAC and, for CTS, the service provider Document review

Survey
Program/corporate reports

CTS/ETB participants
5.2 Suggestions by program personnel, other VAC employees, program participants re opportunities for improvement (probe for partnerships/placements/matching with employers; link with Service Canada Job Bank, more hands-on support for CV development) Interviews
Focus groups
VAC
VAC
Survey CTS/ETB participants
6. How effectively have the programs supported Veterans to meet their personal and professional goals? 6.1 ETB: #/% clients who completed the education/course (or are on track to completion, e.g., completed their first year of study)

6.2 CTS: #/% clients who better understand their qualifications and their post-service professional opportunities

6.3 CTS: #/% clients who prepared a CV with assistance from CTS tools/service provider

6.4 Program utilization over time (i.e., # of clients applying versus # clients accessing the programs (specifically, enrolling education/course for ETB and accessing services for CTS), by type of client
Document review

Data review

Survey
Program/corporate reports
Client database

CTS/ETB participants
7. Were there unexpected outcomes (positive or negative) as a result of receiving services/benefits, including unintended GBA+ related impacts that should be resolved? 7.1 Evidence of unresolved unexpected negative outcomes

7.2 Evidence of unexpected positive outcomes that led to improved program performance
Interviews

Document review

Survey
VAC

Program reports, OVO reports, service delivery reports

CTS/ETB participants
Issue: Performance/Efficiency
8. Are the available program data for CTS and ETB adequate to measure its outcomes/impact? 8.1 Clarity of outcome statements, causal link between outcomes; reflection of best practice

8.2 Evidence/views that the programs are collecting and reporting data to measure its relevance, effectiveness and efficiency

8.3 Adequacy, reliability, timeliness and validity of indicators/data collected by VAC, the service provider (for CTS)

8.4 Evidence/views of use of performance information in decision-making
Interviews

Document review

Data review
VAC

Program/corporate reports, Performance Information Profile

All databases/sources
9. Have the activities of the CTS and ETB been delivered in an efficient and economical manner? 9.1 Average time and resources required to assess/process ETB applications, over time (variation by type of application, region and GBA+)

9.2 Trends in program administrative expenditures (including contract expenditures), in program activity (caseload, average payments, forecasts, VAC/SP turnover, etc.)
Document/data review Program reports/HR systems
9.3 % of CTS and ETB applications processed within service standards; % of formal program plans/short courses established within 4 weeks of receipt of completed education plan

9.4 % of payments made before the start of the program; % of payments made in the correct amount
Document review Corporate reporting
9.5 Average wait time and time in the program for CTS, over time (variation by type of service, region and GBA+) Document review Program reports, reports from service provider
9.6 Factors that contribute to efficient delivery, seamless delivery between the programs; evidence of inefficiencies (bottlenecks, duplication); more economical mechanisms to achieve same or superior outputs / outcomes

9.7 Suggestions, opportunities to improve efficiency
Interviews

Focus group
VAC

VAC