Appendix A – Evaluation Matrix
| Questions | Indicators | Data Collection Method | Source of Data |
|---|---|---|---|
| Issue: Relevance | |||
| 1. What is the relationship between CTS and ETB and between these programs and other VAC programs in the continuum of supports for Veterans? Are these relationships appropriate/ working effectively to meet Veterans’ needs? | 1.1 Descriptive information about the programs describe the linkage/relationship | Document review | Program documents, including policy, applicant guide, FAQs, deep dives, training materials, communications/outreach materials Departmental documents including plans, reports, PIP |
| 1.2 CTS/ETB participants’ take-up of other VAC programs, either before, during or after their CTS/ETB participation | Data review | CSDN database Client database |
|
| 1.3 CTS/ETB participants’ plans for next steps after program participation ends | Survey | CTS/ETB participants | |
| 1.4 Views on the nature, appropriateness and effectiveness of the relationship(s) between the two programs | Interviews Survey Focus groups |
VAC CTS/ETB participants VAC personnel (see notes) |
|
| 2. To what extent are the eligibility criteria for CTS and ETB meeting Veterans’ needs? | 2.1 Degree of alignment between eligibility criteria and expected program objectives/target beneficiaries | Document review Interviews |
Program documents, policies, discussion papers VAC |
| 2.2 Evidence of barriers to access for target beneficiaries, considering GBA+ factors, rural/urban, region, age, rank, French/English | Interviews Survey Focus groups |
VAC CTS/ETB participants VAC |
|
| 2.3 For ETB, eligibility criteria for educational institutions, programs and courses meets the needs of participants | Survey Data review |
CTS/ETB participants Client data base |
|
| 3. To what extent do Veterans have unmet needs for career transition and/or accessing education or training? | 3.1 Evidence of/views on unmet needs or gaps, considering GBA+ factors, rural/urban, region | Interviews Document review Data review Survey Focus group |
VAC Program reports, recommendations from stakeholders/OVO, directives and operational guidelines; service delivery reports CSDN database CTS/ETB participants VAC |
| Issue: Performance/Effectiveness | |||
| 4. Are the programs achieving the expected/ desired reach? | 4.1 # of applications for each program 4.2 # of formal program plans and short courses established 4.3 # favorable eligibility decisions for CTS, for Veterans, spouses, survivors (GBA+, regional/provincial) 4.4 Program utilization over time (i.e., # of clients applying versus # clients accessing the programs (specifically, enrolling education/course for ETB and accessing services for CTS), by type of client |
Document review Data review |
Corporate reports CSDN database CRM database Client database |
| 4.5 Proportion of budget expended, by FY | Data review | Financial database | |
| 4.6 Awareness of the programs among front line staff | Interviews Focus groups |
VAC | |
| 4.7 How participants become aware of the programs | Survey | CTS/ETB participants | |
| 5. Are there alternative program delivery approaches that could be considered that would increase outcomes for Veterans? | 5.1 % clients satisfied with services they received from VAC and, for CTS, the service provider | Document review Survey |
Program/corporate reports CTS/ETB participants |
| 5.2 Suggestions by program personnel, other VAC employees, program participants re opportunities for improvement (probe for partnerships/placements/matching with employers; link with Service Canada Job Bank, more hands-on support for CV development) | Interviews Focus groups |
VAC VAC |
|
| Survey | CTS/ETB participants | ||
| 6. How effectively have the programs supported Veterans to meet their personal and professional goals? | 6.1 ETB: #/% clients who completed the education/course (or are on track to completion, e.g., completed their first year of study) 6.2 CTS: #/% clients who better understand their qualifications and their post-service professional opportunities 6.3 CTS: #/% clients who prepared a CV with assistance from CTS tools/service provider 6.4 Program utilization over time (i.e., # of clients applying versus # clients accessing the programs (specifically, enrolling education/course for ETB and accessing services for CTS), by type of client |
Document review Data review Survey |
Program/corporate reports Client database CTS/ETB participants |
| 7. Were there unexpected outcomes (positive or negative) as a result of receiving services/benefits, including unintended GBA+ related impacts that should be resolved? | 7.1 Evidence of unresolved unexpected negative outcomes 7.2 Evidence of unexpected positive outcomes that led to improved program performance |
Interviews Document review Survey |
VAC Program reports, OVO reports, service delivery reports CTS/ETB participants |
| Issue: Performance/Efficiency | |||
| 8. Are the available program data for CTS and ETB adequate to measure its outcomes/impact? | 8.1 Clarity of outcome statements, causal link between outcomes; reflection of best practice 8.2 Evidence/views that the programs are collecting and reporting data to measure its relevance, effectiveness and efficiency 8.3 Adequacy, reliability, timeliness and validity of indicators/data collected by VAC, the service provider (for CTS) 8.4 Evidence/views of use of performance information in decision-making |
Interviews Document review Data review |
VAC Program/corporate reports, Performance Information Profile All databases/sources |
| 9. Have the activities of the CTS and ETB been delivered in an efficient and economical manner? | 9.1 Average time and resources required to assess/process ETB applications, over time (variation by type of application, region and GBA+) 9.2 Trends in program administrative expenditures (including contract expenditures), in program activity (caseload, average payments, forecasts, VAC/SP turnover, etc.) |
Document/data review | Program reports/HR systems |
| 9.3 % of CTS and ETB applications processed within service standards; % of formal program plans/short courses established within 4 weeks of receipt of completed education plan 9.4 % of payments made before the start of the program; % of payments made in the correct amount |
Document review | Corporate reporting | |
| 9.5 Average wait time and time in the program for CTS, over time (variation by type of service, region and GBA+) | Document review | Program reports, reports from service provider | |
| 9.6 Factors that contribute to efficient delivery, seamless delivery between the programs; evidence of inefficiencies (bottlenecks, duplication); more economical mechanisms to achieve same or superior outputs / outcomes 9.7 Suggestions, opportunities to improve efficiency |
Interviews Focus group |
VAC VAC |
|