Appendix A – Evaluation Matrix

Appendix A – Evaluation Matrix
Evaluation Matrix
Issues / Questions Indicators Collection Methods Data Sources
Relevance
1. To what extent does the program continue to address a demonstrable need, now and in the future? Number of applicants, favorable and unfavorable decisions.

Key informant interviews and documentation review indicate ongoing need for program.

Forecast of population indicates future demand for the program.
Interviews/
Observations


Survey

GBA+ data analysis

Document review
VAC staff
Staff survey

Statistical and program data

Legislation, departmental priorities/mandate letter, policies, processes, OVO reports

Program documents
2. To what extent does the program align with Government of Canada priorities and with federal roles and responsibilities? Evidence of a Government of Canada commitment to Veterans in emergency situations (e.g. homelessness).

Evidence of the program commitment included in departmental plans and priorities documentation.
Interviews

Survey

Document review
VAC staff

Staff survey

Legislation, policies, processes, VAC National Survey, OVO reports, budget documents, mandate letter

Program documents
3. To what extent is the Program responsive to the needs of recipients and target audiences? Program eligibility enables Veterans who require the program to get access to it.

Program funding is sufficient to respond to the needs of Veterans

Prevalence of clients accessing the program on multiple occasions.

Requests exceeding yearly maximums (requests for exceptions)

Types of emergencies for which Veterans are requesting assistance

Demographics of clients accessing program.
Interviews

Survey

GBA+ data analysis

File review

Document review
VAC staff

Staff survey

Program recipient feedback (program survey)

Statistical and program data

Veteran files

Legislation, departmental priorities/mandate letter, policies, processes, OVO reports, financial reports

Program documents
Performance
4. Are processes currently in place to ensure the program is being administered as indicated in departmental guidance? Presence of quality management activities.

Program policy provides direction and definitions which build upon the legislation/regulations.

Alignment of business processes with program policy.

Appropriate governance structures in place to ensure accurate and consistent decision making

Alignment of application/tools/assessments with program decision-making requirements.

Adequate and appropriate training for decision makers. Alignment of payments to policy
Interviews/ feedback from decision makers

Survey

File review

Document review
VAC staff

Staff survey

Veteran files

Legislation, departmental priorities/mandate letter, policies, processes, guidelines

Program documents
5. To what extent is the program achieving its outcomes? Program Information Profile in place and performance data is being collected, reviewed and assessed. Interviews

Document review

File review
VAC Staff

Program recipient feedback

Veterans files

Statistical and program data
Immediate – Veterans payments are made in a timely manner
Percentage of payments issued within 2 business days of approval Data analysis Statistical and program data

Veteran files
Immediate – Veterans have equitable access to VAC programs and services
Percentage breakdown of program clients by sex (M/F) compared to estimated Veteran population

Percentage breakdown of program clients by sex (M/F) and age compared to estimated Veteran population

Percentage breakdown of program clients by sex (M/F) and marital status

Percentage breakdown of program clients by sex (M/F) and community (rural/urban)
GBA+ data analysis

GBA+ data analysis

GBA+ data analysis

GBA+ data analysis
Statistical and program data

Veteran files

Statistical and program data

Veteran files

Statistical and program data

Veteran files

Statistical and program data

Veteran files
Immediate – Veterans and their families utilize the VEF
Number of Veterans and families who receive a VEF payment GBA+ data analysis Statistical and program data

Program recipient feedback
Intermediate – Veterans are referred to resources that assist them to manage their finances
Percent of Veterans who received referrals to other services GBA+ data analysis

Interviews

Survey

Document review

File review
Statistical and program data

VAC staff

Program recipient feedback (program survey), staff survey

Program documents

Veteran files
Ultimate – Veterans are able to adapt, manage and cope within civilian life
Percentage of Veterans who report an easy adjustment to civilian life GBA+ data analysis

Interviews

Document review

Survey
Statistical and Program Data

VAC Staff

Program Documents

Program recipient feedback (program survey)

LASS Survey
Ultimate –Veterans and their families are financially secure
Percentage of Veterans who are satisfied with their financial situation GBA+ data analysis

Interviews

Document review

Survey
Statistical and program data

VAC staff

Program documents

Program recipient feedback (program survey)

LASS Survey
Ultimate – Veterans are satisfied with the services they receive
Percentage of clients who are satisfied with the quality of service delivery they receive from Veterans Affairs Canada GBA+ Data Analysis

Interviews

Document review

Survey
Statistical and program data

VAC Staff

Program documents

Program recipient feedback (program survey)

VAC National Survey
Efficiency/Economy
6. Are there opportunities to improve the efficiency and economy of the program? Administrative costs ratio

Processes allow for timely decision-making.

Application process is timely and accessible.

Opportunities for aligning with other departmental programs are identified and considered.

Authority is delegated appropriately
GBA+ Data Analysis

Interviews

Survey

Document Review

File Review

Survey
Statistical and Program data

VAC staff

Staff survey

Program documents

Veteran files

Program recipient feedback (program survey)
7. Are there any unintended impacts (positive or negative) that have occurred? Positive and/or negative impacts realized from the program Interviews

Survey

File Review

GBA+ Data Analysis

Survey
VAC staff

Staff survey

Veteran files

Statistical and program data

Program recipient feedback (program survey)