Annex 2: Evaluation Matrix

Annex 2: Evaluation Matrix
Questions Sub-questions Indicators Data Collection Method Source of DataFootnote 21
Issue: Relevance
Ongoing need for the OVO
1. What are the needs and expectations of Veterans/clients for an Ombudsman? 1.1 What unique needs are met by the OVO within the context of its current mandate? 1.a Mandate stated needs addressed by OVO
1.b Types of complaints received
1.c Stakeholders views of OVO mandate
Key Informant Interviews (KII)
Document Review (DR)Data base review (DBR)
Case studies (SC)
KII: VAC and OVO management & staff; members of Minister Advisory Group (MAG); members of Veterans Ombudsman Advisory Council (VOAC)
DR: OVO Mandate; annual reports
DBR: CRM (Ombudsman File Tracking System)
CS: file review; systemic review report; OVO KII
1.2 Considering other bodies/organizations that address Veterans’/clients’ concerns/complaints, is there overlap or duplication with the OVO; and, are there any gaps where Veterans’/clients’ concerns are unmet? 1.d Degree of overlap with other bodies and organizations addressing Veterans’ concerns / complaints
1.e % of Veterans / clients complaints received by the OVO that cannot be addressed within mandate of OVO
Key Informant Interviews
Document Review
Data base review
KII: VAC and OVO staff, MAG, VOAC
DR: Order in Council (OIC) P.C. 2007-530; annual reports; five year strategic and integrated business plan 2017; Service Level Agreements (SLAs) with VAC, 2016-2019
DBR: CRM (Ombudsman File Tracking System)
Relevance of the OVO over time
2. Is the OVO’s mandate still relevant considering the current context and Veterans’/clients’ needs? 2.1 Has the OVO adhered to its mandate (why or why not)? 2.a Degree of change since 2007
2b. Evidence of shift in mandate
Key Informant interviews
Document Review
KII: OVO, VAC
DR: Order in Council (OIC) P.C. 2007-530; Report of the Standing Committee 2007; 2007 Planning documentation; OVO five year strategic and integrated business plan 2017
2.2 Have the needs of Veterans’ changed over time? 2c. Systemic Veterans’ issues over time Key Informant interviews
Document Review
KII: OVO, VOAC
DR: Systemic reports of the OVO over time
3. Is there an appropriate level of independence for the OVO? 3.1 Is there an appropriate level of independence for the OVO? 3.a Adequacy of the OVO level of independence to meet Veterans needs Key Informant interviews
Document Review
Comparative Assessment
Case Studies
KII: OVO, VAC, MAG, VOAC
DR: OVO Mandate
CA: Procurement / Corrections Ombudsman Offices; By-Monthly Ombudsmen’s meeting
CS: file review; systemic review report; OVO KII
3.2 How does the OVO’s mandate and powers compare to other ombudsman offices? 3.b Adequacy of OVO mandate and powers compared to others Key Informant interviews
Document Review
Comparative Assessment
Case Studies
KII: OVO, VAC
DR: OVO Mandate
CA: Procurement / Corrections Ombudsman Offices; By-Monthly Ombudsmen’s meeting
CS: file review; systemic review report; OVO KII
Issue: Effectiveness
Achievement of outcomes
4. Are Veterans complaints being resolved, and what are the key barriers in the OVO’s ability to resolve complaints? 4.1 Are Veterans/clients aware of and accessing the OVO? 4.a % of Veterans and other clients surveyed who indicate they are aware there of the Office of the Veterans Ombudsman and the services and benefits it provides
4.b % of official stakeholders (e.g., veterans’ groups) engaged annually
4.c # of Town Halls conducted annually across Canada
Key Informant Interviews
Document Review
Case studies
KII: OVO, VAC, MAG, VOAC
DR: VAC National Surveys/POR, OVO / VAC Annual Reports, Engagement reports/dashboard
CS: File review; systemic review report; report card; KII: OVO
4.2 Are Veterans treated respectfully and fairly by the OVO, and are their concerns addressed by VAC? 4.d % of OVO clients that indicate their complaints were addressed respectfully
4.e % of complaints that were assessed as unfair by the Office of the Veterans Ombudsman that were resolved by Veterans Affairs Canada
Key Informant interviews
Document Review
Database Review
KII: OVO, VAC, MAG, VOAC
DR: Client feedback questionnaire
DBR: CRM (Ombudsman File Tracking System)
4.3 Do Veterans have timely responses to their complaints? 4.f % of complaints closed within 60 working days
4.g % of OVO clients who indicate they received a response to their complaint when indicated
Document Review
Database Review
DR: Client feedback questionnaire
DBR: CRM (Ombudsman File Tracking System)
4.4 Do systemic investigations address issues of concern to Veterans/clients? 4.h Effectiveness of mechanisms to identify research topics
4.i Degree of alignment of research with Veteran’s concerns over time
4.j Are procedures for conducting and reporting systemic investigations effective?
Key Informant interviews
Document Review
Database Review
Case study
KII: OVO, VAC, MAG, VOAC
DR: OVO research reports; Report Cards
DBR: CRM (Ombudsman File Tracking System)
CS: systemic review report; report card; OVO KII
4.5 What internal or external barriers / factors prevent the OVO from achieving its outcomes? 4.k Degree to which barriers prevent achievement of outcomes Key Informant interviews
Document Review
KII: OVO, VAC, MAG, VOAC
DR: OVO / VAC annual reports; OVO research reports
5. What impact/change is resulting from systemic report recommendations? 5.1 What impact is resulting from the advice provided to the Minister of Veterans Affairs and Parliamentary Committees? 5.a % of OVO recommendations on systemic issues implemented by the Veterans Affairs Portfolio
5.b % of OVO recommendations on systemic issues accepted by the Veterans Affairs Portfolio
Key Informant Interviews
Document Review
Case studies (CS)
KII: OVO, VAC, MAG, VOAC
DR: OVO / VAC annual reports; Report Cards, 2017 & 2018
CS: File review, KII OVO
Unexpected outcomes and external factors
6. What additional or unexpected outcomes (positive or negative) have resulted due to the work of the OVO?   6.a Evidence of additional or unexpected outcomes Key Informant Interviews
Document Review
KII: OVO, VAC, MAG, VOAC
DR: VAC / OVO annual reports
Issue: Efficiency
Performance Measurement
7. Is the OVO Performance Strategy adequate to measure its impact? 7.1 Are expected outcomes well-articulated to indicate the impact of the OVO 7.a Clarity of outcome statements
7.b Strength of causal link between outcomes and ultimate outcome
Document Review DR: Evaluability Assessment, Performance Information Profile (PIP)
7.2 Has a performance measurement strategy been developed and implemented? 7.c Evidence of a measurement strategy
7.d Evidence that the OVO is collecting data to measure its relevance, effectiveness and efficiency
Key Informant Interviews
Document Review
KII: OVO, VAC (AED)
DR: Performance Information Profile (PIP); OVO/ VAC annual reports; CRM (Ombudsman File Tracking System)
7.3 Is the OVO performance data reliable, timely, and valid? 7.e Assessment of performance data Key Informant Interviews
Document Review
KII: OVO, VAC
DR: Evaluability Assessment
7.4 Is performance information reported and used in decision-making? 7.f Evidence of reporting using OVO performance data
7.g Evidence of use in decision-making
Key Informant Interviews
Document Review
KII: OVO, VAC
DR: OVO / VAC annual reports
Governance and Management
8. Is the governance structure in the OVO appropriate and efficient for achieving expected outcomes? 8.1 How does the OVO’s structure facilitate or impede its success (e.g., powers of the OVO, funding model, reporting relationships, succession, etc.)? 8.a Degree to which governance and management structures and processes allow the OVO to achieve expected outcomes Key Informant Interviews
Document Review
KII: OVO, VAC
DR: Audit of VAC’s Governance, December 2017
8.2 Has the management approach (direction and processes in the OVO) been strategic and adapted appropriately over time? 8.b Stakeholder perceptions on degree to which changes in direction and processes in the OVO improved its efficiency Key Informant Interviews KII: OVO, VAC (current and former)
9. Considering other ombudsman offices, are there alternative structures or delivery options (e.g., powers, tools) that would enhance the effectiveness and efficiency of the OVO?   9.a Strengths and weaknesses of alternative structures or delivery options to achieve OVO’s outcomes and improve efficiency Key Informant Interviews
Document Review
Comparative Assessment
KII: OVO, VAC
DR: OVO Organizational Chart, February 2019
CA: Procurement and Correctional Ombudsman Offices; By-monthly Ombudsmen meeting
Efficiency of delivery
10. Have the activities of the OVO been delivered in an efficient and economical manner? 10.1 Does the OVO have the right tools and processes for efficient delivery? 10.a Average time and resources required to resolve complaints
10.b Veterans / clients level of satisfaction with complaint remedy
10.c % of complaints in the backlog
10.d % of systemic investigations submitted to the Minister within 90 days for micro investigations and within 1 year for systemic investigations
10.e % of engagement plan activities completed annually
Key Informant Interviews
Document Review
Database review
Case studies
KII: OVO
DR: Front line service efficiency study, client feedback surveys, Report Cards, review of engagement dashboard
DBR: CRM (Ombudsman File Tracking System)
10.2 Do staff have the required capacities, and are they being retained, to promote efficient delivery (e.g., communication and frontline staff)? 10.f Views on staff capacities and retention
10.g Evidence that staff training is offered and accessed
10.h Level of staff turnover
Key Informant Interviews
Document Review
Case studies
KII: OVO
DR: OVO HR and Financial Reports