Appendix C – Evaluation Issues/Questions

Appendix C – Evaluation Issues/Questions
Evaluation Issues and Questions
Issues/Questions Indicators Collection Methods Data Sources
Relevance
1. To what extent are the outputs delivered by the Treatment Benefits Program Management Unit in line with the original intent of the unit, and linked to program objectives?
  • Objectives and outputs of the directorate are clearly defined
  • Activities and outputs of the directorate are directly linked to the purpose/objectives of the Program
  • Activities and outputs of the directorate are directly linked to the purpose/objectives of the unit
  • Program Management accountabilities and roles are clearly defined and understood
  • Management incentives align with goals/program objectives
  • Document Review
  • Interviews / Observations
  • Legislation, policies, processes and procedures
  • Program documents
  • Program Management staff and senior management
  • Briefing notes
  • Emails
  • Direction from senior management
Efficiency
2. To what extent is the Treatment Benefits Program Management Unit optimizing its use of resources and producing the required outputs?
  • Activities/information requests are effectively triaged based on risk/priority
  • The directorate staffing level and expertise are appropriate to meet the output demands
  • Limited overlap or duplication of duties exist
  • Non-directorate staff are appropriately engaged / consulted with when producing directorate outputs (e.g. Health Professionals, Policy, Field Ops, Contractor staff)
  • Use of tools and technology is maximized to help manage work appropriately
  • The key outputs produced by the directorate are appropriate in terms of quality and quantity.
  • Events/changes to program management/delivery have improved production of directorate outputs
  • Appropriate support is provided to program delivery staff
  • Ratio of inputs to outputs produced
  • Planned output production is achieved or surpassed
  • Ratio of inputs to outputs is comparable to similar federal government program management directorates
  • Comparable resource compliment/structure and processes are used by similar federal government programs
  • Document review
  • Data analysis
  • Emails
  • Interviews
  • Observation / Focus Group(s)
  • Program management documents
  • Policies, processes and procedures
  • Process maps
  • Committee/meeting minutes (BRC and TBIN)
  • Program directives / SDAT messages
  • Previous evaluations/Management Response and Action Plans (MRAPs)
  • Organizational Chart
  • Finance salary and O&M
  • SDAT, Issue Resolution Officer (IRO), Ministerial and OVO requests received
  • Program management, other HO staff, field operations and contractor staff input
  • Review of available comparison program management structures (e.g. DND, INAC/Health Canada)
3. Are there any opportunities to improve efficiencies in the managementFootnote 33 of the Treatment Benefits Program?
  • Performance measurement information is collected, reported, monitored and used to support program decision making
  • Key risks impacting program delivery are documented and disseminated to Treatment Benefits Program Management Unit
  • Appropriate consultation with other areas of the department occurs
  • Directorate uses lessons learned as opportunities for future program improvements
  • Have efficiencies been realized since the previous evaluations MRAPs
  • Document review
  • Data analysis
  • Emails
  • Interviews
  • Regular performance monitoring reports
  • Divisional workload tracking
  • Policies, processes and procedures
  • Committee/meeting minutes (BRC and TBIN)
  • Program directives / SDAT messages
  • HO, field operations and contractor staff
  • Review of available comparison program management structures (e.g. DND, INAC)
  • Previous Evaluations/MRAPs