ECG Good Practice Standards
Annex II.1: Template for Assessing the Independence of Evaluation Organizations
Published by: Independent Evaluation Group (IEG), World Bank Group
|
Criterion |
Aspects |
Indicators |
|---|---|---|
|
I. Organizational independence |
The structure and role of evaluation unit
|
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization's operational, policy, and strategy departments and related decision-making |
|
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board |
Whether there is a direct reporting relationship between the unit, and a) the Management, and/or b) Board or c) relevant Board Committee, of the institution |
|
|
The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated |
The unit's position in the organization relative to the program, activity or entity being evaluated |
|
|
The unit reports regularly to the larger organization's audit committee or other oversight body |
Reporting relationship and frequency of reporting to the oversight body |
|
|
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions |
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities |
|
|
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit |
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced |
|
|
Unit has access to all needed information and information sources |
Extent to which the evaluation unit has access to the organization's a) staff, records, and project sites; b) co-financiers and other partners, clients; and c) programs, activities, or entities it funds or sponsors |
|
|
II. Behavioral Independence |
Ability and willingness to issue strong, high quality, and uncompromising reports |
Extent to which the evaluation unit: a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization's programs and activities; b) proposes standards for performance that are in advance of those in current use by the organization; and c) critiques the outcomes of the organization's programs, activities and entities |
|
Ability to report candidly |
Extent to which the organization's mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments |
|
|
Transparency in the reporting of evaluation findings |
Extent to which the organization's disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk). Who determines evaluation unit's disclosure policy and procedures: Board, relevant committee, or management.
|
|
|
Self-selection of items for work program |
Procedures for selection of work program items are chosen, through systematic or purposive means, by the evaluation organization; consultation on work program with Management and Board |
|
|
Protection of administrative budget, and other budget sources, for evaluation function |
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions |
|
|
III. Protection from outside interference
|
Proper design and execution of an evaluation |
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management interference |
|
Evaluation study funding |
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities |
|
|
Judgments made by the evaluators |
Extent to which the evaluator's judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority |
|
|
Evaluation unit head hiring/firing, term of office, performance review and compensation |
Mandate or equivalent document specifies procedures for the a) hiring, firing, b) term of office, c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management |
|
|
Staff hiring, promotion or firing |
Extent to which the evaluation unit has control over: a) staff hiring, b) promotion, pay increases, and c) firing, within a merit system |
|
|
Continued staff employment |
Extent to which the evaluator's continued employment is based only on reasons related to job performance, competency or the need for evaluator services |
|
|
IV. Avoidance of conflicts of interest |
Official, professional, personal or financial relationships that might cause an evaluator to limit the extent of an inquiry, limit disclosure, or weaken or slant findings |
Extent to which there are policies and procedures in place to identify evaluator relationships that might interfere with the independence of the evaluation; these policies and procedures are communicated to staff through training and other means; and they are enforced |
|
Preconceived ideas, prejudices or social/political biases that could affect evaluation findings |
Extent to which policies and procedures are in place and enforced that require evaluators: a) to assess and report personal prejudices or biases that could imperil their ability to bring objectivity to the evaluation; b) and to which stakeholders are consulted as part of the evaluation process to ensure against evaluator bias |
|
|
Current or previous involvement with a program, activity or entity being evaluated at a decision-making level, or in a financial management or accounting role; or seeking employment with such a program, activity or entity while conducting the evaluation |
Extent to which rules or staffing procedures that prevent staff from evaluating programs, activities or entities for which they have or had decision-making or financial management roles, or with which they are seeking employment, are present and enforced |
|
|
Financial interest in the program, activity or entity being evaluated |
Extent to which rules or staffing procedures are in place and enforced to prevent staff from evaluating programs, activities or entities in which they have a financial interest |
|
|
Immediate or close family member is involved in or is in a position to exert direct and significant influence over the program, activity or entity being evaluated |
Extent to which rules or staffing procedures are in place and enforced to prevent staff from evaluating programs, activities or entities in which family members have influence |
Source: U.S. General Accounting Office, Government Auditing Standards, Amendment 3 (2002); OECD/DAC Working Party on Aid Evaluation, Glossary of Key Terms in Evaluation and Results Based Management (2002); OECD/DAC, Principles for Evaluation of Development Assistance (1991); INTOSAI, Code of Ethics and Auditing Standards (2001); Institute of Internal Auditors, Professional Practices Framework (2000); European Federation of Accountants, The Conceptual Approach to Protecting Auditor Independence(2001); Danish Ministry of Foreign Affairs, Evaluation Guidelines (1999); Canadian International Development Agency, CIDA Evaluation Guide (2000).