1、整理課件120112011年年1111月月整理課件2Monitoring and Evaluating ODAThis presentation will cover lWhy monitor? lWhy evaluate? lMonitoring in the World Bank, other multi-lateral development banks, bilateral donorslEvaluation in the World Bank, other multi-laterals, bilateral donors整理課件3What is monitoring? Definit

2、ion of monitoring: a continuous process that uses systematic collection of data about progress in an on-going development project, for use by managers and other stake-holders整理課件4Why monitor?To improve implementation of on-going projects; must provide practical informationTo assess whether developme

3、nt objectives likely to be metTo identify systematic issues and risks across projects, by country, by sector, and overallTo provide input for ex post evaluation整理課件5What is Evaluation?Definition of (ex post) evaluation: assessment of a completed project, program, or policy, its design, implementatio

4、n, and results.整理課件6Why evaluate and who cares?Why? Two reasons:To learn from experienceTo provide accountabilityWho uses (or should use) evaluations: Boards of Directors (MDBs) and Ministers and Directors General for oversight; ODA managers for policies and strategies; ODA staff for program and pro

5、ject designs, ways of doing business; Client countries, external partners, and the public for policies, aligning programs, research整理課件7M & E in contextlWorld Bank commits / disburses $20-23 billion/year; lApproves 280 loans and credits/year; active loan portfolio : 1,400 in 120 countrieslBank a

6、lso provides grants and other non-lending services (diagnostic work)lBank has total administrative budget of $2 billionMost other MDBs commit $1 6 billion/year Bilateral ODA : $200 million (Greece, Luxembourg, New Zealand, Portugal) - $25 billion/year (USA)Bilateral agencies have administrative budg

7、ets of $11 million (Luxembourg) to $700 million (Japan) 整理課件8Monitoring in the World BanklEvery on-going lending operation is monitored by visits from Bank staff about twice a year; lImplementation status and results (ISR) report issued twice each year:lProgress on implementationlProgress in meeting

8、 development objectiveslProblem projects identifiedlPortfolio at risk reports separate methodology updated monthlylOther quality control activities include quality at entry reviews; quality of supervision整理課件9Evaluation: the topicslSelf evaluationlIndependent evaluation: why it matterslTools: projec

9、ts; sectors; countries; etc.lCoverage: how many evaluationslResources/budgetlMain challenge for evaluationlDisclosurelOther Evaluation Issues整理課件10Self-Evaluation in the BankSelf evaluation in the Bank:lEvery project has an implementation completion report, done by staff of responsible unit (about 2

10、80/year)lEvery new country assistance strategy now has completion report covering previous country assistance strategy, done by staff of responsible unit (about 20/year)整理課件11Self-evaluation in other donorslMulti-lateral institutions: almost all MDBs have requirement for self-evaluation, although in

11、 some MDBs (Asian DB, EBRD, EIB), the evaluation department is involvedlBilateral donors: some require for projects over given value, others dont; SDC (Swiss): external evaluators do self-evaluation; GTZ (German) has innovative eVAL (electronic) system.整理課件12Independent evaluationlWhat is independen

12、ce? The extent to which the evaluation unit:Reports directly to a Board or other governing unit; Has staff, budget, and functions independent from management; not involved with management of ODABehaves independently by issuing objective, impartial, and balanced reports整理課件13Independent evaluationIn

13、the World Bank, independent evaluation, since 1973, has looked like this:Board of DirectorsInstitutional IntegrityPresidentPaul WolfowitzIndependent Evaluation GroupInternal AuditingSVP & Chief EconMD OperationsCFOSVP & Gen CounselInspection PanelMD Operations整理課件14Independent evaluationIn b

14、ilateral donors, independent evaluation has evolved to look like this:Ministry of Foreign Affairs Evaluation UnitDirector General Aid agencyOperational unitOperational unitOperational unit整理課件15Independent evaluationlWhy independence matters:lAllows for objective, unbiased evaluationslEvaluation fin

15、dings are not tailored to managements wisheslIndependence lends credibility to findingslIndependence difficult to achieve and maintain整理課件16Evaluation Tools: World BanklProject evaluationslImplementation Completion Report (ICR) reviewslProject performance assessment reviews (PPARs)lCountry evaluatio

16、nslCAS Completion Report (CASCR) ReviewslCountry assistance evaluations (CAEs)lSector, thematic, global evaluationslProcess evaluationslCorporate evaluations (ARDE, AROE)整理課件17Evaluation Tools: interactionsSynergies among different evaluation tools can provide valuable input Global (Global programs)

17、Evaluation tools in the World Bank Corporate (ARDE)Sector(finance, education) Thematic (gender, decentralization) Process (PRSP)Country Assistance Evaluation CASCR Review CASCR ReviewICR Review (validation of self-evaluation)Project Performance Assessment ReviewCountry Assistance EvaluationICR Revie

18、w (validation of self-evaluation)Project Performance Assessment ReviewProject levelCountry levelSector, thematic, policy levelGlobal and corporate level整理課件18Evaluation Coverage: World Bank整理課件19Evaluation coveragelIn most MDBs, full range of evaluations: project, country, sector, thematic evaluatio

19、nslIn most bilaterals, few project evaluations; some country level; most focus on themes or programslMost bilaterals produces 3-10 evaluations/yearlDFID produces 20-25 reports/yr; lPotential weakness of country and thematic evaluations without underpinnings of project evaluations to use as input整理課件

20、20Resources for evaluation: MDBslResources vary, but as percent of total administrative budget, fairly similar: 整理課件21Resources for evaluation: bilateralslBilateral donor agencies are more likely to hire external consultants for virtually all evaluations (UK, Norway, Denmark, Switzerland, Australia)

21、 lBased on numbers from a few bilaterals (DFID, NORAD, SDC), evaluation budget is 2% of total administrative budget; figures of 0.5 2% have been cited by others.lSmaller donors (Greece, New Zealand) are still setting up their evaluation units整理課件22Main challenge for evaluationlHow to make evaluation

22、s usefullMost managers, staff focused on future, not on pastlEvaluation must / should be:lTimelylOperationally relevant: avoid generalizations, platitudeslEasily accessible: avoid long reportslConstructive: good practice should be rewardedlWidely disseminated and disclosed整理課件23Disclosure of evaluat

23、ion reportslWorld Bank: public disclosure of all:lPPARslCAEslSector, thematic, global evaluations lProcess evaluationslCorporate evaluations lEBRD, ADB: selective public disclosure: evaluations involve private sectorlBilaterals: majority disclose, but varies: UK, Norway full disclosure; Germany sele

24、ctive disclosure整理課件24DisclosurelFor World Bank: reports and information on evaluation tools, approaches, on website:l and scroll down to right-had side of home page, or:l/ieglFor other donors: check their websites整理課件25Other issues in evaluationlEvaluation

25、approachlUse of performance indicatorslSources of informationlRatingslProcesseslPartnershipslOther challenges整理課件26Evaluation ApproachWithin context of results-based management, Bank uses objectives-based approach, focus on outcomes and impactBank loans; grants; diagnostic work; dialogue Physical ou

26、tputs; privatization; new laws; agency created; training completed; sub-loans disbursed.More efficient operations; better fiscal management; employment creation; better access to services; InputsOutputsExogenous factorsExogenous factorsOutcomeImpactLonger-term objective, like poverty reduction, bett

27、er educational, health outcomes整理課件27Evaluation ApproachBank assesses:Outcome (relevance, efficacy, efficiency)lUse of performance indicators for outputs, outcomes, impactlRisk to DevelopmentlBank and Borrower Performance整理課件28Sources of information for evaluationBaseline data in project appraisal d

28、ocuments (sometimes)Monitoring reports should have information on performance indicatorsData bases for project area; sector; country at beginning of project, end of projectOther country-level indeces (IMF financial statistics, Doing Business surveys; World Bank Indicators of corruption surveys; UNDP

29、 human development indicators)SurveysInterviews, focus groups整理課件29Use of RatingslWorld Bank uses ratings on project outcomes and outcomes of country assistance. No ratings on sector, thematic, other evaluations lOther MDBs use ratings for project outcomes; not for country assistancelBilaterals: som

30、e use ratings, only for projects; some do not use any ratings整理課件30Evaluation Approach: ratingslProject Ratings in World Bank:lOutcomes: 6 point scale (HS - HU)lRisks to Development: 4 point scale (H - N)lBank and Borrower performance: 4 point scale (HS to HU)lPros and cons of ratinglPro: provides data and accountabilitylCon: takes focus away from lesson learning and richness of evaluation整理課件31ProcessesIndependence does not mean isolationMDBs and bilateral donors incorporate comments from operational managers and staff and from stakeholders in cl