34 Monitoring and evaluation

From Blue Gold Program Wiki

Together, Chapters 34 and 35 describe the monitoring and evaluation (M&E) process and management information system used in Blue Gold. Chapter 34 gives an overview of the processes of designing the monitoring, evaluation and learning and how the information was collected, analysed, communicated and used in Blue Gold Program and Chapter 35 explains how the M&E and other information was included in MIS to provide a consistent set of data for analysis, data presentation and reporting, and facilitated informed, timely and evidence-based project decisions. Part of the M&E processes are also included in Chapter 35 to illustrate how these fit into the MIS.

An analysis of the development outcomes from the M&E process is presented in Section B Chapters 5 to 9.

M&E Objectives[edit | edit source]

The M&E system was designed to collect and manage data to allow a regular assessment of the relevance, performance, efficiency, outcomes and impact against the stated objectives of the Blue Gold Program. The M&E activities aimed to collect, analyse, communicate and disseminate the information. Key information users were the direct beneficiaries (local communities), government and other collaborating institutions, technical assistance (TA) team, donors and similar projects.

The Program Document of August 2012 states (Section 6.4.7 p74): “Monitoring of progress towards the goals will be a task of the TA-consultant, at the start of the Program a baseline survey will be conducted per polder as to define clearly the status then of the households and of the physical and socio-economic environment. The monitoring will be done at an annual basis. The monitoring will have to be done by an independent organisation and will report directly to the members of the Steering Committee and EKN.”

The information gathered and analysed through the M&E process was used to improve strategies programs and other activities, to assist organisational learning and development, to ensure that decisions were based on sound evidence, and to provide accountability. These monitoring, evaluation and learning processes provided insights into the opportunities and challenges of implementing PWM in Blue Gold as well as outcomes and impact of the PWM interventions on livelihood, poverty and food security. Based on these findings, lessons were drawn on the impact and likely benefits of scaling-up the PWM interventions. The lessons learnt from M&E in Blue Gold will therefore be of value to projects in the coastal zone which include participatory water management and agricultural production.

Approach to the Participatory Water Management Project Monitoring and Evaluation Framework[edit | edit source]

Conceptual Basis[edit | edit source]

Soon after the start of Blue Gold, four logical frameworks (or ‘logframes’) were in existence – the GoN/EKN version of August 2012 from the Program Document (Annex 3), the BWDB version (in the BWDB Development Project Proforma, DPP of March 2013[Notes 1]), the DAE version (in DAE’s DPP from May 2013[Notes 2]), and the version which appeared in the TA Inception Report (Annex 11) dated November 2013.  The TA’s November 2013 version was essentially a reworked version of the GoN/EKN logframe from the Program Document, modified to take out the WASH component (which had been contracted to BRAC for Khulna and Max Foundation for Patuakhali).

The BWDB and DAE logframes were a results-based management tool intended to address targets within components which were primarily of interest to the parent organisation.

The Aide Memoire of the 2015 Annual Review Mission suggested (in Annex 11) that the Theory of Change should be revisited to better focus on integration, collaborative working relationships and developing the analytical and reflective capacity within the program. In addition, the Policy and Operations Evaluation Department (IOB) of the Ministry of Foreign Affairs of the Government of the Netherlands had provided inputs in March 2014 and March 2015, which inter alia resulted in a draft Theory of Change, and they also recommended an examination of the underlying assumptions of how Blue Gold interventions contribute to the project goal. Revising the Theory of Change (ToC) offered the opportunity to be more explicit about how the inter-relation between the different program activities could be strengthened to achieve the project goal. And a related benefit was expected to be the harmonisation of the three logframes (ie BWDB, DAE and EKN/TA).

As part of the process of revisiting the ToC, four workshops were organised in February and March 2016 - two in Dhaka, one in Patuakhali and one in Khulna. In Dhaka, the key workshop was entitled ‘Theory of Change for Monitoring & Evaluation’ and involved representatives from BWDB, DAE, DoF, DLS and EKN. A fuller description of the process, the agenda for the workshops and a list of participants is provided in Working Paper 5 of May 2016 – which also provides the summary results chain and five logical pathways to change.

The main aim of the ToC was to interlink the various levels of the program and which moved from input and activity-based indicators towards results/outcomes-based indicators and to provide the basis for program planning, monitoring and reflection.  By improving both the development pathway as well as the broader (relational) perspectives of M&E to enhance impact.

A workshop to present the outcome of the workshops and the impact of the ToC was held on 29th August 2016 chaired by BWDB’s ADG (Planning) with senior representatives from BWDB, DAE, DoF and DLS. The workshop discussed the revised set of indicators for program achievements, the benefit of a single integrated Logical Framework for the whole program, and the organisational implications for the TA team. In his introductory remarks, First Secretary EKN stated that ‘water management is not an end in itself, but a means to come to, among others, higher incomes through increased agricultural production in the broadest sense. But the link between water management and agricultural production is not automatic. In the ToC, this link is explored and provides an important step towards understanding the processes involved.’ Although the benefits of the ToC and the reorganisation of the TA team were recognised, neither BWDB nor DAE was able to authorise changes to their respective logframes – both of which had been authorised through the DPP planning process.

An updated version of the ToC development pathways for participatory water management was developed in 2019 based on the experience of WMGs with participatory water management (refer Chapter 30 "Theory of Change: the emergence of a practical approach to PWM"). Given the setup and operational levels of the program, the M&E framework was designed to combine selected elements of the traditional project type M&E which focused on assessing inputs-outputs relationship and implementation processes, with results-based M&E, or outcome monitoring.

Key elements in the Project’s M&E Framework[edit | edit source]

The M&E framework for Blue Gold comprised five elements, each of which fed into and informed the others:

  1. Definition of appropriate indicators
  2. Establishing the baseline situation
    1. Phase I Baseline in 2014
    2. Phase II Baseline in 2017
  3. Regular performance monitoring of activities levels of achievement of outputs (which led to the development of the WMG Tracker)
  4. Monitoring and evaluating outcomes of the outputs
    1. Participatory monitoring of WMOs
    2. Outcome Surveys
      1. WMG surveys 2018, 2019 and 2021
      2. Polder-level surveys using restructured DAE data
      3. Remote sensing/earth observation survey
  5. Monitoring and evaluating the impact of program
    1. Impact assessment/endline survey 2020

1. Definition of appropriate indicators of attainment of targets[edit | edit source]

Key indicators were suggested for each stage of the M&E for each of three main areas of interventions: development of infrastructure, institutional development and agricultural development. Formats for the key indicators were formulated to guide the collection of data and so that specific progress at each level was recorded. Initially, indicators were based on the original logframe.  In 2016, these were revised to reflect the five main development pathways identified in Blue Gold’s Theory of Change (ToC).

2. Establishing the baseline surveys/situation[edit | edit source]

At the start of the program interventions, information was collected on the pre-implementation levels of each of the key indicators identified for each of the specific objectives of the project. This baseline provided a quantitative data set that guided project implementation and evaluation. It provided a benchmark for measuring a wide range of outcomes and impacts over the life of the project. In addition to assessing the prevailing socio-economic situation with a special emphasis on agricultural, fisheries and livestock production, it served as both a valuable information source for the program as well as a method for tracking the progress and outcomes of the Blue Gold Program. This served as the baseline data to be inbuilt as a benchmark against which progress in putting in place the desired outcomes and levels of achievements of the objectives over time were measured during the implementation of the projects by regularly assessing the levels of achievement of some of the important indicators.

Blue Gold implemented the interventions in 22 polders in two rounds[Notes 3]. The majority of the first round of polders aimed to have relatively a low level of investment (termed ‘fine-tuning’ in the BWDB DPP) – so nine IPSWAM polders were included. The second round covered a wider spread of polders, the selection for which was finalised in 2015. Baseline studies were conducted in two phases using sample surveys. The Phase I Baseline survey was conducted in 2014 covering 9 polders and 1,400 households. And the Phase II Baseline survey was conducted in 2017 covering 7 polders as representatives of the balance of the 13 polders with interviews of 3,651 households. In the first phase 9 polders got the interventions while rest of the polder were from 2nd and 3rd phasing out polders. Both databases and reports are available in the MIS for the use of the stakeholders.

Table 34.1: Sample design of the baseline phase-I survey
Zone Polder Village Total HHs Sample villages Sample HHs
Khulna 22 12 2,768 25 600
29 47 13,395
30 31 6,511
Total Khulna 90 22674
Patuakhali 43/1A 11 4,542 45 900
43/2A 14 8,133
43/2B 25 8,575
43/2D 21 9,593
43/2E 4 2,745
43/2F 12 6,457
Total Patuakhali 87 40045
Total 9 polders 177 62,719 70 1400
Table 34.2: Sample design of the baseline phase-II survey
Zone Polder Mouza Total HHs sample HHs
Khulna 25 50 18,816 755
31 Part 14 4,196 169
28/1 14 6,056 242
34/2 part 23 11,227 448
Patuakhali 55/2A 31 13,966 558
47/4 12 11,853 474
Satkhira 2 & 2 Ext. 50 25,077 1,005
Total 7 polders 194 91,191 3651

The reports on these studies include:

3. Regular performance monitoring of planned activities and expected outputs[edit | edit source]

In Blue Gold, outputs were considered as the immediate products and used to monitor the progressive achievement at WMG-level to check whether the project was meeting quality, quantity and timeliness targets. To collect and store output information relating to every water management group (WMG), a ‘WMG Tracker’ was developed (refer to Chapter 35 “Establishing a WMG Tracker”).

4. Monitoring and evaluating outcomes/results[edit | edit source]

A main purpose of M&E was to measure and assess performance to manage the development results - outputs and outcomes. M&E at this level, therefore, helped to improve performance and achieve results. It was one thing to implement activities and report about the many quantitative achievements, and another for these achievements to cause the necessary changes or livelihood improvements within the coastal communities. Outcomes of BGP were measured through the organisational development of WMOs as well as increases in agricultural production and profitability.

Participatory outcomes monitoring for WMOs[edit | edit source]

Refer to Chapter 35 “Self-assessment of WMG performance

Outcome surveys for monitoring and evaluation[edit | edit source]

Here we examine outcomes resulting using three different instruments: WMG outcome surveys, DAE agricultural data and earth observation data.

WMG outcome surveys 2018, 2019, and 2021[edit | edit source]

To evaluate the outcomes of BGP interventions through the changes in production and profitability resulting from changes in cropping patterns and improved farming at WMG level, BGP conducted series of WMG surveys in 2018 and 2019. In 2021, BGP team implemented another WMG survey, that made available data on the changes in production and profitability of main crops over the three seasons 2017/18, 2018/19 and 2020/21. An independent consultancy firm was responsible for data collection, analysis and reporting of these surveys.

Before carrying out the first full WMG survey, a pilot study was conducted in P2 in Satkhira and P43/2B in Patuakhali by the TA team and an independent economic analyst appointed by EKN. Based on the findings of the study of the two pilot polders, the study team made some estimates of changes in profitability and income, along with suggestions for further data gathering and analysis to fill gaps in data from these two polders and for assessment of benefits in other polders of BGP. Along with this pilot study, BGP conducted a full phase study to understand the economic changes in all phase I &II polders (P22, P26, P29, P30, P43/1A, P43/2D, P43/2E and P43/2F, P31 Part, P43/2A, P43/2B, P55/2A, P55/2C and P2 & 2Ext.) except polder 2 & 2 Ext. and P43/2B as we had enough information of these two polders from the pilot study. The report on the 2018 WMG Survey was submitted in October 2018.

The WMG survey was repeated in 2019 with 509 WMGs in 22 polders. These surveys compared the pre-project with the present situation of the water management system, along with changes in land use and yields for the main crops, land tenure arrangements, employment and income generation in the three main seasons (rabi, kharif-1 and kharif-2). Based on this data Technical Reports TR 25 and TR 26 were prepared and shared with all stakeholders.

A final WMG survey was conducted in 2021. The rationale for planning another round of WMG survey were: WMG survey 2019 was conducted when 53% of the construction works were completed. BGP conducted the impact assessment/endline survey 2020.  This survey was conducted when 72% of the construction works were completed. Data shows that the production of aman 2019 and boro 2020 were hampered significantly due to pest/disease attract and uneven weather conditions. So, with a bad production year in the endline survey was not able to cover the full outcomes and impact scenarios of BGP interventions. The 2021 WMG survey was therefore be extended to include all registered WMGs (509) of BGP and 1018 households (2 households per WMG) covering changes in household level in a normal production year compared with the start of BGP for a limited number of key indicators. As a result, along with earlier WMG surveys and end-line survey 2020, WMG survey 2021 will gave a complete outcomes and impact scenario of BGP interventions.

In the WMG surveys, the outcome indicators were carefully identified to enable the assessment of perceptions and responses of the target groups to the program activities and outputs. These surveys helped the implementers to understand the level of acceptance (adaptation) of the expected outputs among the communities of coastal areas. Outcome indicators therefore measured changes in responses of coastal communities particularly concerning the agricultural production and profitability of the specific interventions of BGP. The focuses were on the changes in production, profitability, income and employment generation through adopting new technologies. A thorough analysis of this information at this level showed whether the program was in the right direction or the strategies have some problems. The questionnaires, database of the surveys and all related reports are available in the MIS of BGP.

The reports on these studies include:

Comparative agricultural performance[Notes 4][edit | edit source]

DAE, as one of the two implementing organizations, plays a vital role in increasing agricultural production and profitability through educating farmers on technologies for improved crop production and by utilizing the benefit of improved water management and organizational strength of the WMOs. DAE assembles agricultural data for administrative units, starting with ‘Block level’ (on average there are three Blocks per Union) where DAE Sub-Assistant Agricultural Officers (SAAOs) are responsible for data collection. The data is then aggregated at Upazila and District levels and reported to DAE’s Director Field Services. This means compiling data from all Blocks within a specific polder, on an areal pro rata basis where the Block is divided between on one or more polders. BGP restructured those preserved data as a secondary source of information and analysed the data to compare practices and land utilization pattern, changes in cropping intensity and yield over the period 2012/13 to 2020/21.

Table 34.4: DAE data analysis of BGP polders
Analysis in 2021 (TR 28) Upazila
Analysis in 2017 (TR 22)
Polder 2012/13 2013/14 2014/15 2015/16 2016/17 2017/18 2018/19 2019/20 2020/21
P22 Y Y Y Y Y Y Y Y Y Paikgacha
P26 Y Y Y Y Y Y Y Y Y Dumuria
P29 Y Y Y Y Y Y Y Y Y Dumuria, Batiaghata
P30 Y Y Y Y Y Y Y Y Y Batiaghata
P31-part Y Y Y Y Y Y Y Y Y Batiaghata
P2 and P2 Ext. Y Y Y Y Y Y Y Y Y Satkhira sadar & Assasuni
P43/2A Y Y Y Y Y Y Y Y Y Patuakhali Sadar (PS)
P43/2D Y Y Y Y Y Y Y Y Y PS
P43/2E Y Y Y Y Y Y Y Y Y PS
43/2B Y Y Y Y Y Y Y Y Y Galachipa, PS & Amtali
43/1A Y Y Y Y Y Y Y Y Y Amtali
P43/2F Y Y Y Y Y Y Y Y Y Amtali

In this analysis 12 polders were taken as representative of 22 polders of BGP. The selected polders were polder 2 & 2 ext. from Satkhira, polder 22, 26, 29, 30, 31-part from Khulna, and polder 43/1A, 43/2A, 43/2B, 43/2D, 43/2E, 43/2F from Patuakhali. The study covered 53 blocks under 8 Upazilas (see the table).  These exercises have been done in two stages. In the first stage in 2017, BGP analyzed the data from 2012-13 to 2016-17 and prepared a report (TR 22). In the second stage in 2021, BGP updated the analysis that covered the whole project period 2012-13 to 2020-21 and prepared a report (TR 28). In the second stage an independent consultant was responsible to conduct the survey.  The questionnaire, database of the surveys and all related reports are available in the MIS of BGP.

The reports on these studies include:

Remote Sensing/Earth observation survey[edit | edit source]

To evaluate the outcomes of BGP interventions through the changes in the agricultural productivity with cropping intensity and changes in the cultivated area as main indicators, BGP took another survey by making use of Earth Observation (EO) methods (combining Landsat and Sentinel-I). An efficient organisation on EO, Satelligence was contracted for doing this work. The objectives of this survey were 1) to create a baseline for agricultural productivity, 2) show changes in agricultural productivity - which could be related to BGP interventions, and 3) make agricultural productivity information accessible and understandable for the BGP technical team to use for further (intervention) planning. Performance of the polders was estimated for the start of BGP (2011-2015) and two years during the project (2017 and 2018). A report was prepared based on this remote sensing data and shared with all the stakeholders of the BGP.

To update this work BGP contracted another reputed organisation on EO, Terrasphere. They provided analysis and map of agricultural productivity to make a comparison between 2011 and 2020/21 for 22 polders of BGP. A report was prepared based on the analysis of the updated remote sensing data. The database of the surveys and all related reports are available in the MIS of BGP.

The reports on these studies include:

5. Monitoring and evaluating the impact of Blue Gold program[edit | edit source]

As the ultimate goal of the entire M&E process, the focus in impact assessment was to analyse the effects of the outcomes measured in relationship with the overall goals or development objectives. Impact M&E involves the application of information and data generated from the different stage of the outcome monitoring surveys to conduct an end of the project evaluation. With this data, an impact assessment/end-line survey was conducted in 2020. This data collected and analysed in comparison with the baseline information about actual effects of implementation of PWM of BGP on poverty and food security and a sustainable socio-economic development of the communities of Southwest Coastal Zone of Bangladesh.

Impact assessment/Endline survey 2020[edit | edit source]

The impact assessment/endline survey was designed to be follow-up to baseline surveys carried out in 2014 and 2017 to assess and measure 1) changes in agricultural and other livelihoods, income, food security, living standards for households benefitting from BGP; 2) Extent to which BGP has addressed major constraints faced by farmers and achieved its objectives; 3) Identify major problems that remain for farmers, including those that may stem from climate change; 4) Gather data on participation in community institutions and in water management, and assess possible factors that may motivate households to join WMGs. The investigation also highlighted the issue of gher as significant areas in the Khulna and Satkhira zones are occupied by ghers and   the effect of recent events (cyclone Amphon and COVID-19) on livelihoods and living standards.  Another organization and independent consultant were responsible for data collection, analysis and reporting of this survey.

The impact assessment survey covered 9 polders from the 17 polders that were surveyed in of baseline-1 &2. The impact survey used a panel sample[Notes 5], interviewing the same households as those in the baseline survey.  The table below shows the required sample. In total the impact survey covered 92 villages and mouzas from nine polders, with a target number of 4,111 sample households and a required minimum of 3,719 sample households.

Table 34.3: Sample design of the impact assessment survey
Polder District Sample HHs Interviewed HHs Analysed HHs
target minimum
25 Khulna 755 680 754 743
31P Khulna 169 169 166 162
28/1 Khulna 242 218 239 237
34/2P Khulna 448 403 433 431
2 & 2 ext. Satkhira 1005 905 1018 982
55/2A Patuakhali 558 502 534 529
47/4 Patuakhali 474 427 456 452
43/2B Patuakhali 240 216 234 230
43/2D Patuakhali 220 200 204 203
total 4111 3719 4038 3969

20 qualitative interviews were conducted in 9 polders (2 interviews from each polder except the polder 2 and 2 ext.) for in-depth understanding of the quantitative findings. Polder 2 and 2 ext. is the biggest polder in term of area and numbers of households. So, 4 interviews were conducted in this polder.   Data was analysed by polder and zone (Khulna, Satkhira and Patuakhali), with further analysis by key parameters such as farm size and poverty/income levels. The questionnaire, database of the survey and the report is available in the MIS of BGP. The report on this study was issued as TR 27 Impact of the Blue Gold Program in May 2021.

Independence of M&E Reporting[edit | edit source]

One of the recommendations of the 2016 Annual Review Mission was that the TA team should pay greater attention to the collection and analysis of data to provide: evidence of economic changes for use by Annual Review Missions; increased knowledge and understanding of beneficiaries’ responses to project interventions and adaptations; and quality-assured data for the Policy and Operations Evaluation Department (IOB) with material to carry out a post-project review of Blue Gold.

To ensure impartiality and objectivity, the economic analyst is contracted by, and reports directly to, EKN Dhaka. The independence of the economic analyst from the TA team has been an important modality in achieving the acceptance of the analysis and reporting by the implementing agencies BWDB and DAE.

Another degree of independence has been achieved by using external organisations to carry out the field surveys. The use of polder-based TA staff (now termed Community Development Facilitators) as enumerators was identified as a potential conflict of interest[Notes 6] and resulted in a decision to contract external organisations to carry out field data collection.

The international economic analyst (Edward Mallorie) carried out fieldwork in Blue Gold polders with the national TA M&E Expert (Dr Sharmin Afroz) in May 2018, through which an effective and complementary working relationship has developed. This complementarity creates a powerful combination of high level analytical and reporting ability, detailed local socio-economic knowledge and interpretation of events which is well placed to identify, portray and explain trends as polder communities respond to the complex mix of economic, weather events, market forces, and personal circumstance.

Notes[edit | edit source]

  1. Updates were made to BWDB’s logframe in BWDB’s revised DPPs dated June 2018 and September 2020
  2. DAE’s logframe was updated in the revised DPP approved in October 2018
  3. The first polders selected for Blue Gold interventions included nine IPSWAM polders: 22, 29, 30, 43/1A, 43/2A. 43/2B, 43/2D, 43/2E, 43/2F;  plus three with no previous history: 26, 31-part and 2. The selection of the second round of ten polders was finalised in 2015, comprising 25, 27/1, 27/2, 28/1, 28/2, 34/2 part, 47/3, 47/4 and 55/2A and 55/2C.
  4. Using DAE production data collected at ‘Block’ level, the information was reorganized to provide data for the 22 Blue Gold polders over the period 2012-13 to 2020-21
  5. A panel sample also allows analysis to trace the impact for a group of households defined by a baseline parameter.  For example, have the poorest households benefited?   
  6. Because the role of CDFs is to empower the polder communities, their relationship is often complex, multi-faceted with a great deal of co-dependence. For this reason, the impartiality of the CDFs when collecting data from their communities could be compromised.

See more[edit | edit source]

Previous chapter:
Chapter 33: Horizontal Learning
Blue Gold Lessons Learnt Wiki
Section G: Project Management
Next chapter:
Chapter 35: Management Information System

Section G: Project Management
Chapter 28: Project Management Arrangements Chapter 29: Technical Assistance: Context, Scope, Contractual Arrangements and External Service Contracts Chapter 30: Evolution of TA Organisational Arrangements organisation
  1. Introduction
  2. Implementing Modalities
  3. Development Project Proformas (DPPs)
  4. Project Meetings
  5. Memoranda of Understanding (MoUs)
  6. Review Missions
  7. Annual Work Plans
  8. Polder Development Plans
  9. Progress Reports
  1. Context and Scope
  2. Contractual Arrangements
  3. TA Service Contracts
  1. Scope
  2. Scope of Technical Assistance in the Program Document
  3. Early Arrangements for the TA Organisation
  4. Evolution of TA Organisation
  5. Theory of Change: the emergence of practical approach to PWM
Chapter 31: Capacity Building Chapter 32: Agricultural Extension Methods and Communication Chapter 33: Horizontal learning
  1. Capacity Building Programs
  2. International Exposure
  3. Refocused Training
  4. Refocused TA FFS
  5. Vocational Education Training
  1. Communication aimed at beneficiaries
  2. Communication aimed at organisations
  1. Horizontal Learning – the approach in BGP
  2. Horizontal Learning – An assessment of BGP’s experience
Chapter 34: Monitoring and evaluation Chapter 35: Management Information System Chapter 36: Environmental Due Diligence
  1. M&E Objectives
  2. Approach to the Participatory Water Management Project Monitoring and Evaluation Framework
  3. Key elements in the Project’s M&E Framework
  4. Impact assessment/Endline survey 2020
  5. Independence of M&E Reporting
  1. Background
  2. Establishing a WMG Tracker
  3. Management Information System (MIS)
  4. MIS Design and Development
  5. MIS Results Reporting
  6. WMG Tracker Closure
  7. Polder Dashboard
  8. Polder "Health Checks"
  9. Participatory Monitoring
  10. Post-Project Monitoring
  11. Self-assessment of WMG performance
  1. Objectives of the EIA Study
  2. Overview of EIA arrangement and consideration
  3. The modalities for carrying out the EIAs
  4. Alternative future modality
Blue Gold Wiki
Executive summary: A Call for Action
Section A: Background and context Section B: Development Outcomes Section C: Water Infrastructure


Summary and Introduction


Section D: BGP Interventions: Participatory Water Management Section E: Agricultural Development Section F: Responsible Development: Inclusion and Sustainability




Section G: Project Management Section H: Innovation Fund Files and others