Qualitative perspectives on the impact evaluation of

0 downloads 0 Views 210KB Size Report
Feb 6, 2015 - At its heart is reflection on a particular impact evaluation ..... mix normative and descriptive: 'it is better to be a man than a ... The education composite is then broken down to identify on which ..... .org/INTEMPOWERMENT/Resources/41307_wps3510.pdf .... Especially chapters by Winch, Hollis, Jarvie.
This article was downloaded by: [Sarah White] On: 19 April 2015, At: 08:24 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Development Effectiveness Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/rjde20

Qualitative perspectives on the impact evaluation of girls’ empowerment in Bangladesh Sarah C. White

a

a

Department of Social and Policy Sciences, University of Bath, Bath, UK Published online: 06 Feb 2015.

Click for updates To cite this article: Sarah C. White (2015): Qualitative perspectives on the impact evaluation of girls’ empowerment in Bangladesh, Journal of Development Effectiveness, DOI: 10.1080/19439342.2015.1004609 To link to this article: http://dx.doi.org/10.1080/19439342.2015.1004609

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

Downloaded by [Sarah White] at 08:24 19 April 2015

Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions

Journal of Development Effectiveness, 2015 http://dx.doi.org/10.1080/19439342.2015.1004609

Qualitative perspectives on the impact evaluation of girls’ empowerment in Bangladesh Sarah C. White*

Downloaded by [Sarah White] at 08:24 19 April 2015

Department of Social and Policy Sciences, University of Bath, Bath, UK This paper presents qualitative analysis of an RCT evaluating a programme aiming to delay marriage and empower adolescent girls in Bangladesh. It begins by suggesting a model of three dimensions of empowerment for use in evaluation. It then describes the social context of early marriage in Bangladesh. The model of empowerment is then used to discuss the design and preliminary findings of the RCT. The paper stresses the importance of approaching research as a social process, to improve data quality and rigour of analysis. It closes by arguing for teams which balance equally researchers with quantitative and qualitative expertise. Keywords: qualitative; impact evaluation; girls; empowerment; early marriage; Bangladesh

Introduction The ‘huge increase in the number of impact evaluations of development interventions in the last ten years’ (White 2014, 18) mobilises two key promises, the ability to show whether interventions are ‘transforming lives’ and to demonstrate ‘causality’. These are not small or insignificant claims. ‘Transformation’ is the term currently in vogue to describe the goal of development. Attribution of a causal relationship between an outcome and a particular intervention is the Holy Grail sought by development agencies to justify their activities and even existence. Together, therefore, transformation and causality go to the heart of development industry aspirations. It is not surprising that techniques that promise to deliver these should have gained such popularity. As a modernist project, development is captivated by two romances: the romance with science and the romance with the new. Its romance with the new means it is continually reinventing itself, creating new terms, new methodologies, new rules of the game. This is not all romance, of course. It also makes good economic sense, as new ideas and technologies sustain institutions through publications, training courses, capacity building, development programmes and consultancies. Nor is it all new, despite the claims, since much of the novelty involves old ideas reset and rebranded, or indeed knowledge that is commonplace in one part of the industry being discovered as new in another. The preoccupation with the new is also ‘good’ politics: it helps to sustain established patterns of Western dominance, since it tends to be staff members of prestigious institutions in the West who have the time and resources to devise, patent and promote innovation, reinforcing patterns of tutelage with the two thirds world. While romance with the new forms the general background to the excitement around impact evaluation, the romance with science is what gives it its shape. Questions about impact – what difference has the project made in people’s lives? – may of course be addressed through a wide variety of methodological approaches.1 Increasingly, however, in practice the *Email: [email protected] © 2015 Taylor & Francis

Downloaded by [Sarah White] at 08:24 19 April 2015

2

S.C. White

term ‘impact evaluation’ is identified with a limited set of experimental or quasi-experimental methods, with the randomised control trial (RCT) as the ‘gold standard’ (Camfield and Duvendack 2014). RCTs are developed from drugs trials, which are based on an organic rather than social model of the human subject. A large and distinguished literature in the philosophy of science suggests that attempting to translate natural science paradigms to the study of human action is mistaken (see for example Barnes and Edge 1982; Giddens 1976; Hesse 1980; Hollis and Lukes 1982; Kuhn 1970; Wilson 1981). This is compounded by the paradox that while assessment methods become more abstract and technical, the ambition of development becomes more encompassing and more social (White 2012). Whereas previously, we are told, the primary objective of development was economic growth, it is now social transformation. Women’s empowerment, the subject of the RCT reviewed in this paper, represents a prominent example of this trend. Whereas the world of impact evaluation is dominated by quantitative economics, this paper, like others in this issue, proceeds from a qualitative perspective, grounded in sociology and social anthropology. At its heart is reflection on a particular impact evaluation being undertaken at present, a large RCT of a programme aiming to delay marriage and empower adolescent girls in Bangladesh (Kishoree Kontha (KK), or Adolescent Girls’ Voice). In reviewing this, I draw on my experience in undertaking ethnographic doctoral research on gender in rural Bangladesh (White 1992) and subsequent studies in Bangladesh on child rights, religion and marriage (for example, White 2002, 2010, 2013) as well as more recent research in a mixed method team (see White and Jha 2014).2 There are two main arguments in this paper. The first is that research is a social process and quality data collection and analysis depends on this being given central attention. The second is that methods used in evaluating impact must depend on the kind of impact we are aiming to assess. This follows the call of Guijt and Roche (2014, 50) for ‘relevant rigour’ and ‘rigorous relevance’. More specifically, my contention is that there are some aspects of empowerment for which quantitative methods are appropriate forms of assessment, while for others qualitative approaches are required. In order to determine this, we need to be clear about what empowerment means. I begin, therefore, with a brief introduction to thinking on women’s empowerment in development, including the talisman-like place that the empowerment of girls is occupying at present. Drawing on approaches across the political spectrum I suggest a model of three dimensions that together constitute empowerment, and can be used to structure its evaluation. I then describe the KK programme and the RCT to evaluate its impact. These are very closely intertwined, so it is not always easy to distinguish them. The next section reviews the design of the RCT, followed by a discussion of the findings as presented in the midterm report. As the RCT is ongoing, a final report is not yet available for analysis. I go on to assess more critically the design of the RCT and then discuss what might be different if a qualitative perspective were included. The paper closes by reflecting on the need for – and difficulties faced by – teams that genuinely incorporate at an equal level researchers of quantitative and qualitative expertise.

Empowerment: a contested concept While some mainstream international organisations like co-operative for assistance and relief everywhere (CARE) pronounce their ‘organisational commitment to women’s empowerment and gender equality’ (Karim et al. 2014, 14) as though empowerment were a commonly accepted, unproblematic good, in fact understandings of women’s empowerment are highly contested. Agencies like CARE may present it as a robust,

Downloaded by [Sarah White] at 08:24 19 April 2015

Journal of Development Effectiveness

3

tangible achievement, but others emphasise its unpredictability, variability, disjunctures and ‘hidden pathways’ (Cornwall and Edwards 2014, 2). Contestation as to the meanings of empowerment takes many forms, reflecting different underlying paradigms (more liberal and more radical) of both gender and development. The differences also have temporal and spatial dimensions. Batliwala (2007) describes how in India in the 1980s and 1990s, empowerment meant fostering spaces in which women could share experience, develop critical reflection on ideologies and societal structures and launch collective action for change. Alsop and Heinsohn (2005, 5) state the World Bank first recognised empowerment as an important means to poverty reduction in its 2000/2001 World Development Report, and comment that at their time of writing, empowerment was evident in the documentation of more than 1800 World Bank funded projects. This signals not only a remarkable move from the margins to the centre of the development industry, but also the way women’s empowerment had come to be valued as the means to other, economic, ends. Primary focus was also placed on economic empowerment at the individual level. Batliwala (2007, 563) describes how the content of the term shifted: we see the transition of empowerment out of the realm of societal and systemic change and into the individual domain – from a noun signifying shifts in social power to a verb signalling individual power, achievement, status.

As empowerment moved from the margins to the centre of development discourse so the form of empowerment projects expanded from informal, small-scale groups of selforganised women to include also formal, large-scale external interventions. Whose agency is seen to achieve empowerment also marks a critical line of difference. Those in the more radical tradition maintain that empowerment must be achieved by women themselves, it cannot be imposed by outsiders (for example, Rowlands 1995). By contrast, mainstream organisations like CARE unself-consciously assert their own agency as they identify ‘women and/or girls as their “impact groups” – that is, social groups whose lives they wished to change’ (Karim et al. 2014, 214). Disagreements about the concept of empowerment, not surprisingly, arise also with respect to methods of assessment. While some argue that progress towards women’s empowerment should be central to the post-2015 global development agenda (Harper et al. 2014) others consider that the concept is so imprecise that it should be jettisoned from the assessment of development impact3 (Aslanbeigui, Oakes, and Uddin 2010). While for Alsop and Heinsohn (2005, 4) ‘Degrees of empowerment are measured by the existence of choice, the use of choice, and the achievement of choice,’ others dispute even the possibility of incorporating it as a programme objective: ‘Empowerment is a process, not a fixed state, status or end-point, let alone a measurable outcome to which targets can be attached’ (Cornwall and Edwards 2014, 7). Against this complexity, the gender focus has also shifted, from the established emphasis on women to a new discovery of girls as the fount of development solutions. This reflects the influence in particular of wealthy private foundations. The Nike Foundation’s ‘Girl Effect’, now jointly funded by the UK Department for International Development (DFID), is a prominent example. The focus on adolescent girls is in line with the development industry’s perennial search for an ever more innocent target group, which cannot in any way be seen as responsible for the ills that beset it. Calls for intervention can thus confirm, rather than undermine, the neoliberal stress on individual responsibility. Adolescent girls also have obvious advantages in media and Public

Downloaded by [Sarah White] at 08:24 19 April 2015

4

S.C. White

Relations visuals. There is also a politics of language here, with the repeated use of ‘girl’ evoking an automatic protective response in comparison with the term ‘young women’ – which is the way those in their later teens might prefer to style themselves.4 It is interesting how this has replaced the earlier usage in development discourse of ‘girl child’, which implied ‘child’ alone referred to boys. Adolescent boys, a group with a much more troubling image, are simply erased in this homage to ‘girls’. The individualism of this campaign is extreme. The Girl Effect website encourages donors to invest in ‘an adolescent girl’. The very engaging video5 depicts a single girl, who gets pregnant and gives birth without any male figure appearing. Relationships with others are represented entirely in negative terms, through threatening images of invasive hands,6 except for consultation with a female doctor and the relationship between the girl become mother and her own daughter, who then reproduce themselves autonomously across the generations. The narrative similarly identifies relationships as hazard and empowerment as independence. Because of how others see her as marriageable, ‘a girl’s’ future at age twelve is ‘out of her control’. The pinnacle of achievement is identified, ‘now she is calling the shots’. The only reference to the collective within the video is the ‘we’ who ‘have a situation’. The ‘we’ is presumably the global community, implicitly in opposition to local actors defined as the problematic source of ‘the situation’. The video closes by claiming ‘50 million 12 year old girls in poverty equal 50 million solutions’. A quote on the webpage from Mark Lowcock, DFID permanent secretary, reinforces this individualism: ‘If you change the prospects of an adolescent girl on a big enough scale, you will transform societies’. The underlying paradigmatic conflict between the various conceptualisations of empowerment means that their differences cannot be resolved. However, for pragmatic purposes it is possible to identify from across the literature three key dimensions that together constitute empowerment. These can be used to assess both the model of empowerment that is being employed and the forms of empowerment that have been achieved. The first element is material and ideological change in the structuring environment. These should be such that the scope for women’s agency is expanded. The second is improved tangible outcomes for individuals and collectivities. These should be evident both in observable achievements and in enhanced agency or room for manoeuvre. The third is the expansion of ‘horizons of possibility, of what people imagine themselves being able to be and do’ (Cornwall and Edwards 2014, 16, emphasis added). This concerns people’s self-confidence as an individual or group and visions for what they personally and collectively may be able to become and achieve. The three elements combine objective and subjective perspectives and may occur in any order, or concurrently. Some of the variation between views of empowerment reflects which dimension they emphasise. My own view would be that all three should be present for empowerment to be said to have occurred. The inclusion of subjective elements means that empowerment cannot simply be attributed from outside, its assessment must involve the reflections of those who are its subject. The identification of three separate constituents indicates that empowerment is not a predictable process: even if one cog turns another may remain stubbornly immobile, or even swivel backwards. This model will be used to structure the discussion that follows. It is presented graphically in Figure 1. Adolescent marriage in Bangladesh Adolescent marriage plays a pivotal role in the Girl Effect video described above, as the source of all ills. Bangladesh is one of the countries specifically highlighted on the website: ‘By delaying child marriage and early birth for one million girls, Bangladesh could

Downloaded by [Sarah White] at 08:24 19 April 2015

Journal of Development Effectiveness

Figure 1.

5

The three constituents of empowerment.

potentially add $69 billion to the national income over these girls’ lifetimes’ (Nike Foundation n.d.). The stress on national income is relatively new. Historically arguments for increasing the age at which girls get married were typically couched in terms of population control.7 Delaying marriage was expected to reduce fertility both through reducing the number of years in which women were sexually active; and through the secondary effect that unmarried girls were more likely to stay at school, and educated mothers were consistently found to have fewer children. Since the remarkable demographic transition that Bangladesh has undergone (Caldwell et al. 1999; Egerö 1998), arguments for delaying marriage are more commonly made in terms of women’s health (delaying the age of first pregnancy), education, and – as in the present case – empowerment. The most recently available Bangladesh Demographic and Health Survey (2011) reports that of women currently aged 20–49, 74 per cent were married by age 18 (the legal age of marriage for women)8 (NIPORT et al., 2013, 50). However, along with changes in other demographic indicators, the prevalence of girls marrying in their early teens is rapidly declining. While 52 per cent of women aged 45–49 said they were married by the age of 15, this was the case for only 17 per cent of women aged 15–19 (NIPORT et al., 2013, 51). The median age at marriage among women aged 20–49 is 15.8 years, whereas it was 14.2 years in 1996–1997. Women’s age of marriage shows a strong positive correlation with both education (girls who have completed secondary or higher education marry 5 years later than those with none) and household wealth (girls from the highest wealth quintile marry on average 2 years later than those from the lowest wealth quintile) (NIPORT et al., 2013, 52). In terms of the framework in Figure 1, there is prima facie evidence for the view that raising the age of girls’ marriage in Bangladesh as a whole would be a positive change in the structuring environment. Local narratives that talk about how young brides can be moulded to fit the marital family, and how older (and more educated) girls may be troublesome wives and (especially) daughters-in-law, provide some support for this. However, while adolescent marriage may predispose vulnerability, it does not necessarily entail it. Through my own research in rural Bangladesh I have known women who were married extremely young, who

Downloaded by [Sarah White] at 08:24 19 April 2015

6

S.C. White

yet led strong, healthy, happy, loved and loving, fulfilled lives. I have also known women whose whole lives were ruined by an early inappropriate marriage. In the context of individual lives, the conviction that marrying later will be empowering for girls isolates one variable from a multilayered complex of issues. A key reason for early marriage in Bangladesh is concern for girls’ security. Once girls reach puberty they are vulnerable to a range of physical and reputational attacks, such that parents can feel that the safest course is to get them married. While girls’ and women’s mobility has greatly increased over the past 30 years, the safety for women of public space may if anything have declined (Feldman 2010). Widely publicised incidents of acid attacks by young men whose advances have been spurned are only the most high profile example of a broad range of sexual harassments that girls and young women face (for example, Chowdhury 2005). These include ‘eve-teasing’ comments and physical assault from men and malicious gossip from men and women. The questionnaires recognise this context in asking girls about their experience of harassment. However, this is a critical part of the structuring environment both for empowerment and for early marriage that neither the programme seeks to address nor the RCT to assess an impact on. Another issue is the longer term impact on specific individuals of delaying marriage. There might be two negative outcomes of this. First, as young women get older it may be more difficult for them to find a suitable groom. In rural Bangladesh at least, a good marriage is critical to women’s well-being (White 2013). It is the key rite of passage into womanhood, and it is through marriage and particularly the production of sons that women gain status and power. Second, if marrying later means a larger dowry is demanded, this may leave a young woman vulnerable to harassment in her marital family if her parents struggle to pay, and potentially undermine the support she could look for in her natal family if difficulties arise later in the marriage. That women value such relationships is shown in the long-established practice of women foregoing their inheritance rights to land in order to protect them (White 1992). In summary, raising the age at which girls and young women marry in Bangladesh represents a positive move in the structuring environment for empowerment. Whether it results in mobilising improved tangible outcomes and expanded horizons for girls and young women is a matter of social practice and empirical investigation. The Kishoree Kontha (KK) Programme9 The KK (adolescent girls’ voice) programme took place during December 2007–August 2010. It provided training and education to 42,244 girls aged 10–19 in 460 villages across 5 subdistricts in southern Bangladesh. The programme was implemented by Save the Children (USA) and national non-governmental organisation (NGO) partners. It is complementary to a government programme entitled the Female Secondary School Assistance Project, which provides a stipend to girls who stay at school and achieve grades at a certain level. The programme began with ‘community mobilisation’, to motivate adult villagers to support the programme, including providing a ‘safe space’ (usually a village house) for girls to meet. New staff were hired as ‘Field Trainers’ by local NGOs to undertake this work, typically young women who had completed at least high school education. Their responsibility was then to motivate girls to join the programme and to train peer educators who would provide the girls with training and education support.10 A group of modally 20 girls was intended to meet for up to 2 hours, 5 days a week for 6 months. Providing for adolescent girls to leave their own houses and meet together in a ‘safe space’ was itself seen as positive, increasing girls’ mobility and widening their horizons. All groups would

Journal of Development Effectiveness Table 1.

7

Different implementation packages of the KK programme.

Packages

Short name+

Basic

KK/0

Livelihoods Full Oil only Control*

KK/oil Oil C

Components Training in rights, reproductive health and negotiation skills Education support Basic plus financial training Livelihoods plus oil No education or training, only oil No education or training nor oil

No. villages 76 77 77 77 153

Downloaded by [Sarah White] at 08:24 19 April 2015

Notes: +Used in reporting midline results. *The control villages were assigned for comparison purposes by the RCT.

be taught about reproductive health, rights and some social – such as negotiation – skills. All groups would also give help with literacy and numeracy to girls who could not read or write, and study support to school-going girls. In some villages, in addition, financial training (on planning, budgeting and savings) would be given. At the end of 6 months it was intended that the girls, their parents and other village adults produce a plan for applying project learning in their daily lives (Scales et al. 2013). The Field Trainers would move on to establish new groups elsewhere. It was intended to cover 90 per cent of adolescent girls in the target communities within four 6-month cycles. In addition to the KK programme, in some villages a financial incentive was given to all girls aged 15–17 so long as they remained unmarried. This did not depend either on remaining in school, or on attending the KK programme. A total of 6536 girls received this incentive. The incentive took the form of 4 litres of cooking oil three times a year. The logic for this is that families incur costs by keeping daughters unmarried, as the amount of dowry demanded by grooms’ families rises with the age of the bride. The 12 litres of cooking oil worth $22 was deemed an appropriate subsidy to offset this, following a calculation by the Bangladeshi branch of the Population Council that the cost of a dowry rises by $14.7 dollars for every year a girl remains unmarried.11 The RCT seems to have been factored into the programme design since it was implemented through four different packages to test the efficacy of different dimensions of the programme. This is set out in Table 1. In terms of the model of empowerment set out in Figure 1, the KK programme addresses the structuring environment. This is not only the case for the age of marriage, as argued above, but also for KK’s model of knowledge transfer. This is designed to address a structuring factor – girls’ ignorance about reproductive health and limited literacy and numeracy – rather than encourage them to engage themselves in a more directly empowering form of knowledge work, such as Freirean-style critical reflection (for example, Freire 1972).

The RCT The evaluation aims to answer three key questions. First, can financial incentives be effective in delaying girls’ marriage? Second, does later marriage result in improved health, empowerment, and greater well-being for adolescent girls and their children? Third, what are the causal factors responsible for driving these changes: how important is the opportunity to stay longer at school, for example, compared to earning one’s own income, or simply marrying later?

Downloaded by [Sarah White] at 08:24 19 April 2015

8

S.C. White

The RCT studies a random subsample of 9100 households, approximately 25 per cent of households with adolescent girls in the programme area. A baseline survey was undertaken in 2007, interviewing a household head (n = 9047) and all girls aged 10–17 (n = 11,233). All schools (n = 920) and marriage registrars (n = 146) were surveyed, and one village leader (n = 460) was interviewed in each community. This same cohort was followed at midline (2011) and is planned for end line (2015). Questionnaire data were collected through interviews. For unmarried girls, these cover position in the household; education, including levels of attendance and potential obstacles to continuing; daily activities; matters discussed with adults in the family; sources of drinking water; income generating opportunities, activities, earnings and spending; savings and credit; mobility; gender attitudes; mental health; views on marriage; health; nutrition; menstruation; reproductive health knowledge and views; sexual harassment; sexually transmitted diseases; ‘developmental assets’ (DAP), a subset of items from US-based scale to measure ‘internal strengths and external support’ (Search Institute 2014); disaster planning; detailed questions relating directly to participation in and learning from KK; experience with other NGO programmes; and detailed contact information (including expectations of moving). The interviews also involved a small test of (very) basic literacy and numeracy (3 levels). Married girls were given the same questionnaire, with some additional questions about their marriages; pregnancy, birth and reproductive health practices. Questionnaires for parents and parents-in-law include a household roster; current status of girls (living/dead; location and so forth); education and marriage; economic status; religious practice; gender attitudes; girls’ mobility; and knowledge of/participation in the programme itself. There is no mention of qualitative methods in the RCT proposal. Up to this point there has been no qualitative work aside from the piloting of the questionnaire. Personal communication from the principal investigator (PI) states that this was extensive, involving several months of work, but it was not written up or recorded in any way. She explained that quantitative researchers are neither taught how to do this, nor is there any incentive for them to do so, since the journals in which they publish have no interest in it. There is an intention now to develop a qualitative element within the project, to be led by the Bangladeshi PI. While she is an experienced development worker, her training is as a medical doctor rather than as a sociologist or anthropologist. Furthermore, this pattern of overall leadership being in the hands of a quantitatively trained ‘international expert’ while qualitative work is led by the ‘national collaborator’ clearly reinforces an entrenched inequality between the two kinds of methods.12 The absence of qualitative fieldwork within the RCT is matched by the lack of any mention of sociological or anthropological readings on Bangladesh that might have injected a more social sensibility into the project design. Critically, there is no discussion of what empowerment might be in the context of girls or young women in Bangladesh. In fact, what brief discussion there is, is entirely in terms of ‘maternal and child health’ and ‘health service utilization’. It seems almost as though ‘empowerment’ was a label attached late in the process, perhaps reflecting the ubiquity of women’s empowerment in promotional material (Cornwall and Edwards 2014, 8). Reflections on the RCT Design Perhaps the most striking aspect of the RCT is that it is highly ambitious. The scope of the programme itself – 6 months’ peer education plus some cooking oil – seems very slight compared to the scale of impact the RCT is seeking to assess. In addition, as noted above, the age of marriage is already rising in Bangladesh, so calculating the net effect of a programme

Downloaded by [Sarah White] at 08:24 19 April 2015

Journal of Development Effectiveness

9

such as this is by no means straightforward. Furthermore, rather than limiting itself to the structuring environment, the RCT aims to assess all elements of the empowerment model. Tangible outcomes for individuals are to be captured in education, income generation, savings and credit, and ultimately pregnancy and child health indicators. Evidence of increased exercise of agency is sought in questions about discussion and negotiation, as well as sections on mobility and the DAP. Horizons are explored through questions about gender norms and values. In many ways, therefore, the RCT seems to go considerably beyond the programme itself. The numbers involved are also huge. The main part of the study will survey well over 20,000 people (including 11,350 adolescent girls) on three occasions each. A further census survey was administered to 75,000 households to provide background figures on age of marriage. Ensuring data quality with such large numbers requires major effort. Conduct of the survey was subcontracted, but monitoring was carried out by a directly employed group of 12. Difficulties noted in the midline report include the migration to other parts of Bangladesh of a third of the sample between the baseline and midline surveys (3660 girls) with several villages completely washed away as the course of the river shifted. In one area surveying was disrupted by rumours that its intent was to poison,13 convert to Christianity or abduct the girls. Some months later they were able sufficiently to rebuild trust that surveying in the area could be resumed. Overall, impressive efforts have been put into tracking – 88 per cent of the girls interviewed at baseline were re-interviewed in the midline survey. The quality of data in the background census, however, seems to have presented some problems. Need for ‘clarifications and cross-checks’ as well as recollecting some global positioning system (GPS) data meant that the data had not been received from the survey company by the time of the midline report. The main survey is also dauntingly long. The questionnaire for unmarried girls includes 1700 items plus a numeracy and literacy test. I would expect this to take a minimum of 2 hours and potentially a lot longer to complete. Even with a reliable and well-supported survey team, issues of respondent (and interviewer) fatigue are likely to affect data quality. This may in part underlie the difficulties they found getting especially urban respondents to participate in the midline survey, leading to Tk100 ($1.40) being given in compensation. It is also indicated by the fact that in some cases girls would agree only to a mini survey that could be completed over the phone. There is also a danger of data overload. Each round of the survey produces a massive amount of data to gather, enter, clean, and analyse, and then these have to be amalgamated to produce the panel data set. It is notable, for example, that the midline report does not present any breakdown of data by household wealth, despite this being a known factor affecting both time spent at school and age of marriage. In general the extent of analysis in the midline report is quite limited, which suggests potential issues of time with respect to analytical capacity. The survey is also challenging in its diversity. It seeks very different kinds of data from the girls, from mundane activities to personal views to testing of knowledge and competencies. This is asking a great deal from respondents within a single encounter and yet there is no suggestion that the surveys are completed in sections or over a number of sittings. In researching well-being, we have found that establishing the right kind of interaction that allows us to ask sometimes quite personal questions about subjective perspectives requires a more conversational approach and precludes asking detailed and complex questions about other parts of life. By contrast, there is no sense of awareness here that research is a social process. The questionnaire reads like a technical instrument to deliver data according to the researchers’ interests, rather than something that has been designed with the experience of a human respondent in mind. The RCT language of ‘treatment groups’ reinforces this in its very unfortunate dehumanising and objectifying terminology.

Downloaded by [Sarah White] at 08:24 19 April 2015

10

S.C. White

The length of the survey also means that some parts may be more effectively piloted to ensure they fit the context than others. The ‘developmental assets’ and mental health questions, for example, seem to be rather generic, a selection of items drawn from other contexts rather than ones developed anew through piloting with adolescent girls in Bangladesh. Work on subjective statements in other contexts points to the importance of each item being carefully piloted with detailed cognitive debriefing, to ensure that words are understood in the same way in each context. Brangan (2015) for example describes how she used a standard life satisfaction question with a respondent in a South African township. Her respondent answered, ‘I am not satisfied with life because I am still going on, getting skills… I still want to further my studies and become whatever I want to be.’ This is not the unhappiness of frustrated aspiration. Rather, Brangan comments, ‘Her dissatisfaction seems somehow positive. She radiated pride at the things she had achieved, and was confident that she would continue to progress’. Following common practice, the subjective items in the RCT questionnaires are not analysed individually, but as part of a scale. Their fit with one another is thus very important. It is thus concerning that the questions on attitudes and perceptions seem to mix normative and descriptive: ‘it is better to be a man than a woman’ is a clear normative statement, whereas ‘My parents treat me the same as they treat my brothers’ is descriptive. The DAP similarly seems to mix different kinds of items together – ‘I have parents who try to help me succeed’ is rather different to ‘I feel in control of my life and future’. Further issues with the grouping together of diverse items are discussed under findings below. Very little information is given on how the ‘control’ was established, though it is clear that control communities were near to the programme areas. It is obviously impossible empirically to identify villages that are comparable in every way except for the absence of the programme. This is one of the ways in which natural science and social science differ. However, the midline report shows a comparison from the baseline survey of key characteristics across the ‘treatment’ groups identified in Table 1. This indicated relatively few significant differences, none of which identified the control group as significantly different from the others. The midline report does however indicate that take-up in the programme communities had been lower than expected (54% of the age cohort of girls had attended at least one KK session), while there had been some leakage into the communities designated as control groups, with 13 per cent of girls in them having attended at least one KK session. GPS data showed uptake was lower in border areas than the centre of sub-districts. The take-up of the oil incentive was 47 per cent amongst eligible girls, but a minimal 0.55 per cent in the control group. Finally, there are odd lapses in attention. For example, while sampling and the randomisation thereof is a major preoccupation in programme design, the gender of the ‘household head’ who should be interviewed was not specified, so there is a mix of data from fathers and mothers, or fathers-in-law and mothers-in-law. This seems a little strange, given that both attitudes and knowledge might vary by gender. In addition, no questions (beyond the household roster) ask about boys in the household, nor does the programme apparently involve them in any way. It is interesting that this absolute form of bias should go unquestioned and unremarked. Kishoree Kontha Findings When it comes to assessing the findings, a note of caution is in order. At the time of writing the evaluation is still ongoing, so there has only been partial analysis of results.

Journal of Development Effectiveness

11

Table 2. Midline results of significant differences found in implementation package communities against control*. Packages Basic Livelihoods Full Oil only

Short name+

Less likely to be married?

More educated?

KK/0

0

0

KK/oil Oil

0 7%

7% 0

Downloaded by [Sarah White] at 08:24 19 April 2015

Note: * Significance is measured at p < 0.05, that is, there is less than 5% chance of the (here 7% degree of) difference occurring by chance.

The midline report stresses that results are preliminary and may change as new data becomes available. In particular, most of the girls at the midline (2011) were not old enough for final outcomes on education or age of marriage to be known. The midline report thus concentrates its analysis on the older cohort of girls (15 or 16 years at baseline) but the smaller sample size may have affected the ability to produce robust results. The intention here is not therefore to report the ultimate findings of the RCT but rather to discuss the methodological issues that arise through reflection on the findings presented in the midline report. The midline report of results begins by considering whether the programme has affected girls’ age of marriage or education, using a composite made up of a ‘family’14 of indicators for the education measure.15 The results are shown in Table 2. The oil-only group is shown to be 7 per cent less likely to be married than the control group. The oil-with-education/awareness (KK/oil) is shown to have a 7 per cent higher score on the education composite. No significant difference was found for the KK programme alone. The education composite is then broken down to identify on which specific items significant differences were found. These are shown in Table 3. Table 3 shows that no significant differences were found for 6 of the 12 items in the education composite. Also, although the oil-only intervention does not show significance in the composite, there is significant correlation with three important items (whether a girl is in school; highest class passed and reading score). By comparison, the KK/oil, which

Table 3. Significant differences in individual education results compared with control. KK Enrolled in school Highest class passed Days in school Time doing homework When last studied Reading score Maths score Expected highest class Expected obstacle class Discuss education with parents? Expenditure on education Credit for education Total significant items (out of 12)

KK/Oil

Oil

*

* *

* * * *

*

1

5

*

3

Downloaded by [Sarah White] at 08:24 19 April 2015

12

S.C. White

shows significance for the education composite, does not significantly correlate with whether a girl is in school or not. Both the packages including KK show significantly higher expenditures on education, suggesting that this might be accounted for by expenditures connected with the programme itself. The KK/oil combination indicates raised expectations, showing a significant increase in the highest class girls expect to achieve. However, such hypothetical questions are susceptible to ‘desirability bias’ in which respondents give the answer they believe the researchers wish to hear. When the results are considered item by item, the difference between KK/oil and oil-only seems much less robust. It is only on the maths score that the KK/oil shows a clear advantage over the oil only, and this needs to be weighed against the fact that the oil-only package shows a higher likelihood of girls being still in school. In reporting these findings on education, the midline report recognises that the results for the KK/oil and oil-only package are close, and the differences between them may disappear when the full data are available. Methodologically a more fundamental issue, however, is whether the discrepancy between the picture that emerges from the item-wise analysis of data compared to the education composite undermines the value of grouping items into a ‘family’ of variables.16 When the ‘family’ is separated out, it is shown to be composed of quite different kinds of items. This does not matter if it is simply a collection of associated items gathered together to see what emerges as significant, but it does become problematic when the ‘family’ is compressed into a single indicator. Are such diverse items telling us the same thing? And what is driving the logic of how the questions are answered? These issues recur all the more strongly when it comes to the diverse range of indicators which are used to assess ‘empowerment’. Again these are presented by ‘family’, under the headings of income generation, credit, savings, health, health care utilisation, mental health, attitudes, mobility, knowledge, time spent on different kinds of activities, negotiation, decision-making, harassment, contraception, child-birth choices. For 8 of the 15 variables no significant difference was found with the control group. The KK/0 shows a significant effect only for mobility (the narrative states this was driven by freedom of dress) and child-birth choices (where the significance is negative). The only significant correlations for oil were negative, in relation to income generation activities and knowledge. The combined programme KK/oil showed significant positive effects on health, negotiation and contraception. There is a real difficulty in interpreting these results. The midline report shows itemwise breakdown of results only for the knowledge and negotiation composites. Since the ‘knowledge’ composite is made up of issues taught in the KK programme, this seems more an output than an impact measure. The table shows significantly higher scores than the control group only for girls receiving both KK and oil programmes. The particular items involved knowledge of the risks of early pregnancy and of contraception methods. The marked difference with the lack of significance for the KK/0 group raises the question as to whether receiving the oil might have increased girls’ commitment to attend the classes. Although getting the oil was not conditional on attendance, it might have been perceived to be so. Personal communication from the PI confirms this supposition: there was indeed a positive association between receiving oil and attendance at the KK meetings. The strongest effect is observed for the KK/oil package in relation to the negotiation composite, at 6 per cent with a probability of less than 1 in a 1000 that this occurred by chance. On the face of it, this is quite a strong positive result. However, the negotiation composite is made up of nine items, six of which are hypothetical, asking girls what they

Journal of Development Effectiveness Table 4.

13

Significant differences in individual negotiation items compared with control. KK KK/Oil Oil

Have you discussed education goals with your parents or adult relatives? Have you tried to negotiate about marriage? When did you last speak with adult relatives about income and/or savings?+ Will you try to negotiate if your parents want you to marry before you are ready? What would you do if tomorrow your parents tell you that they found a good guy for you to marry?* Would you try to negotiate if a proposal of marriage was made tomorrow? Reasons for negotiation: want to study Reasons for negotiation: too young Reasons for negotiation: want to meet and get to know the guy

*

* neg * *

*

*

*

*

*

*

Downloaded by [Sarah White] at 08:24 19 April 2015

Notes: + Full item is not given in the report – this is the one that seems most likely from the questionnaire. * Refuse scored 5, ‘gladly accept’ scored 1.

would do if something were to happen. This is shown in Table 4, with the hypothetical items shown later and in italics. Only one of the three questions about what girls have actually done shows a significant difference with the control, and the results are ambiguous, with KK/oil showing a negative effect, oil only a positive one. By contrast, four of the six hypothetical items show significant positive results. Thus, the whole effect of this composite relies on hypothetical questions which might or might not reflect actual behaviour. It is quite possible that girls influenced by KK may believe that they would ‘negotiate’ with parents about a marriage proposal, but not in practice find themselves able to do so. The narrative suggests these results may indicate that girls in the KK/oil and oil-only treatments are more likely to negotiate marriage proposals, and that this may explain why fewer of these girls are married. However, the KK/oil programme, as shown in Table 2, does not show a significant effect on girls’ delayed marriage, while the strength of significance on these negotiation items is greater for the combined programme than for the oil only! A plausible alternative explanation would be that girls receiving the oil incentive are very aware of the intention to delay their marriages, and thus happily supply the ‘right’ answer to questions about this. Where girls have participated in addition in the KK programme, this message has been even more strongly reinforced. Such patterns of compliance in respondent behaviour, taking clues from the way questions are framed and seeking to give the answers that the interviewer is thought to want are well recognised in the literature (see, for example, Schwarz 1999). The girls may also feel a loyalty to the KK programme, and feel they should give the ‘right’ answers because of this, whatever their actual experience. This section and the previous one have raised some questions about the robustness of the quantitative approach, particularly drawing on insights that follow from seeing research as a social process. It identifies concerns with size, that the ‘power’ claimed for such large ‘n’ studies brings also the vulnerability of mass data collection whose quality is difficult to sustain. The same concern with data quality arises due to the length of the survey and the wide range of types of data it seeks to collect. Questions are raised about the value of grouping disparate items as a ‘family’, which are then reported as a composite indicator, because this obscures the social logic of how people might respond to particular kinds of question. Apparently significant results may similarly reflect this social logic, rather than a ‘real’ change in behaviour. The following section focuses on the

14

S.C. White

Downloaded by [Sarah White] at 08:24 19 April 2015

second main theme of the paper, that the methods used in evaluation must reflect the kind of impact they are intended to assess.

Structuring in the Qualitative This section moves on from what is to what might be if qualitative methods were brought in to shape the evaluation. Reflecting on the model in Figure 1, it is clear that each of the constituents of empowerment might be assessed through both qualitative and quantitative methods. Even the most subjective dimensions of the model, the ‘horizons of possibility’, can and are frequently assessed through quantitative measures. This has the advantage of enabling the comparison of responses across many respondents using a common numerical scale. But it has the disadvantage that you sift out all sense of what those horizons might be, as such scales have the strong effect of ‘disciplining’ respondents to answer in certain ways, rather than enabling them to express themselves as they would choose (White and Jha 2014). While empowerment may involve material changes and be assessed in part through quantitative measures, it is at heart a qualitative process, because it concerns the meaning of the changes that have taken place. Increased income may be assessed solely in objective, quantitative terms, for example, but whether this income becomes a means of empowerment, cannot. In Bangladesh specifically, women’s uptake of income generating activities often signals poverty, and the lack of other options rather than a positive choice (White 1992). The past 30 years have seen women’s employment in general become much more acceptable, but this still depends on the type of work done. How work is seen depends on local elaborations of class, ethnicity and religion as well as gender, its basis is social, not economic. Work that means going ‘outside’,17 in conditions that may challenge purdah18 norms and involve activities distant from the nurturing and caretaking thought suitable for women, may compromise rather than raise a woman’s status (White 2013). Some observers might maintain that it is precisely such non-conventional behaviours that constitute empowerment. In practice, however, whether empowerment is achieved depends on whether women are appropriately supported, especially through it being a conscious, collective action undertaken with others. Perhaps more often, the sanctions on nonconformist behaviour are so strong that the last thing women want to do is to challenge norms openly. On the contrary, women commonly hide the extent of their economic activities to sustain the equivocal protection of male ‘guardianship’ (Kabeer 1997). The obvious first phase in a project such as this is a qualitative study of what constitutes empowerment for adolescent girls in these communities. This could serve two purposes. First, it could map out a model that then could be used to identify key indicators for analysis in any subsequent quantitative study. This would seem a prerequisite when a data analysis plan has to be prepared in advance to satisfy paradigms of quantitative rigour. What matters to empowerment in this community at the levels of the structuring environment, tangible outcomes, or horizons of possibility? What indicators might be identified as markers of these? What do girls and boys, young women and men, see as constraints? What are concerns of their families? What is the current situation, and where do things already seem to be moving? How does all this differ by class, social status, ethnicity and religion? What about marriage? How do decisions about marriage get made? What are the different concerns that get weighed up in the process? How are the wider family and community involved? Identifying key indicators at this early stage

Downloaded by [Sarah White] at 08:24 19 April 2015

Journal of Development Effectiveness

15

would also mean that the bulk of the questionnaire could be significantly reduced, which should have an immediate positive impact on data quality. The second value of an initial qualitative phase would be to identify families and communities for ongoing in-depth qualitative study. However good the initial phase might be, people are endlessly surprising and will only gradually reveal the complexity of their lives. Having qualitative research running in tandem with the quantitative thus provides a richness that cannot be gained any other way. This may reveal complexity that is uncomfortable to the mechanistic model of an RCT, but it is complexity that is real. There may, for example, be spatial dimensions to empowerment, such that people behave and experience themselves very differently depending on where they are and who they are with. In Bangladeshi folk imagery, for example, the newly married bride in her father-in-law’s house is the quintessential figure of one who exists for the service of others. However, the rules of behaviour are very different when those same women visit their natal homes. Kotalova (1996) describes observing this directly, as young women travelling in buses removed items of outer dress and sat and talked more freely as they drew nearer to their natal villages. Building on this would suggest that girls may feel there are different arenas in which they operate (for example, in friendship groups; with older family members; out and about; in employment or enterprise) in which they are able to express themselves quite differently, including more or less freely and with greater or lesser assertiveness. Just as the conceptualisation of empowerment needs to admit more fluidity, so its indicators need to become more sophisticated. Take the common example of decision-making. Decisions vary significantly in their importance and sensitivity. The implication of having responsibility for decisions also varies: being responsible for procuring and preparing food for the family on a tight budget, for example, may be anything but empowering. A simple tally of who makes decisions is likely to be very unreliable, because it relates to issues that are essentially matters of judgement. Processes of decision-making are complex, especially where the issues are significant – even in one’s own life it is not always easy to identify all the factors involved, and what ultimately led to one choice being taken over another. Different actors also have different views. The historical reaction of feminists to the scandal of interviewing only the male head of household was to interview only women on gender issues. This has the merit of producing a clear story – in my own research separate interviews with husbands and wives have produced awkwardly diverse data even on apparently straightforward issues like assets, landholding or numbers of children (!) – but it merely substitutes one source of bias for another. When the matter at issue is an intangible like decision-making, where ‘negotiation’ may be subterranean, intertwined with the politics of kitchen and bedroom, it is unrealistic to expect a simple tally or a single informant will produce an accurate picture. Getting to know people and the ways they think over time also develops a familiarity that aids the accuracy of interpretation of quantitative findings. More significantly, perhaps, those same people can provide a reference group with whom to check back quantitative findings as they emerge, to explore whether these have ‘face validity’ with the people from whom they were drawn, and to discuss the meanings and emergent analysis with them. For example, the apparent efficacy of a few litres of cooking oil in encouraging parents to delay their daughters’ marriage seems a fascinating example of ‘nudge economics’ (Thaler and Sunstein 2008) whereby a relatively minor financial input can have disproportionate effect. But this finding is only suggestive: it begs for a qualitative study of decision-making around marriage to explore just how – and if – calculations over the oil played a part. Ultimately the question of whether a qualitative approach is included as a serious partner within the study depends on how researchers view the people whose lives they are researching. Are they the objects of study, to be tested from afar to see which lever needs

16

S.C. White

Downloaded by [Sarah White] at 08:24 19 April 2015

to be pressed to produce the change that outsiders desire? Or are they human subjects, who are already thinking about their lives and trying to do the best they can, whose engagement with and reflection on the research can add immeasurably to the value of the insight it has to offer?

Conclusion This paper has made two main arguments. First, it argues that research is a social process. If we are concerned about the quality of data we collect, instruments need to be designed with the experience of the respondent the primary concern. We also need to be aware of the danger of false positives in analysis, as people provide the ‘right’ answers suggested either by the questionnaires or the programme itself. Second, it argues for an open choice of methods, which reflects the kind of impact they are intended to assess. Typically it is the default settings of the research team that determines the methodology, not the demands of the problem at hand. In the case of empowerment the scope of what is to be assessed is extraordinarily complex, combining objective and subjective dimensions, and hotly contested. The paper suggests a pragmatic approach to this, identifying three key constituents of empowerment, and suggesting that these be used to help focus on the processes under consideration and so structure their evaluation. While there is now increasing lip service paid to the importance of qualitative and mixed methods in impact evaluation, Camfield and Duvendack (2014, 6) warn: the practice of qualitative and mixed methods studies rarely matches the rhetoric as the range of research designs is limited, qualitative research is typically post hoc rather than ex ante, [and] analyses that integrate qualitative and quantitative data are rare.

Qualitative approaches will only be taken seriously when there are qualitative specialists at senior levels in research teams, fully involved in all stages of design, implementation and analysis. Research is not only a social process in the field, it is also a social process in the office, reflecting the community of actors involved. What is needed goes beyond quantitative researchers doing a short field trip and chatting to a few people and calling it mixed methods. It even goes beyond having two strong distinct qualitative and quantitative studies. What is needed is a shared process in which qualitative researchers are fully involved in the design and analysis of quantitative instruments, and vice versa. Having participated in several such teams, I do not state this lightly. It is very hard to work in a genuinely mixed methods team. It challenges the comfort zones not only of technique, but also of epistemology and even ontology, basic ways of being in the world. It forces reflection on one’s own institutional and professional investments and locations, including the challenges of participatory and post-colonial critiques. But it would be mistaken to lay all the responsibility on individuals, because the bias towards a single disciplinary approach is embedded in the structures of the development and even more the academic industry. The major professional incentives do not reward mixed methods work, top journals are not interested in publishing it, rules on word length, style of writing, and logic of argumentation do not support it, and reviewers are rarely broad-minded or welleducated enough to see beyond the blinkers of their own discipline. In closing, it must be recognised that even the perfect combination of methods amongst academics only goes a very small part of the way. The vision of empowerment set out by Batliwala (2007) was of a self-driven dynamic process of women working

Journal of Development Effectiveness

17

together in community. The KK programme is very different. It is an external programme designed by a set of large external agencies that aim to engineer social, cultural and economic change in the lives of others. As mentioned above, it is sometimes difficult to distinguish the RCT from the programme, since the programme design seems itself to have been ‘experimental’. While including qualitative researchers as equal partners should provide more scope for the respondents’ own perspectives, it is still ultimately an extractive process, with the power of design and analysis in the hands of outsiders. Rowlands (1995, 105) warns that ‘most professionals are trained to work in ways that disempower’. If girls’ and women’s empowerment is such a key aim of development, we perhaps need to ask how far Rowlands’ warning might apply to practices of assessment, and what might it might mean to approach the field of impact evaluation more explicitly as a field of power.

Downloaded by [Sarah White] at 08:24 19 April 2015

Acknowledgement This article was commissioned by 3ie and the author is grateful for this support, as also for the helpful comments of other contributors to this volume.

Disclosure statement The views expressed in the paper, however, are those of the author alone.

Notes 1.

2.

3. 4. 5. 6. 7. 8.

9. 10. 11.

See, for example, the Assessing Rural Transformations project, which is developing rigorous methods for qualitative impact evaluation (http://www.bath.ac.uk/cds/projects-activities/assessing-rural-transformations/) and NGO approaches to assess changes in quality of life resulting from their projects in White and Abeyasekere (2014). My understanding of the world of RCTs and their form of impact evaluation has been greatly enhanced by the conference ‘Making Impact Evaluation Matter’, organised by 3ie in partnership with the Asian Development Bank and the Philippine Institute of Development Studies, Manila, September 2014. Aslanbeigui et al. make this argument only in relation to microfinance, but from the tenor of the article I would be surprised if they would be unhappy to extend its message to the whole of development. This is consistent with, and builds upon, the use of ‘child’ to refer to all those under 18 in the context of ‘child rights’. There have been a number of videos over the years, all using the same basic format. The one described here was viewed on 15 December 2014 and is entitled ‘We have a problem’. Available at https://www.youtube.com/watch?feature=player_embedded&v=1e8xgF0JtVg). I would argue these hands are presented in sexualised way, raising associations with groping and inappropriate touch, at the very least. In Bangladesh at least, the priority donors placed on reducing population growth was a major impetus behind the adoption of programmes targeting women in the late 1970s and through the 1980s, including those under the banner of ‘empowerment’. In a surprising move, the Bangladesh government announced in September 2014 its intention to reduce the minimum age of marriage for girls from 18 to 16 years and boys from 21 to 18 years, while increasing the penalties for violation, http://www.bbc.co.uk/news/world-asia29216141, consulted 29 September, 2014. After widespread protest, both within and beyond Bangladesh, the government decided to keep girls’ minimum age of marriage at 18. All the information in this section is taken from the Midline Report unless otherwise indicated. The peer educators were given between 16 and 40 hours training. Given the scale of dowries in Bangladesh and their huge variation, not least according to the wealth of families, it is very difficult to understand how such a calculation could be made, and

18

12. 13. 14. 15. 16. 17.

Downloaded by [Sarah White] at 08:24 19 April 2015

18.

S.C. White how the sum arrived at could be so small. Unfortunately, the full reference was not provided in the report so I was not able to pursue this. When I raised this issue in the 3ie Manila conference, my interlocutors confirmed that this pattern of ‘national’ PI heading any qualitative work held in ‘100%’ of cases. Plates were given as a token of thanks for people’s time. The rumours suggested eating from these plates would result in poisoning. ‘Family’ is the term used in the report, referring to Kling et al. (2007) (full reference not given in the report, so cannot be shown here). The education composite is made up of measures of school attendance; time studying; educational expectations; maths and reading test scores; financing and discussion of education. I recognise, of course, that such differences are common in shifting from univariate to multivariate analysis. ‘Outside’ is itself socially defined, so going to the family fields may not constitute going outside, as count as part of one’s own ‘household’ (bari). Purdah (literally a curtain or veil), or the seclusion of women, is an important value within gender relations in Bangladesh. There is wide variation in how this is observed in practice.

References Alsop, R., and N. Heinsohn 2005. “Measuring Empowerment in Practice: Structuring Analysis and Framing Indicators.” World Bank Policy Research Working Paper 3510. Accessed August 18, 2014. http://siteresources.worldbank.org/INTEMPOWERMENT/Resources/41307_wps3510.pdf Aslanbeigui, N., G. Oakes, and N. Uddin. 2010. “Assessing Microcredit in Bangladesh: A Critique of the Concept of Empowerment.” Review of Political Economy 22 (2): 181–204. doi:10.1080/ 09538251003665446. Barnes, B., and D. Edge, eds. 1982. Science in Context: Readings in the Sociology of Science. Milton Keynes: Open University Press. Batliwala, S. 2007. “Taking the Power Out of Empowerment – an Experiential Account.” Development in Practice 17 (4–5): 557–565. doi:10.1080/09614520701469559. Brangan, E. 2015. “Staying well in the South African Township: Stories from Public Health and Sociology.” In Wellbeing: Culture, Method, Politics, edited by S. C. White and C. Blackmore. Basingstoke: Palgrave Macmillan. Caldwell, J. C., B.-E. Khuda, B. Caldwell, I. Pieris, and P. Caldwell. 1999. “The Bangladesh Fertility Decline: An Interpretation.” Population and Development Review 25: 67–84. doi:10.1111/j.1728-4457.1999.00067.x. Camfield, L., and M. Duvendack. 2014. “Impact Evaluation – Are we ‘Off the Gold Standard’?.” European Journal of Development Research 26 (1): 1–11. doi:10.1057/ejdr.2013.42. Chowdhury, E. H. 2005. “Feminist Negotiations: Contesting Narratives of the Campaign against Acid Violence in Bangladesh.” Meridians: Feminism, Race, Transnationalism 6 (1): 163–192. doi:10.2979/MER.2005.6.1.163. Cornwall, A., and J. Edwards. 2014. “Introduction: Negotiating Empowerment.” In Feminisms, Empowerment and Development. Changing Women’s Lives, edited by A. Cornwall and J. Edwards, 1–31. London: Zed books. Egerö, B. 1998. “Bangladesh in the Fertility Transition: The Role of Social Change and Gender.” South Asia: Journal of South Asian Studies 21 (2): 87–106. doi:10.1080/00856409808723344. Feldman, S. 2010. “Shame and Honour: The Violence of Gendered Norms under Conditions of Global Crisis.” Women’s Studies International Forum 33 (4): 305–315. doi:10.1016/j. wsif.2010.02.004. Freire, P. 1972. Cultural Action for Freedom. Harmondsworth: Penguin. Giddens, A. 1976. New Rules of Sociological Method. London: Hutchinson. Guijt, I., and C. Roche. 2014. “Does Impact Evaluation in Development Matter? Well, It Depends What It’s For!.” European Journal of Development Research 26 (1): 46–54. doi:10.1057/ ejdr.2013.40. Harper, C., K. Nowacka, H. Alder, and G. Ferrant. 2014. “Measuring Women’s Empowerment and Social Transformation in the Post-2015 Agenda.” Overseas Development Institute. Accessed December 19, 2014. http://www.oecd.org/dev/poverty/ODI%20-%20post-2015%20social% 20norms_final.pdf

Downloaded by [Sarah White] at 08:24 19 April 2015

Journal of Development Effectiveness

19

Hesse, M. 1980. Revolutions and Reconstructions in the Philosophy of Science. Brighton: Harvester Press. Hollis, M., and S. Lukes, eds. 1982. Rationality and Relativism. Oxford: Blackwell. Kabeer, N. 1997. “Women, Wages and Intra-household Power Relations in Urban Bangladesh.” Development and Change 28 (2): 261–302. doi:10.1111/1467-7660.00043. Karim, N., M. Picard, S. Gillingham, and L. Berkowitz. 2014. “Building Capacity to Measure Longterm Impact on Women’s Empowerment: CARE’s Women’s Empowerment Impact Measurement Initiative.” Gender and Development 22 (2): 213–232. Kotalova, J. 1996. Belonging to Others. Cultural Construction of Womanhood in a Village in Bangladesh. Dhaka: UPL Ltd. Kuhn, T. S. 1970. The Structure of Scientific Revolutions. Chicago, IL: University of Chicago Press. National Institute of Population Research and Training, Mitra and Associates, ICF International. 2013. Bangladesh: Demographic and Health Survey, 2011. Dhaka. Rowlands, J. 1995. “Empowerment Examined.” Development in Practice 5 (2): 101–107. doi:10.1080/0961452951000157074. Scales, P. C., P. L. Benson, L. Dershem, K. Fraher, R. Makonnen, S. Nazneen, A. K. Syvertsen, and S. Titus. 2013. “Building Developmental Assets to Empower Adolescent Girls in Rural Bangladesh: Evaluation of Project Kishoree Kontha.” Journal of Research on Adolescence 23: 171–184. doi:10.1111/j.1532-7795.2012.00805.x. Schwarz, N. 1999. “Self-Reports: How the Questions Shape the Answers.” American Psychologist 54 (2): 93–105. doi:10.1037/0003-066X.54.2.93. Search Institute. 2014. “40 Developmental Assets for Adolescents.” Accessed August 17, 2014. http://www.search-institute.org/content/40-developmental-assets-adolescents-ages-12-18 Thaler, R. H., and C. R. Sunstein. 2008. Nudge: Improving Decisions and Health, Wealth and Happiness. New Haven, CT: Yale University Press. White, H. 2014. “Current Challenges in Impact Evaluation.” European Journal of Development Research 26 (1): 18–30. doi:10.1057/ejdr.2013.45. White, S. C. 1992. Arguing with the Crocodile: Gender and Class in Bangladesh. London: Zed. White, S. C. 2002. “From the Politics of Poverty to the Politics of Identity? Child Rights and Working Children in Bangladesh.” Journal of International Development 14 (6): 725–735. doi:10.1002/jid.919. White, S. C. 2010. “Domains of Contestation: Women’s Empowerment and Islam in Bangladesh.” Women’s Studies International Forum 33 (4): 334–344. doi:10.1016/j.wsif.2010.02.007. White, S. C. 2012. “Making Development Personal.” In New Perspectives in International Development, edited by M. Butcher and T. Papaioannou, 191–214. London: Open University and Bloomsbury Academic. White, S. C. 2013. “Patriarchal investments: Marriage, Dowry and Economic Change in Bangladesh.” Bath Papers in International Development and Wellbeing, January 19. White, S. C., and A. Abeyasekere. 2014. Wellbeing and Quality of Life Assessment: A Practical Guide. Rugby: Practical Action. White, S. C., and S. Jha. 2014. “The Ethical Imperative of Qualitative Methods: Developing Measures of Subjective Dimensions of well-being in Zambia and India.” Ethics and Social Welfare 8 (3): 262–276. doi:10.1080/17496535.2014.932416. Wilson, B., ed. 1981. Rationality. Oxford: Blackwell. Especially chapters by Winch, Hollis, Jarvie and Agassi, and McIntyre.