PAPER: INCREASING THE RIGOUR AND TRUSTWORTHINESS OF PARTICIPATORY EVALUATION: LEARNING FROM THE FIELD

TITLE:

Increasing the rigour and trustworthiness of participatory evaluations: Learnings from the field

WRITER:

June Lennie


Paper presented at the Australasian Evaluation Society 2005 International Conference 10 -12 October

Brisbane, Queensland www.aes.asn.au

MY SUMMARY

Abstract –pp 1

Participatory evaluation and participatory action research (PAR) are increasingly used in community-based programs and projects. These methodologies emphasise knowledge generated and constructed through lived experience rather than through social science (Vanderplaat 1995). The scientific ideal of objectivity is usually rejected in favour of a holistic approach that openly acknowledges and takes into account the diverse perspectives, values and interpretations of participants, stakeholders and evaluation professionals. However, evaluation rigour need not be lost in this approach. Increasing the rigour and trustworthiness of participatory evaluations and PAR increases the likelihood that results are seen as credible and are used to continually improve programs and projects.

Drawing on learnings and critical reflections about the use of feminist and participatory forms of evaluation and PAR over a ten year period, valuable sources of rigour identified include: (pp. 1)

•     Participation and communication methods that develop relations of mutual trust and open communication.

•     Using multiple theories and methodologies, multiple sources of data, and multiple methods of data collection.

•     Ongoing meta-evaluation and critical reflection.

•     Critically assessing the intended and unintended impacts of evaluations using relevant theoretical models.

•     Using rigorous data analysis and reporting processes.

•     Participant reviews of evaluation case studies and reports.

In PAR projects and participatory evaluations, the emphasis is mainly on knowledge generated and constructed through lived experience rather than through social science (Vanderplaat 1995). The scientific ideal of objectivity is usually rejected in favour of a holistic approach that incorporates the diverse perspectives, values, agendas and interpretations of participants, stakeholders and evaluation professionals. However, as Dick (1992 and 1999), Guba and Lincoln (1989), Thomas (2000) and others suggest, rigour need not be lost in this approach.

PAR encourages the active involvement of participants and stakeholders in designing and conducting projects and supports capacity building processes. It can be a valuable method for involving a diversity of people in projects, generating appropriate action, new ideas and long term visions, fostering ongoing change and improvement, and regularly reflecting on outcomes (McTaggart 1991). Thomas (2000, p. 112) argues that PAR 


Three main reasons have been put forward for increasing the involvement of community participants and other stakeholders in evaluations: pp.3

(1) to increase utilisation of evaluation results; (2) to represent the values and concerns of the multiple groups involved in decision-making; (3) to promote the empowerment of disenfranchised stakeholder groups previously left out of the process (Papineau & Kiely 1996, p. 81).

Different forms of participatory evaluation emphasise different levels of participation. Empowerment evaluation is notable in that it encourages active involvement of a diversity of stakeholders in all stages of the evaluation and has a number of clearly articulated principles, including improvement, democratic participation, organisational learning, accountability, and using evidence-based strategies (Fetterman & Wandersman 2005).

Participatory feminist evaluation methodologies and feminist PAR are openly political approaches which are often underpinned by praxis feminist theories and methods. These methods seek to understand, give voice to and validate women’s needs, values and lived experiences and to take the macro and micro contexts into account (Lather 1991; Lennie 2002b). The evaluation usually aims to improve programs in ways that better meet women’s diverse needs and agendas and to bring gender differences and issues to the fore. While gender differences are a key focus, other differences such as age, ethnicity, educational level and occupation are also taken into account. In its more critical forms, feminist PAR and evaluation also includes analysis of the gendered power relations involved in a program and the contradictory outcomes of the use of participatory methodologies that can often be overlooked (Lennie et al. 2003). Pp 3

Issues raised by participatory evaluation and PAR

The use of participatory research and evaluation methodologies raises many complex theoretical, methodological and ethical issues that have implications for the quality of the evaluation and the trustworthiness of the findings and outcomes. They include:

1.      The need to ensure stakeholder representativeness. Several studies have highlighted the barriers and issues that arise when researchers and evaluators attempt to involve a broad diversity of participants and stakeholders which need to be addressed

2.      The potential that the conflicting agendas and perspectives of various stakeholders groups will hinder the success of the evaluation

3.      The need to critique the concepts of empowerment and participation when assessing the impacts of participatory research and evaluation. Idealistic or naïve assumptions are sometimes made that community participation will automatically lead to empowerment. However, Humphries (1994) points out that the concept of empowerment can be used to justify oppressive practices: while the forms of participation identified range from co-option to collective action (Martin 2000 pp 200) pp 3

** Humphries, B (1994). Empowerment and social research: Elements for an analytic framework, in B. Humphries and C.Truman (eds) Re-thinking Social Research. Anti-Discriminatory Approaches in Research Methodology, pp. 185-2004. Aldershot:Avebury

Increasing rigour and trustworthiness: Learnings from the field 
pp4

Increasing the rigour and trustworthiness of participatory evaluations clearly requires the use of methods, criteria and strategies that are appropriate to those involved in a particular program or project and the skills, knowledge and resources available. The ideal is that rigour is incorporated into all stages of the evaluation and that the theories and assumptions of evaluation professionals and others are continually questioned. 
Through my work on feminist PAR and participatory evaluation projects with people in rural and regional Queensland communities and diverse government and industry partners over the past ten years and my reviews of the literature in this field, I have identified a number of strategies that can increase the rigour and trustworthiness of each stage in participatory research and evaluation projects. These strategies include: pp4

Community participation, engagement and communication methods that develop relations of mutual trust and open communication.

Using multiple theories and methodologies, multiple sources of data, and multiple methods of data collection.

Ongoing meta-evaluation and critical reflection.

Critically assessing the intended and unintended impacts of evaluations using relevant theoretical models.

Using rigorous data analysis and reporting processes.

Participant reviews of evaluation case studies, data analysis and reports.