Appendix C: Pre-birth testing Project Evaluation (CRESA)

By Julie Warren and Sam Mortlock, Centre for Research, Evaluation and Social Assessment (CRESA)

Summary and conclusions

The Bioethics Council is carrying out a public engagement process that seeks deliberative dialogue, the focus of which is pre-birth testing. Pre-birth testing includes a range of tests before birth that are available to identify whether a child may suffer from possible illnesses, diseases or disorders (including genetic and other conditions). Its use, particularly in genetic testing, is widening. The deliberative process is occurring in two stages, with the first stage (the Framing Exercise) completed over July and August 2007. This involved small groups of invited members of the public intensively working through a prescribed process over a 1-day period. The participants, through a facilitated process, were asked to collectively define the problem situation by identifying the underlying concerns and values, and develop three or four approaches to how pre-birth testing might be dealt with by decision-makers. Six of these exercises were carried out in July and August 2007.

The next stage, the wider deliberative process, will involve taking the approaches developed to a wider audience through a deliberative process. 

The evaluation is also in two phases to match that of the dialogue process. Together they are intended to inform the Bioethics Council, as it builds on best practice in both deliberative dialogue processes and their evaluation. The evaluation approach has been informed by evaluations of similar deliberative dialogue processes internationally, and uses previously tested success criteria relating to the ‘acceptance’ of the exercise to the participants and the ‘good process’ that indicates the exercise has been conducted well. The ‘acceptance’ criteria include: representativeness, independence, early involvement, influence and transparency. The ‘good process’ criteria include: resource accessibility, task definition, structured decision-making, and cost effectiveness.

The first phase of the evaluation collected data on the achievement of these success criteria as well as seeking feedback from participants and facilitators about their experience of the exercises. Specifically, the evaluation sought their views of what worked and did not work in the New Zealand context (given that the dialogue process implemented is not a New Zealand model) and how the process could be improved.

Evaluation findings show a high achievement of the ‘acceptance’ success criteria. That is, participants rated the planning and principles underpinning the Framing Exercises highly. They were generally very positive about the extent to which the Framing Exercises provided a fair representation of views of the public about pre-birth testing, lacked any bias in their running, had clear purpose and were sufficiently resourced to enable effective participation. They also had confidence that the Bioethics Council would use the approaches generated as a basis for wider deliberation.

Participants were less positive about the ‘running’ of the Exercises. To some extent, this was a reflection of the newness of the process to New Zealand and the lack of familiarity of the process by the facilitators. For instance, they were less positive about the extent to which the nature and scope of the Framing Exercise was defined; and about the structure and conduct of the Exercise. Some were also concerned about the active role that some Council members took in the Exercises.

Participants’ motivations for participating in the Exercises included, in more or less equal measure, an interest in the technologies themselves and in the development of public deliberative processes in New Zealand.  Some were also interested in ensuring that the values and viewpoints of Māori and other minority groups are represented.

People’s expectations of the day and the value they gained were consistent with their motivations to participate. That is, they revolved around listening to and understanding a range of values and viewpoints on pre-birth testing, and participating in organising insight into public deliberative processes.

The improvements participants suggested were predominantly around delivery, including the facilitation. They were generally happy with the intention and structure of the day, although they had some concerns about the time available (most thought there was insufficient time to work through the process). Suggested improvements to the delivery of the process included a better description of the process, clearer directions at the start of the day and more clarity around the role of Council and Secretariat members (with some suggestion that their roles could be more passive, with participation when technical input was required).

The role of the facilitator or moderator is key to the effectiveness of the Exercises. The implementation of the earlier Framing Exercises reflected the facilitators’ lack of familiarity (and sometimes understanding) of the intended structure of the event. However, as the facilitators became more familiar with the Framing Exercise process, their management of the process seemed to improve. Nevertheless, their reflections of the process, and the experiences of the participants, suggest that early cabability building is still necessary to ensure maximum benefit from future events.

Evaluation findings identified key competencies for facilitators as: a detailed understanding of the intended process; a firm, instructive approach to the facilitation; good organisational skills; and flexibility to accommodate participants’ capabilities.

A major consideration for planning and conducting future Framing Exercises is their duration. Time constraints, when the Exercises were held over a 1-day period, meant that suggested time allocations for each phase of the Framing Exercise structure in the ‘curriculum’ guidelines could not be adhered to. Carrying out future exercises in two full or half days could address these constraints.

The evaluation findings clearly demonstrate the high value participants placed on being involved in the development of deliberative processes in New Zealand. Indeed, an interest in the development of such processes, and involvement in that development, was often as strong a motivating force as an interest in pre-birth testing itself.

The findings also provide evidence that the intention of the deliberative process implemented has resonance for the New Zealand public overall and, it seems, for particular cultural groups. The structure of the day was also sufficiently flexible to accommodate different cultural approaches to deliberation. However, Māori and Pacifica participants would like resources such as the brochure, the DVD, illustrative stories and any statistics used to reflect their cultural values.

There needs to be more recognition of the need for capacity building to maximise the benefits of such public deliberation models. For facilitators, time and other resources need to be devoted to ensuring that they have appropriate understanding of all aspects of the model — the theory and the practice.

And participant readiness to engage also needs attention. It may have been helpful to develop more resources to use at the start of the exercises that clearly lays out what the event will entail, what would be expected of participants and what outputs might look like.

[ Previous page | Next page ]