Layers of meaning: adding depth and texture to measure and understand impact during the pandemic
We all know the importance of good listening skills. For example, it’s always been an important focus in the leadership and management training I’ve developed and facilitated. During the last year, however, listening has been an even more important part of our impact evaluation work than usual. While surveys are often a cost effective and quick way of collecting data for evaluation from a broad range of customers or stakeholders, having the opportunity to include semi-structured interviews as part of our data collection and analysis has been invaluable during a year when so much unexpected change has happened. Without the opportunity to dig a little deeper–to add depth–to understand customers, students or stakeholders’ lived experience of the policy change or intervention within the context of a worldwide pandemic, it would have been harder to build up a complete picture of the range of drivers influencing change, outcomes and impact for our clients. The interviews allowed us to identify the threads, gain depth and texture–to build and articulate an often complex change process. We also found that interview participants often thanked us for listening–the opportunity to talk about their experience and know that their feedback was valued.
As a team, our academic research training and experience helps us when planning for, and collecting, data. So, what makes a good evaluation interviewer and how does careful listening help? A few thoughts from the team below:
Impact measure – recognising what the client wants to know and understand and listening carefully for points to explore further. This supports the ability to be able to dig deeper in the interview, to keep the interview on track but also to know when it’s okay to let the interviewee talk about something slightly different, which is important to them and may be an unexpected consequence of a policy change or intervention.
Comfort - making the person being interviewed feel as comfortable as possible. The aim of the interviews as part of an evaluation is usually to explore the genuine experiences of the users. By knowing how to build a meaningful connection with the interviewee, you are more likely to get honest and well-developed answers. Allowing the interviewee time to think and to talk and showing you are listening, all helps to ensure you and the participant get the most from an interview.
Empathic listening – as an interviewer you are trying your best to understand the perspective of the interviewee and not be judgemental. Remembering that you are there to gain their perspective is important – there is no right or wrong answer. Despite interviews taking place virtually, recognising differences in voice and body language (if webcams are turned on) can give insight into how the interviewee is feeling. A simple strategy of checking/recapping with the interviewee shows that you have captured their point of view, will put them at ease and can also strengthen rapport. It also means you have accurate data as you check interpretation.
Boundaries – respecting the boundaries of the interviewee. Simply asking them if they are happy to share or erring on the side of caution is recommended. Linked to the point below on transparency, we never turn on the video or voice recorder until we have double-checked that the interviewee is happy for us to record the interview.
Transparency – being clear with the interviewee how the data they are providing will be used – and knowing the details yourself. This information should be shared ahead of the interview. For example, we found people are often more forthcoming knowing that we would entirely anonymise their input, including anything that would identify them individually, in our reports. It is important to be listening carefully, so you can clarify and confirm whether something the participant has said might identify them – and discuss how that might be managed.