Escreva para pesquisar

Em Profundidade Tempo de leitura: 5 minutos

A mudança mais significativa é útil para avaliar a gestão do conhecimento? 5 coisas que aprendemos


From the cover of “The Most Significant Change (MSC) Technique” guide by Rick Davies and Jess Dart.

 

The Most Significant Change (MSC) technique—a complexity-aware monitoring and evaluation methodis based on collecting and analyzing stories of significant change to inform adaptive management of programs and contribute to their evaluation. Based on Conhecimento SUCESSO’s experience with using the MSC questions in four evaluations of knowledge management (KM) initiatives, we have found it to be an innovative way para demonstrate a impact of KM on the ultimate outcomes that we are trying to achieve—outcomes like knowledge adaptation and use and improved programs and practice. 

As knowledge management (KM) practitioners, we often get asked why family planning and reproductive health (FP/RH) and other public health programs should invest in KM interventions. Specifically, people want to know what kinds of outcomes they can expect to achieve by investing in KM. Demonstrating the impact of KM on higher-level outcomes—like using data and information to inform program or policy decision making or applying knowledge to improve health systems—can be challenging because it can be hard to tease out the specific impact of KM tools and techniques when they are used in concert with other public health activities.

This is where complexity-aware monitoring and evaluation (M&E) methods come in. We’ve used the questions from the Most Significant Change (MSC) technique—a complexity-aware M&E method—to contribute to the evaluation of several KM initiatives that we’ve led and found it to be a useful method for demonstrating the benefits (outcomes) of these KM interventions.

What Is MSC?

MSC is a participatory M&E method based on stories rather than indicadores (see figure). It was designed to be used throughout a project’s lifecycle to inform ongoing course corrections (in other words, for ongoing monitoring and adaptive management), but it can also contribute qualitative data on a project’s outcomes and impact.

Figure outlining 3 key steps of the Most Significant Change technique. First, collect stories of significant change by asking people: What do you think was the most significant change? Why was this significant to you? What difference has this made now or will make in the future? Second, select the most significant stories by asking panels of stakeholders to discuss the value of the reported changes in the stories and using a method like majority rules, iterative voting, or scoring to select the most significant stories. Third, feedback the selected stories and the rationale with previous and subsequent panels to promote dialogue and learning, which can reinforce or change what people value and give an indication of what the project should focus on.
Figure.

MSC stories are based on respondents’ answers to three key MSC questions:

  • What do you think was the most significant change?
  • Why was this significant to you?
  • What difference has this made now or will make in the future?

The questions, at first glance, seem simple—maybe even too simple! But as we used the MSC questions in evaluations of KM initiatives as diverse as our Learning Circles program, O campo, our KM capacity strengthening interventions in Asia and East Africa, and our partnership with five francophone African countries to integrate KM into costed implementation plans (CIPs), we found that MSC is actually powerful!

Five Lessons We Learned About MSC

1. The MSC questions are quite mighty, allowing different perspectives to emerge from diverse experiences.

We were nervous about using just the MSC questions in our interviews and focus groups. We didn’t think we’d get data on certain outcomes (positive or negative) because the MSC questions were so broad, so we also asked our respondents more specific questions. We found people’s responses to the more specific questions were generally duplicative of what they had shared to the MSC questions, and sometimes less rich.

For example, in response to the MSC questions, a Francophone African Learning Circles participant shared:

Participating in this [Learning Circles] session allowed me to learn new knowledge and it really had an impact in my life because first of all, it allowed us to know new experiences of our peers who are in other countries. through this opportunity, we understood that there are practices that exist in other countries but which are not in our country and why not duplicate, these beautiful experiences in our country too? … The second thingthis session has allowed us to build this link and these relationships between us [participants from other African countries]. And the third thing is that it allowed us to learn how to manage our knowledge. Because it must be said that we do a lot on the field: we do activities, we do initiatives. But the sustainability and especially the documentation of these initiatives—these sessions allowed us, in any case, [to learn] how to document and how to manage the body of knowledge that we can gain from the experiences

When asked more specifically to explain whether the Learning Circles format was useful in generating and sharing lessons about what works and what não no planejamento familiar programas, the same participant referred back to learning from other participants’ experiences as an indication of the usefulness of the Learning Circles format:  

I would have to say that the format was quite useful in terms of managing, sharing lessons and programs, about what works and what doesn’t. it allowed us to see, for example, in Guinea here, what are the elements or what are the initiatives that work in the area of FP? And what não work? We shared and other young people in other countries also compartilhadod their experiences.

2. The MSC questions also help uncover unexpected outcomes.  

Structured forms of M&E are useful for linear and clear causal pathways. But in complex environments or interventions, you need something with more flexibility. This flexibility helps you discover things that you didn’t necessarily design for. Once you discover these aspects, you can factor them in your design to strengthen those components even further or address problematic areas.

For example, in our Learning Circles evaluation, we were pleasantly surprised to learn from several participants that their participation in Learning Circles contributed to their career advancement, something we hadn’t intentionally factored into the initial design of the program: 

I can see it [Learning Circles] as really good for my career to a point where right now I think I’m moving to even more senior positions just because of the knowledge. – Participant from Anglophone Africa

“… I was also a part of this whole regional network and the impact that [my participation in Learning Circles] has created was that earlier I was only looking after the India-level network, but after trading the insights [from Learning Circles] that I provided to the organizational senior management, they also asked me to lead this Southeast Asia regional network as well. – Participant from Asia

3. It eus helpful to pair the rich qualitative data from MSC with quantitative data.  

We have found that the MSC quotes (and qualitative data in general) provide such rich descriptions of people’s experiences and a level of understanding of the outcomes and impact of the work you’re evaluating that you just can’t get with quantitative data. It’s also nice, however, to pair the qualitative data with numbers and statistics, when possible, to demonstrate the extent to which those experiences and outcomes may be representative of a larger group of people.

For example, in our Learning Circles evaluation, we paired a more traditional survey of participants with MSC-centered interviews, finding that a majority 75% of survey respondents said they applied knowledge gained from Learning Circles to inform program design, improvements, or policy. Our MSC interviews painted a picture of what knowledge adaptation and use looked like:  

… as a reproductive health program implementer in Uganda, I learned how we could make use of the different networks to do more advocacy … I was learning from the Kenya participants how they were doing it on their side, especially using social media, how we could identify key actors for family planning that we could also make use of in Uganda.

…. It’s really helped me change my real thinking, especially in the programming on gender-based violence and I could now better grasp what to do in regard to gender-based violence, even develop funding proposals, build a stronger case within my organization for gender-based violence [programming].

4. You still need experienced researchers to collect and analyze MSC data. 

Although people generally understood the MSC questions and responded accordingly, it’s still important to have experienced interviewers who can probe when needed. Once the data are collected, you also need people with some experience in qualitative data analysis to analyze and synthesize the data. We used ATLAS.ti to code and analyze the data, but free and simpler software options are also available such QDA Miner Lite, Taguette, or even Google Docs/Sheets or Microsoft Word/Excel.

5. MSC is a great method to contribute to evaluation of KM interventions!

Our team had experience with using MSC for ongoing monitoring of FP/RH program interventions under The Challenge Initiative. We thought it could also be useful for M&E of KM interventions, but we didn’t have any experience or evidence to back that hunch up. After using MSC in four evaluations of different KM interventions, we now feel confident to encourage other KM practitioners to give MSC a try!  

Ruwaida Salem

Diretor Sênior de Programas, Johns Hopkins Center for Communication Programs

Ruwaida Salem, Diretora Sênior de Programas do Johns Hopkins Center for Communication Programs, tem quase 20 anos de experiência no campo da saúde global. Como líder de equipe para soluções de conhecimento e autora principal de Building Better Programs: A Step-by-Step Guide to Using Knowledge Management in Global Health, ela projeta, implementa e gerencia programas de gerenciamento de conhecimento para melhorar o acesso e o uso de informações críticas de saúde entre profissionais de saúde em todo o mundo. Ela possui mestrado em saúde pública pela Johns Hopkins Bloomberg School of Public Health, bacharelado em dietética pela University of Akron e certificado de pós-graduação em design de experiência do usuário pela Kent State University.