Building meaningful frameworks for measuring diverse projects contributing to common outcomes requires a clarity and simplicity that is truly challenging to achieve. This has been my focus this year for a few programs at the provincial scale.
Communities in rural and urban areas have distinct challenges, and there are different cultures in different regions. This influences the nature of issues being addressed, as well as the kinds of approaches needed to successfully build community and effect change. “Shared measurement” rolls off the tongue, but finding ways to structure it among diverse organizations in a wide range of communities takes a thoughtful approach.
I’ve found a few things that help:
1. Aligning common measures at the right level. Common measures can’t be so specific that there can be no comparison, nor so broad that there is no longer any meaning.
2. Focusing on a small number of common quantitative metrics that are feasible and meaningful for all parties to collect. These may look more like outputs than outcomes, but they can be rolled up better than the most perfect metrics that only 20% can reasonably collect.
3. Building in a process for organizations to gather data about their specificity. What are they experiencing in their community? What are the particular approaches they take, and in what context? This starts to give meaning to the metrics- allowing us to see how the overall outcomes are working, and why.
This is all easier said than done, but I am heartened but the responses from groups once they see their results presented as shared outcomes- it starts to feel powerful. Qualitative data and stories that illustrate local experiences in a particular context add depth of meaning. Taken as a whole, this can be a valuable way of demonstrating that our differences are what makes us effective in a variety of settings, and contributing to common outcomes makes us effective at scale.
I’ve been hearing a lot about failure lately: “I don’t want to admit that I have failed”, “Let’s re-frame these results so it doesn’t seem like we failed”. Most of these comments have come from what I consider to be successful people, running successful programs and businesses. Why is this conversation coming up about failure?
People are trying new things. We live in a complex world that requires innovation to adapt and improve our approaches. It’s courageous to try something new, especially when we’ve received funding and feel like we’re accountable for results. Innovation requires a certain kind of risk, because we are doing something we aren’t actually sure will work. If we already new what would work, and if our context never changed, we could confidently continue what we are already going.
Why is innovation so important?
We live in a changing world with new challenges, complexity, and ever-shifting influences. Innovation allows us to imagine new solutions. This requires a lot of leadership and willingness to learn.
Why is innovation so hard?
We may understand the need to change, simply because we know our approach needs to improve or because we can envision a better way. Knowing change is needed, however, isn’t the same as knowing what to do about it. Until we try something, we don’t know if it will work: we are operating in a constellation of needs, stakeholders, funding, relationships and other pressures that will impact the best-laid plans. We can’t know how something will work until we try it. Trying something new means exposing ourselves to a world of unknowns. Our challenge is to act with our best information, intentions and approach. Then we need to reflect, because there will be nuances to our experience that can teach us a great deal about how we might move forward effectively. Listening carefully is the key to our ability to learn and improve, bringing a clear understanding of what didn’t work forward, just as much as what did work.
I am much more worried about failure of imagination, failure to act, and failure to reflect, than I am about hearing “this completely failed, let’s learn from it.” The very reason that we tried something new was to see if it worked. If it didn’t, let’s not repeat it, and let’s understand why.
How do we as evaluators create a safe space to talk about failure? It’s a conversation that helps us evolve and grow as a profession. It’s key to supporting our clients to benefit from their experience; saying something failed shouldn’t be about admitting weakness, it should be about celebrating a new approach and building collective wisdom around how it worked, what didn’t work, and what lessons can be learned and shared.
What can we do to support talking about failure?
· Create a safe space for the conversation
· Make it clear from the beginning that learning is the goal
· Focus on the experiment, not the success or failure of the organization carrying it out
If we knew exactly how to do something, it wouldn’t be innovation. We can create the opportunity to build on our failures through innovation, action, and reflection.
I’ve just completed an intensive series of reports to close multi-year evaluations with several clients. The experience has led me to reflect on my team members and how grateful I am to work with so many amazing people.
As an external evaluator I am at times seeing things from a different perspective than the Executive Directors, Project Managers and administrative staff that I work with. This is my advantage and my challenge; I have a fresh way of seeing things, but I am also one step removed from the day to day actions and changes that impact program outcomes. Sometimes the data I see is out of date and I don’t realize it, and sometimes the data I happen to have doesn’t tell the full story.
Working in collaboration with my clients, I am able to bridge these gaps and get a truer picture of what happened, and why. I am grateful to have an excellent Research Assistant who also helps me to prepare the data we do have and identify big-picture questions that we need to take back to our clients.
As we close this period of reflection I am grateful to the wonderful people I work with, and wish everyone continued success in improving healthcare, building collaboration and contributing to system-wide changes that make all of our lives better.
I was reviewing my achievements and learnings for 2015: working with great new clients, building even stronger relationships with existing clients, taking on new leadership roles, letting go of volunteer commitments I could no longer maintain, communicating using media that is new to me, presenting and learning from others at conferences and events, and building the facilitation and training side of my work.
I was preparing to make some resolutions for the new year, when an email from the Chopra Lifestyle Centre helped me reorient my thoughts. The article suggested that people are more likely to achieve goals than to stick to resolutions. In my own work I would have suggested the same thing. I had forgotten this simple yet transformative thing when considering my year ahead!
Goals help to orient us. They keep us “on track” exactly because if we know our goals, we can always figure out our direction no matter where we are on the path, or how we’ve gotten lost along the way. Goals give us a tangible “what” to work towards, and leave it up to our creativity to bring our goals into reality.
Resolutions tend to be more about “how” we do things, which is harder to adapt, less motivational, and not as easy to visualize. With that in mind, I have some inspiring goals I’ll be working on in 2016.
Thanks to everyone who helped me learn and grow in 2015. I wish you all the best in 2016.
Last week the Canadian Evaluation Society BC & Yukon Chapter held a conference on the theme of Collaboration, Contribution and Collective Impact.
This sold-out event reminded me how much we want to collaborate in our own field. We work together with clients, partners and stakeholders all the time, incorporating new practices and innovating to build relationships and improve our outcomes.
Paul Kishchuk talked about wisdom outcomes being integral to making a difference as evaluators. We learned new skills like graphic facilitation from @jackiecamsden, designed to help us collaborate. I couldn’t attend the Evaluation Therapy session, but I heard from many people how much they enjoyed the chance to intervene in common situations and act out their own recommendations, quite literally. Incorporating a new element like theatre adds to our toolbox in unexpected ways.
Funders talked frankly about how hard it is to standardize metrics, because to respect context naturally leads to adaptation and differentiation, but to tell a bigger story, we need to define common measures.
What did I take away from the experience?
1. We learn and grow better together
2. Collaboration is complex and we benefit by being creative, observant and adaptable to make sense of it in evaluation
3. Collaboration is a confluence of individuals, mechanisms and systems, and we need to approach it with this understanding
Thanks to the many excellent presenters and to the attendees who provoked meaningful conversations. I look forward to next year!
How do you deal with the complexity of collaborating organizations that are on different timelines, with power differentials, and varying levels of data quality? Krishna Belbase of the Evaluation Office of UNICEF introduced the Resource Pack on Joint Evaluations developed by the UN Evaluation Group, at the CES 2015 conference in Montreal. He suggested that it is structured for UN agencies, but could be adapted to suit other organizations. The Resource Pack is a rich resource not only because of the simple yet comprehensive guide it provides for evaluation, but also because of the way it details the governance structures needed to support evaluation in organizations working together on evaluation.
In today’s world many evaluations are done with some element of collaboration, and the Guidance Document and Toolkit that make up the Resource Pack can be used to help define the key functions, structures, and questions to ask when determining how to govern evaluation.
The Guidance Document helps tease out the various functions like communication, management, technical input, and logistics. The Toolkit then walks you through the steps from deciding to work together on an evaluation, preparing for the evaluation, implementing the evaluation, to utilizing the outcomes. It addresses sticky issues like readiness and buy-in, and provides advice at every stage from developing terms of reference to disseminating findings.
Do you need a steering committee, management group, reference group, stakeholder group, or advisory group? The Toolkit lays out the considerations for making important decisions about the most appropriate governance structure for your situation. Overall, the Resource Pack on Joint Evaluations is a great resource for any organization looking to support decision-makers and leaders in structuring their governance, and provides tools such as checklists, examples and good practices to evaluation practitioners.
Check out this amazing resource: Resource Pack on Joint Evaluations
I just came back from a wonderful experience at the Canadian Evaluation Society's annual conference, Evaluation for the World We Want, CES/SCÉ 2015. I was happy that my presentation on Measuring Collaboration was well-received. It turns out that I'm not the only one struggling to figure out what collaboration really means and how to identify the appropriate tools for measuring it. I've been collecting a list of tools for measuring collaboration
as I come across them. Please share your tools for measuring collaboration and I'll add them. The best thing about #EvalC2015 was how much everyone shared- because we're all in this together!
My favourite lessons from CES 2015:
1. Include legislation in the logic model to ensure it gets tracked (Nancy Carter and Robert Chatwin)
2. Government is responsible for evaluating the value of programs- did they provide a public good, as is the goal, not the dollar value (Mirianaud Oswald Agbadome)
3. Paradigm shifts happen. Evaluation can lead to systems change. 'coeur et rigeur' (Laure Waridel)