Building meaningful frameworks for measuring diverse projects contributing to common outcomes requires a clarity and simplicity that is truly challenging to achieve. This has been my focus this year for a few programs at the provincial scale.
Communities in rural and urban areas have distinct challenges, and there are different cultures in different regions. This influences the nature of issues being addressed, as well as the kinds of approaches needed to successfully build community and effect change. “Shared measurement” rolls off the tongue, but finding ways to structure it among diverse organizations in a wide range of communities takes a thoughtful approach.
I’ve found a few things that help:
1. Aligning common measures at the right level. Common measures can’t be so specific that there can be no comparison, nor so broad that there is no longer any meaning.
2. Focusing on a small number of common quantitative metrics that are feasible and meaningful for all parties to collect. These may look more like outputs than outcomes, but they can be rolled up better than the most perfect metrics that only 20% can reasonably collect.
3. Building in a process for organizations to gather data about their specificity. What are they experiencing in their community? What are the particular approaches they take, and in what context? This starts to give meaning to the metrics- allowing us to see how the overall outcomes are working, and why.
This is all easier said than done, but I am heartened but the responses from groups once they see their results presented as shared outcomes- it starts to feel powerful. Qualitative data and stories that illustrate local experiences in a particular context add depth of meaning. Taken as a whole, this can be a valuable way of demonstrating that our differences are what makes us effective in a variety of settings, and contributing to common outcomes makes us effective at scale.
I’ve just completed an intensive series of reports to close multi-year evaluations with several clients. The experience has led me to reflect on my team members and how grateful I am to work with so many amazing people.
As an external evaluator I am at times seeing things from a different perspective than the Executive Directors, Project Managers and administrative staff that I work with. This is my advantage and my challenge; I have a fresh way of seeing things, but I am also one step removed from the day to day actions and changes that impact program outcomes. Sometimes the data I see is out of date and I don’t realize it, and sometimes the data I happen to have doesn’t tell the full story.
Working in collaboration with my clients, I am able to bridge these gaps and get a truer picture of what happened, and why. I am grateful to have an excellent Research Assistant who also helps me to prepare the data we do have and identify big-picture questions that we need to take back to our clients.
As we close this period of reflection I am grateful to the wonderful people I work with, and wish everyone continued success in improving healthcare, building collaboration and contributing to system-wide changes that make all of our lives better.
I was reviewing my achievements and learnings for 2015: working with great new clients, building even stronger relationships with existing clients, taking on new leadership roles, letting go of volunteer commitments I could no longer maintain, communicating using media that is new to me, presenting and learning from others at conferences and events, and building the facilitation and training side of my work.
I was preparing to make some resolutions for the new year, when an email from the Chopra Lifestyle Centre helped me reorient my thoughts. The article suggested that people are more likely to achieve goals than to stick to resolutions. In my own work I would have suggested the same thing. I had forgotten this simple yet transformative thing when considering my year ahead!
Goals help to orient us. They keep us “on track” exactly because if we know our goals, we can always figure out our direction no matter where we are on the path, or how we’ve gotten lost along the way. Goals give us a tangible “what” to work towards, and leave it up to our creativity to bring our goals into reality.
Resolutions tend to be more about “how” we do things, which is harder to adapt, less motivational, and not as easy to visualize. With that in mind, I have some inspiring goals I’ll be working on in 2016.
Thanks to everyone who helped me learn and grow in 2015. I wish you all the best in 2016.
How Evaluation Saved my Family Christmas
A few weeks ago, my dad sat me down and started asking about whether we’d be having stockings for Christmas, and who would do it, and suggesting that maybe we shouldn’t have stockings anymore. My sister called me the same night to talk about whether we should have turnips in the mashed potatoes or not. I could feel the start of it…Christmas drama.
Instead of engaging in the usual backroom negotiations and lobbying for my preferences, I used appreciative inquiry to think about how I could make this the best Christmas ever. What do we all really want? I know that deep down we all want to have a joyful holiday that brings us together to eat good food and enjoy one another’s company. The worry about what kind of gifts to get or who should cook the gravy using what technique was distracting us from what is truly important.
I spent some time thinking about the tools I use in my daily work, and I decided to try a Christmas survey. What better way to find out what everyone wants? To be successful, this would need to incorporate some important principles:
1. Full engagement- Everyone would have to participate to be heard, and the experience would have to be fun to inspire their participation
2. Transparency- The raw results would have to be shared to build trust
3. Simplicity- The survey would have be quick and simple
4. Actionable recommendations- I’d have to translate the raw data into actionable items that everyone could understand and participate in
I did my best to make the invitation to the survey fun, and to make the survey itself an enjoyable experience. I incorporated the following elements:
· Casual, fun language in the survey invite
· Explicit statement of goal in the invite: “We want to make this the best Christmas ever”
· Everyone is asked to click on “I will help make this the best, most fun Christmas ever” at the end of the survey
· The survey is highly visual with fun and symbolic Christmas images like trees and ornaments
· The survey questions are short and simple
The full results are presented so that everyone can see the raw data. On top of the raw results, I help with interpretation by clearly showing which choices were most and least popular. This was important in an environment where one person in a family might speak for another. For example, in one of the comments, someone said: “Thanks for organizing! Absolutely no purchased items for me or anyone in our family :-) Can't wait for cookie day - I vote sugar cookies”. Needless to say, the other family members actually voted that they wanted stockings and gifts. The survey allowed each individual to express their wishes without going through a family representative who might alter their response. Transparency was key to establishing trust in the process and further supporting buy-in.
The survey has only two questions:
· How do you feel about the following activities?
· What do you want to eat?
The scales were simple and fun:
· Activities: Love it! * Um…maybe. * Kill me now.
· Food: Yum J * If I must… * Gross.
I incorporated images into the survey and the “report” to keep it entertaining.
The results were important to share, but in order for people to take the data and act on it, the recommendations needed to be clear. In the case of Christmas, this took the form of a list of activities, menus for dinner and dessert, and a budget for gifts and stockings.
Several people made new suggestions in the activities section, so we had a one-question ranking survey sent out to follow up, and we are now starting a new family tradition- we’ll be watching James Bond together.
This is the Best Christmas ever!
I’m happy to say that I’m more convinced than ever that the tools of evaluation can help us to encourage engagement, build relationships, and create a shared sense of purpose and agreed process. We’ll be having English trifle for dessert, which is my favourite, and everyone has been completely on-task and excited about Christmas.
Last week the Canadian Evaluation Society BC & Yukon Chapter held a conference on the theme of Collaboration, Contribution and Collective Impact.
This sold-out event reminded me how much we want to collaborate in our own field. We work together with clients, partners and stakeholders all the time, incorporating new practices and innovating to build relationships and improve our outcomes.
Paul Kishchuk talked about wisdom outcomes being integral to making a difference as evaluators. We learned new skills like graphic facilitation from @jackiecamsden, designed to help us collaborate. I couldn’t attend the Evaluation Therapy session, but I heard from many people how much they enjoyed the chance to intervene in common situations and act out their own recommendations, quite literally. Incorporating a new element like theatre adds to our toolbox in unexpected ways.
Funders talked frankly about how hard it is to standardize metrics, because to respect context naturally leads to adaptation and differentiation, but to tell a bigger story, we need to define common measures.
What did I take away from the experience?
1. We learn and grow better together
2. Collaboration is complex and we benefit by being creative, observant and adaptable to make sense of it in evaluation
3. Collaboration is a confluence of individuals, mechanisms and systems, and we need to approach it with this understanding
Thanks to the many excellent presenters and to the attendees who provoked meaningful conversations. I look forward to next year!
TedxMontrealWomen took place this weekend, and I had the good fortune of attending with two brilliant women. We listened and reflected throughout the day on the presentations organized around the theme of daring greatly. It was both a personal and professional challenge to get to know ourselves better and bring more of our best selves into the work we do.
For those of us who work with complex issues and challenges that often seem intractable, the inspiring words from women in science, technology, business, social development and art served as a moving reminder that we need to tap into our passion to be at our most powerful.
Skawennati is an artist and independent curator creating future images of thriving Aboriginal communities using digital media. She works with youth to express their culture and personal journeys through video games and created an Aboriginally-determined territory in cyberspace called CyberPoWow. Skawennati and the other speakers have found their purpose and are fully engaged with making the world a better place by giving expression to their powerful dynamism and unique talents.
We left the event recharged and ready to create stronger relationships and build an ever more mindful practice.