Skip to Navigation

Case Studies

It is helpful to see a finished OA in order to understand how the process of engaging in one might help an organization improve its performance. Below are a few real examples. When necessary, the name and a few details of the cited organizations in these case studies were changed to protect confidentiality; a note indicates when such modifications have been made.

In this page: Case Study #1: Evaluation of the COMPASS network Case Study #2: Joint Evaluation of the International Trade Centre Case Study #3: Institutional Assessment of Alluvial Diamond and Gold Miners' Association of Kono District (ADAGMAK) - Coordinators of miners and dealers

Case Study #1: Evaluation of the COMPASS (Community Partners with Schools) network and of its Relevance and Effectiveness

Description of the Organization:

COMPASS (Community Partners with Schools) is not an organization but a network between health and social service providers and schools in Simcoe County, north of Toronto. In Simcoe County, there are seven 'COMPASS Cluster Teams', based on geography, that meet monthly over the school year.

The purpose of this network is to link health and social service providers with schools to:

  1. facilitate the sharing of knowledge between stakeholders such as the current needs of children and services available and how to access them; and
  2. provide a mechanism for collaborative interventions for children in need.

In addition, COMPASS established a program that directly delivers select children's mental health services.

Details on the Evaluation Conducted and its Linkages with the IOA Model:

In May 2009, COMPASS requested the services of the Universalia Management Group to conduct an evaluation of its network. The purpose of this external evaluation was to:

  • measure the effectiveness and relevance of both the network capacity building of the cluster teams and the direct service delivery, although emphasis was on the former

Because efficiency was not a major thrust of the evaluation, the IOA Model, which was used to conduct this evaluation, had to be modified. Fortunately, the IOA Model is an adaptable framework that allows such flexibility. 

The tool used for this evaluation included two separate on-line surveys, one for school principals or school representatives, and one for members of the cluster teams. Numerous one-on-one interviews were also held as well as several workshops and focus groups.

The approach used to measure the performance of the direct service delivery was fairly straight forward and largely based on levels of client satisfaction with different facets of service delivery. The approach to measuring capacity-building performance of the network rested primarily on how well the cluster teams were performing against key criteria of a successful capacity building network (as defined with the client). These elements were:

  • relationship building;
  • participation;
  • knowledge transfer;
  • clear and common vision and objectives;
  • efficient processes; and
  • measurable actions or outcomes.

The surveys and interviews posed questions related to each of these themes allowing the evaluator to score the cluster teams against the key elements of successful capacity building network.  For instance, the answers relating to the theme of "efficient processes" were compiled and averaged and were then compared to the score of the "participation".  This type of analysis allowed the evaluation team to compare the cluster teams and the key criteria so as to identify the main strengths and weaknesses of the capacity building network.

Considerable additional work including evaluation of the management of the program was also done with all stakeholders to quantify and qualify other performance metrics and identify areas for improvement.

Results of the Assessment:

The results of the capacity building assessment described above consistently showed the network was effective at facilitating relationships but comparatively weak at producing measurable activities or outcomes. In fact, the ranking by criteria, as listed in the prior section, was consistent even when the response data was filtered by cluster team or by type of organization the respondent was from (school or service provider). Qualitative evidence harvested through interviews and workshops supported this observation.

In addition to a number of other important findings, the evaluation of the network's performance at capacity building indicated a critical need to improve the action/outcome orientation of the network.

The COMPASS program is currently implementing the results of the evaluation and Universalia has been asked to support the development of a best-practices tool-kit for cluster teams.

Client Testimonial:

"The consultants from Universalia met and exceeded our expectations in the program evaluation for which we engaged them. The team combined a professional, thorough approach to research and analysis with a strong participatory approach. We particularly appreciated their ability to shape suggestions from many participants into a focused plan, and their attention to program sustainability."

Learn More about this Case Study:

For more details on this case study and on Universalia activities in Ontario, feel free to contact Mr. Andrew Hamilton by phone at 416-704-9732 or by email at ahamilton@universalia.com.

The links appearing in above text and in the OA Summary of Case Study #1 below will also guide you to the tools and the model used for this evaluation.  Finally, we invite you to reflect on the question below and to share your impressions with us.

OA for Thought: "How would you modify the IOA Model to evaluate your organization or network?"  Feel free to share your thoughts on this question with us: write to us!

Summary - OA Case Study #1

  1. Country, Province: Canada, Ontario
  2. Type of organization: Network of health and social service providers and schools
  3. Size of the organization: Small (around 20 - 30 participants from schools and service providers for each cluster team)
  4. Sector of intervention: Link health and social service providers with schools to facilitate knowledge sharing and provide a mechanism for collaborative interventions for children in need
  5. Motive for conducting the evaluation: Assess the performance (in terms of relevance and effectiveness) of the direct service delivery and capacity building performance
  6. Duration of the evaluation:4  months
  7. OA Model the tool corresponds most closely to: IOA Model
  8. OA conducted:Externally

top of page ↑


 

Case Study #2: Joint Evaluation of the International Trade Centre (ITC)

Description of the Organization:

The International Trade Centre (ITC) is the technical cooperation agency of the United Nations Conference on Trade and Development (UNCTAD), and the World Trade Organization (WTO) deals with operational, enterprise-oriented aspects of trade development. While UNCTAD and WTO work principally with governments, ITC works mainly with the business community, concentrating on business aspects of trade development.

Details on the Organizational Assessment Conducted:
The evaluation was supported by a group of seven donors led by Danida.  The objectives of the assessment were to:

  • Determine ITC's results and success in fulfilling its objectives [...] by measuring output, outcome and impact to the extent possible;
  • Determine the comparative advantage of the organization within the international trade and development community, which has changed significantly, especially during the past decade; and
  • Offer recommendations on priority setting and on steps ITC should take to improve its future performance.

The evaluation was conducted externally, by a consortium of four companies, from January 2004 to October 2005.  The evaluation team consisted of 27 members who were divided into an International Evaluation Team, a Local and Regional Experts Team, an Administration Team, and a Peer Reviewers Team. 

Field and technical studies were conducted during summer 2004. Field studies were conducted in ten countries, covering all continents. The selection of countries was based on the pattern of ITC activities and projects in different regions. Technical studies were conducted on various ITC services and products including the use of e-questionnaires administered on this website. In total, 16 product areas of ITC were evaluated.

Results of the Assessment:

The set of recommendations given by the consortium is comprehensive. The ITC managed to undertake several measures to improve its performance.  The recommendations resulting from the OA, as well as the measures undertaken by the ITC to fulfill them, are listed in the following report: Evaluation Follow-Up.

Learn more about this case study:

For more details on this case study, you can visit the following website, which is entirely dedicated to the evaluation: Joint Evaluation of the International Trade Centre (ITC). The website includes all reports related to the assessment, including the Terms of Reference describing the objectives of the evaluation, the Synthesis Report of the Evaluation, and the Management Response from ITC to recommendations of the Evaluation Team. You can also click on the links appearing in the OA Summary of Case Study #2 below to consult the tool and the model this Case Study refers to.

Summary - OA Case Study #2

  1. Country: Global
  2. Type of organization: Joint technical cooperation agency of the UNCTAD and the WTO
  3. Size of the organization: Large (more than 200 employees)
  4. Sector of intervention: Help developing and transition countries to achieve sustainable human development through exports; concentrates on business aspects of trade development
  5. Motive for conducting OA: Assess the comparative advantages of the ITC, evaluate the performance of the ITC's interventions, and assess the management capacity of the ITC and the Donor-Funding Modalities, in order to develop recommendations for future management and development of the ITC
  6. Duration of the OA: Almost 2 years
  7. OA Tool used for assessment: SWOT analysis, using result-based logical framework
  8. OA Model the tool corresponds most closely to: IOA Model
  9. OA conducted: Externally
  10. New measures taken for improvement: Numerous ones, listed in the management response of the ITC to the evaluation recommendations, in March 2006, in the following report: Evaluation Follow-Up

top of page ↑


Case Study #3: Institutional Assessment of Alluvial Diamond and Gold Miners' Association of Kono District (ADAGMAK) - Coordinators of miners and dealers

Description of the Organization:

The Alluvial Diamond and Gold Miners' Association of Kono District (ADAGMAK) is a community-based organization (CBO) based in Sierra Leone.  "ADAGMAK is dedicated to supporting and advocating for the rights of diamond miners and dealers in Kono District, with a particular focus on campaigning against child mining, leading the formation of mining cooperatives, combating smuggling, coordinating affairs of legal miners and dealers and monitoring the implementation of mining legislation. It is also an important member of the Peace Diamond Alliance."

Details on the Organizational Assessment Conducted:

ADAGMAK was able to conduct an organizational assessment, with the funding of USAID, and with the facilitation of Management Systems International (MSI).  The goal of this self-assessment would be to develop a self-directed institutional development program in order to be a more effective member of the Peace Diamond Alliance. The OA was conducted internally, involving all members, using the Institutional Development Toolkit (link to this tool specifically; access allowed).  This toolkit "provides a methodology to help an organization assess its own institutional strengths and weaknesses". The assessment was completed during a short workshop, on September 10 and 14, 2004.

Participants collectively assess an organization's Vision/Oversight as well as its Management Resources, Human Resources, Financial Resources, and External Resources. Based on these results, participants decide which areas within the organization will be targeted as priorities in an institutional strengthening plan, and then develop an organizational improvement plan. The scoring can serve as a baseline against which the success of future institutional strengthening efforts can be measured.

Results of the Assessment:

Following the assessment, ADAGMAK decided that the subsequent improvements were needed:

  • Improve capacity to attract and maintain a donor (funding organization).
  • Overall Management Systems Improvement
  • Identify and implement a formal training program that focuses on core skills - management, accounting, computing and fund raising - to enhance staff capacity to perform their tasks more effectively.
  • Infrastructure Development

A strategic plan was developed to implement these changes over the year to come. The fact that this CBO has limited resources challenges the achievement of ADAGMAK's new objectives.  However, the group has high motivation and commitment, so it should be able to attain most targets.

Learn more about this case study:

You can first click on the links appearing in the OA Summary of Case Study #3 below to consult the tool and the model this Case Study refers to.  You can also read the full assessment report, which is very clear and concise: Institutional Assessment of ADAGMAK - Full Report.

Summary - OA Case Study #3

  1. Country, District: Sierra Leone, Kono District
  2. Type of organization: Community-Based organization (CBO)
  3. Size of the organization: Small (less than 50 employees)
  4. Sector of intervention: Support and advocacy for rights of diamond miners and dealers
  5. Motive for conducting OA: Develop a self-directed institutional development program so to be a more effective member of the Peace Diamond Alliance
  6. Duration of the OA: 2 days
  7. OA Tool used for assessment: Institutional Development Toolkit, developed by Mark Renzi, from Management Systems International's (MSI)
  8. OA Model the tool corresponds most closely to: IOA Model
  9. OA conducted: Internally
  10. Use of Facilitator: Yes
  11. New measures taken for improvement: Develop a workplan to improve capacity to attract and maintain donor (funding organization); overall management systems improvement; identify and implement a formal training program that focuses on core skills - management, accounting, computing, and fundraising - to enhance staff capacity to perform their tasks more effectively; and infrastructure development.

top of page ↑