OPEN TOOLKIT
The main objectives of the Open Toolkit are:
- to inform Toolkit users of what the MYSTIC programme is about and show them which tools and practices it can provide to help design a successful intervention in the youth field
- to show users how to access the programme tools and practices
- to provide users with Guidelines, advice and tools on how to adapt the programme, and its tools and practices, to suit their own context and needs.
The MYSTIC Open Toolkit gathers guidelines, procedures, tools, and best practice examples to support the replicability of the programme. It serves as a user manual for anyone interested in implementing effective youth interventions.
- Policy makers interested in innovative youth policies
- Public sector agencies at regional and local levels
- NGOs and associations working with marginalised youth
- Professional and volunteer youth workers
- Young people eager to drive social change
The Toolkit consists of two inter-connected components:
- The MYSTIC Handbook – a step-by step guide to what MYSTIC does and how it can be adapted to deliver youth-led interventions (this document)
- The MYSTIC Resource Catalogue – a set of online resources organised in the same way as the Handbook that provides examples of how to apply the Handbook. material to help programme users apply what’s in the Handbook in practice.
Please refer to the below sections of the Open Toolkit Resources




Further Reading / Extras
Resources: Manuals and Guidelines Access downloadable documents outlining the operational phases of the MYSTIC programme.
Best Practices: Collection of case studies and successful intervention testimonials.
Digital Tools: Catalogue of online resources and interactive tools to facilitate project design and management.
I. DISCOVER
- Familiarisation with the MYSTIC programme: in-depth exploration of available tools and services.
- User needs analysis: assessment of the specific needs of the target youth group.
Primary Task of this Step
The Primary Task of Step 1 is to familiarize yourself with the MYSTIC programme and the tools and services it has to offer, so you can then make decisions about how you need to adapt them to suit your own intervention.
Guiding Principles
● Make sure you and relevant people in your organization familiarize yourselves with the MYSTIC programme and the tools and services it can provide
● Understand that the programme is designed to be flexible - its methodology and tools are adaptable to suit the needs of different kinds of organisations and young people
● Watch the MYSTIC videos provided on the MYSTIC website so you know how to run the different programmes
● Take a tour of the programmes and explore how they works
- Read the MYSTIC Programme Tour Guide
- Download and read the MYSTIC User Manuals
- Take an online tour of the MYSTIC website
- List the tools and services you need and what needs to be adapted
MYSTIC is built around three interlocking core pillars – the MYSTIC ‘trilogy’ – which constitute its key outputs and tools. These are shown in the illustration below.
Primary Task of this Step
The Primary Task of Step 1.2 is to identify the specific needs of your target group so as to better develop your intervention. This task aims to carry out a user needs assessment.
Guiding Principles
▪ Be clear who your target users are for the intervention and what their needs are
▪ Compare your user needs against what the MYSTIC programme can provide
▪ Develop a vision for the intervention that is user-led
▪ Ensure the intervention is co-designed in collaboration with your target group
- Produce a categorisation of the intervention user groups
- Produce a needs analysis for these groups
- List the MYSTIC tools and services the groups need and what needs to be adapted



II. DEFINE
- Developing the Theory of Change: defining objectives and expected outcomes.
- Identifying stakeholders: mapping key players and their roles.
Primary Task of this Step
The Primary Task of Step 3 is to define the intervention ‘journey’ of your intervention – from the challenge it is presented with at the start of the journey to where it hopes to be at the end.
Guiding Principles
- Be clear who your target users are for your programme and what their needs are
- Compare your user needs against what the programme can provide
- Develop a vision for the programme that is user-led
- Ensure the program is co-designed in collaboration with your target group
- Define the problem you want to solve
- Define the changes you expect after your intervention is completed
- Specify the resources you have to make the change happen
- Identify the activities that need to be carried out
- Define what these activities produce – the outputs
- Identify the expected changes that occur when these outputs are used
- Define the things that need to be on place for these to happen
Theory of Change helps you define the intervention ‘journey’ – from the challenge it is presented with at the start of the journey to where it hopes to be at the end.
Connecting the presenting challenge to the journey’s end- the impacts and changes the programme hopes to make to the existing problem - are:
inputs (the resources available to deliver the programme);
activities (the actions carried out by the programme);
outputs (things produced by these activities);
immediate outcomes, (changes in awareness and knowledge);
intermediate outcomes (changes in behaviour and structures).
Underlying this ‘change journey’ are ‘theories’ (assumptions and hypotheses), for example a theory of what is causing the ‘presenting problem’; a theory of what is needed to bring about the desired solution; assumptions that if we take Action ‘X’, this will produce Output ‘Y’, which will then lead to Outcome ‘Z’.
1. Start with defining the problem you want to change
2. Describe the change you want to see to the problem – the programme impacts
3. Specify the outcomes that lead to these impacts
4-5 Work backwards to specify the resources you have, the activities that use these resources, the outputs these produce
6. Specify the causal links in each step
7. Clarify the assumptions
The illustration below shows how to produce a Theory of Change for a programme using a simple template filled in with ‘post-it’ notes.
Another way of presenting a theory of change approach is shown in the diagram below.
This focuses on the personal and organisational actions needed to deliver change.
Primary Task
The primary task of this Step is to identify other key stakeholders – in addition to young people– whose involvement will be needed to set up and run a successful programme and to understand how they can best contribute to its implementation.
Guiding Principles
● Ensure everyone with a ‘stake’ in the training programme has a voice in its design and implementation.
● Understand each stakeholder groups’ needs, expectations and what they can bring to the table.
● Make sure the programme is designed to reflect the profiles of the stakeholders that need to be involved and maximize their strengths and the resources (digital competences and soft skills) they can bring.
- Identify and map the key stakeholder groups
- Set up a stakeholder database including key stakeholder contact details
- Produce a categorisation of the stakeholder groups
- Analyse and categorize the members of the database to in terms of stakeholder type; interests; degree of influence and attitude to the programme
- Create a visual representation of the stakeholder population and their characteristics
Your intervention needs the active involvement of different people and groups to achieve its social and educational impact and, to do so, it needs different stakeholders to play different roles.
We can identify four important reasons to include Stakeholders in an intervention:
1. Increase the quality and quantity of input and reduce the chances of irrelevance.
2. Encourage ownership and involvement with the project goals.
3. Increase the chances of success.
4. Widespread consultation improves relationships and increases stakeholder self-esteem. It also reduces the chances of misinformation and complaints of lack of transparency.
Generally, when thinking about stakeholders in the context of a youth intervention people think in terms of young people and youth services. However, it is important to include other stakeholders as well: professional organizations, potential partners, competitors, community leaders, community groups, employees who take young people for work experience, volunteers, etc.
Stakeholders can be grouped into four broad types:
● Sponsors or project owners are often those who initiate change by mobilizing the resources needed and charging people with the responsibility for getting it done.
● Change Teams are those charged with the responsibility for executing the change and ensuring it happens.
● Reference Groups include those people that change teams must refer to in order to arrive at the right solution. They ensure that the change will work.
● Users are a broad group of people who benefit from the change solution.
(Note: The Reference Group and some of the Change Team may also be classed as Users).
A Stakeholder Map - or actor network map - is a tool to create an overview of all stakeholders who may have an interest or a role to play in a programme.
Examples cover:
● potential partners who could provide resources – e.g. premises to host the programme (platform, budgets, etc.)
● networks of organisations/people who could help raise awareness
● potential funders This tool helps identify who these stakeholders are; what resources they could bring to the programme and the relationships between them.
Stakeholder maps can be produced in a number of ways, but the most often used are either a Stakeholder Table or Network Map.
The Stakeholder Table shows:
● Names of key potential stakeholders in the programme
● Type of stakeholder – e.g. NGO/civil society; civic authority; Business
● Assets – the resources they could bring to the programme – e.g. funding; learning content; extra-curricular activities
● Role – the potential role they could play in the programme – e.g. contribution to the Community Labs; funding provider
The Network Map shows a visual representation of the location in which the programme will be delivered, with the key stakeholders situated within it, in approximate distance from each other. Each type of stakeholder can be represented by a different colour and/or symbol. Lines show how these different stakeholders are connected.
The next step after stakeholder mapping is to assess the relative importance and influence the key stakeholder groups are likely to have on your programme. This can help to prioritise the actions that need to be taken to get them involved. You can do this using a different stakeholder map to reflect how the different stakeholder categories are positioned with regard to two indicators:
- interest – representing an estimate of the interest each group is likely to have in the programme, for example as users, content providers, funders and supporters –
- influence - representing an estimate of the degree of influence (or ‘power’) each group is likely to exert over the programme development and future evolution.
These maps can be based on data collected through things like stakeholder surveys, key informant interviews and literature reviews.
The X axis represents the spectrum of dispositions toward your change project; from Against at one extreme – to for at the other.
The Y axis represents the spectrum of involvement from high at the top to none at the bottom.
The Y axis intercepts at the mid-point of the X axis. This represents a position on the X axis equivalent to a neutral disposition – neither for, nor against, the change.
The thickness of line represents the strength of relationship.
The degree to which each stakeholder can influence the change is reflected in the size of the circle used.



III. DEVELOP
- Designing the intervention: defining strategies and concrete actions.
- Implementation planning: developing a detailed operational plan.
The Primary Task of Step 5 is to use the results of Step 1 – ‘Familiarisation with MYSTIC’ – and Step 3 – Developing a Theory of Change - to adapt the MYSTIC Programme framework and tools to your own needs and the needs of your users.
In particular this Task aims to convert the overall programme vision and Theory of Change developed in Step 1 into your own specific programme design and implementation plan.
▪ The programme should be user-driven and co-designed in collaboration with participating young people
▪ The programme should clearly incorporate the specified the needs of the user target groups
▪ The programme should reflect the ‘lived experience’ of young people in the locations in which it is implemented and focus on developing the social innovation skills young people need to ‘fix what’s broken’ in their communities
▪ The programme should use a ‘blended learning’ approach that combines face-to-face learning with on-line learning and incorporate a programme of action research to enable participating young people to apply the social innovation skills they have learned in real projects that fix problems in their communities
▪ The programme should include developmental activities that take young people out of their normal routine and allow them to expand their experience and horizons – for example creative workshops, outdoor activities
▪ The programme should be designed to reflect the profiles of the stakeholders that need to be involved and maximize their strengths and the resources they can bring to the table
- Take into account who the stakeholder is – Step 4
- Apply the results of the user needs analysis – Step 2
- Use the design thinking model to develop an intervention process
- Use your theory of change to storyboard the intervention
- Review the storyboard in collaboration with users and stakeholders
Design Thinking
Design thinking applies a five-stage process to develop solutions to a ‘presenting problem’ in 'human-centric' ways, by focusing on what’s most important from the perspective of 'users' and by applying 'out of the box' and 'disruptive' ideas to address the presenting problem.
● Empathise - this involves gaining an 'empathetic' understanding of the presenting problem, through consulting experts, users and stakeholders, with the emphasis on immersion in the physical environment to gain a deeper personal understanding of the issues that affect young people
● Define - this involves synthesising the information gathered to define the problem statement in a ' human-centred manner' –e.g. defining ‘what skills gaps do young people face and how can they be mitigated?’
● Ideate - this involves 'thinking outside the box' to identify new solutions to the problem statement created in the preceding stage, and looking for alternative ways of viewing the problem – this would involve young people collaborating with other key stakeholders to think of creative ways to solve scenarios through delivering your intervention
● Prototype - this involves creating a mock-up of the proposed solution to the problem, which can then be investigated by sharing with users and stakeholders
● Test - this involves validating the prototype to assess its potential effectiveness, usability and benefits.
Mind Map
Mind – or Concept – mapping is a graphical technique aimed at illustrating how the design and implementation plan of a programme works through showing the relationships between concepts, actors and activities. Most mind mapping approaches start with the 'problem statement' at the centre of the map. The programme design team then write ideas/solutions to the problem around this central statement, concentrating on 'thinking outside the box' to identify new solutions to the problem statement, and looking for alternative ways of viewing the problem. The ideas/solutions are then connected together using lines/curves. There are a number of software programmes available to do this – some open source (e.g. bubbl.us; www.mindmeister.com; https://coggle.it/)
The MindMap typically starts at the centre with the problem or solution (programme). Brainstorming the problem then reveals the things that need to go into the design and implementation of the programme – like products, benefits, resources – rather like in the Theory of Change.
The mindmap then shows the interconnections between these things
Storyboard/Journey Mapping
Storyboards represent the programme ‘journey’ – as described conceptually in the Theory of Change outlined above in Step 3 – as a series of key actions the programme participant takes as they progress through the programme. They help to customize the overall process of the programme to the individual needs of participants/users. You could develop different storyboards for each of the ‘Personas’ developed (see Step 2) so you have a clear visual picture of how different types of user progress through the programme.
The graphic shows a ‘storyboard’ illustrating a Community Lab in South London.
It uses a design thinking approach to show how the Lab works to fix a problem – a vandalized football pitch.
Journey Mapping is a more detailed application of the storyboard approach.
The map models the ‘user experience’ of a potential programme participant so the programme design can be customized to represent a step by step model of how different types of user experience the programme.
A journey map represents a sequence of events, the interaction between the user and the programme, the user’s mood in each of the events delivered by the programme and the ‘touchpoints’ – the moments or spaces in which the user and programme interact - that support the interaction between the user and the services provided by the programme.
This step-by-step description is based on the user’s point of view.
A journey map is a powerful tool for visualising the user experience. It helps the programme designer to understand the context of users, to identify possible gaps in the services the programme intends to provide, and a clear perspective on what potential programme users are looking for and what they want to achieve.
Source: DesignscapesToolkit
The customer journey can be used in developing a new or adapting an existing programme to ensure that different user needs are built into the programme from a user perspective and identify possible opportunities for innovation of the programme. In the example shown above, the journey map represents the student journey through a semester of a Masters course, including indications about the timing of each phase, the milestones and the characteristics of the activities.
Co-creation workshop
In Step 2 above we outlined how co-creation workshops can be used to involve the programme users and other stakeholders directly as active collaborators in your intervention. Co-creation workshops are also useful for developing your intervention and adapting the MYSTIC framework and tools to your needs. The workshop could be used to review and if necessary, revise the programme personas, storyboard, journey map and implementation plan.
▪ Be clear about who is involved in the intervention and what their roles are
▪ Understand the different delivery options available for deliveing the intervention and their advantages and disadvantages
▪ Be aware of the obstacles you are likely to face, for example in the institutional, political and policy environment
▪ Ensure you have a good idea of the phases involved in setting up the intervention and implementing it, and the time scales involved
▪ Select the right delivery vehicle for the intervention
▪ Make sure the relevant people in your organization familiarize themselves with the intervention.
- Revisit your user needs and stakeholder analysis
- Download and read the MYSTIC Course and Game tutorials
- Experiment with and explore the MYSTIC foundation course and IRP
- Develop an implementation plan for the programme
- Develop appropriate management and monitoring systems
- Design the operational systems for running the programme
- Identify and specify appropriate roles and role specifications
Project Implementation Tools
Project implementation tools translate your project’s Theory of Change and Mind/Concept Map (Step 2.2) into an implementation plan that has a logic, a sequence of steps/activities and outputs and a timeline.
Typical tools used are:
▪ Logic Network
▪ PERT chart
▪ GANNT chart
Like a Logic Network, PERT is a method for analysing the tasks involved in completing a given project, especially the relationships between tasks and their inter-dependencies. It shows which tasks need to be done first and which tasks are dependent on others.
A Gantt chart is a project management bar chart that tracks tasks across time. It shows the phases, tasks, milestones and resources needed to deliver a programme. The tasks are set out in linear format across the programme timeframe from start to finish, with a start and end date shown for each task.
Service Walk-through
Before publicising/Launching the programme it’s worth getting a small number of users involved in the final validation of the programme by running a ‘service walk through’ with them. The service walk-through is a tool that provides programme designers with a way to understand the experience of a service from the user point of view. The technique uses the ‘journey’ through the service as a way of getting users to understand how they will experience it. You can use various ways of representing this journey. One way is using ‘lego blocks’ to show how the various components of the service fit together. Another way is to literally accompany users through a tour of the physical space in which your programme will be delivered.
In this example lego blocks are used to simulate how the programme uses physical spaces to deliver its services and how they connect with each other. Users are ‘walked through’ the service using the blocks and their observations – including potential issues raised – are recorded to feed into future service revisions.
Source: Boletsis, 2018/Designscapes Toolbox



IV. DELIVER
- Programme execution: practical implementation of activities.
- Mentoring and support: continuous support for participants.
● Make sure you and relevant people in your organization construct arenas for dialogue and mutual learning and development, engaging in a ‘co-creative’ relationship with participants
● Explore solutions to problems that emerge and are identified by the participants and relate to their lifeworld
● Develop, implement and evaluate solutions to these issues at the local level - focusing on 'what's broken and how to fix it' - aiming to make a real contribution to social change in young people's lives and communities
● Participants engage as drivers of social change, finding and co-designing solutions to problems identified by them
● Ensure that all programme content and interactions are accessible to a diverse range of participants, including those with cognitive disabilities or other particular needs ● Foster a sense of community and collaboration among participants to facilitate peer learning and support, ensuring that everyone's contributions are valued
● Adapt the delivery methods and content based on the evolving needs and feedback of participants with specific requirements
● Encourage participants to reflect on their experiences and adjust their approaches based on ongoing assessment and self-evaluation.
- Develop an inclusive programme recruitment strategy that actively seeks out participants with diverse needs and abilities
- Customise the programme registration process to gather information about any support needed
- Ensure all programme content is designed with universal design principles, making it accessible to a wide variety of participants Create a safe and supportive learning and development environment
- Provide adequate training for the programme delivery team, particularly in safeguarding and support
- Ensure participants are properly briefed and prepared for the programme
- Use an ‘action research’ methodology to enable participants to explore real problems and solutions - 'what's broken and how to fix it'
- Monitor and evaluate the programme outcomes (Step 9) Reflect on the outcomes and feed what has been learned into another cycle of action
The ‘Alchemic’ Model
MYSTIC takes what might be called an ‘alchemic’ approach to supporting young people, reflecting a process of combination, transformation and creation. It aims to create an environment in which the hidden, latent and possibly ‘illegitimate’ talents, creativity and potential of young people – particularly young people with fewer opportunities - can be surfaced, nurtured, validated and productively applied to community development and social innovation. In practice, this means that an effective programme needs to combine four things.
Critical reflection aims to encourage participants to think about, review and reflect on key questions that describe their life, their world, their situation, their behaviours, their future.
Collaborative dialogue aims on the one hand to support participating young people to exchange their perspectives and experiences with each other but more importantly aims to encourage young people and other stakeholders, like youth workers, to ‘step into each others shoes’.
Action learning applies a participatory approach to problem-solving - there is no learning without action and no action without learning. Action learning would involve activities like experiential adventures, team games, creative group work and collaborative problem-solving.
Co-creation aims to encourage participating young people to apply their talent and creativity by working with each other, participating youth workers/volunteers and the programme delivery team members to think outside the box and together design and storyboard ideas for action research projects that ‘fix what’s broken’ in their communities.
In the MYSTIC project, the alchemic model was delivered through three interconnected programme components: the MYSTIC Foundation Course; the Immersive Residential Programme – IRP – and the Community Co-Labs.
Delivery of the alchemic model through the foundation course, IRP and Community Co-Labs follows a linear progression, with each element feeding into the succeeding element. This progression broadly follows Bloom’s taxonomy of learning from ‘remembering and understanding’ through ‘applying and analysing’ to ‘evaluating and creating’. The Foundation course helps MYSTIC programme participants understand the landscape of youth inclusion and acquire the core skills to work in it; the IRP provides a space for them to apply this understanding and core skills to develop practices they can use in situations in their communities to solve problems, and the Co-Labs provide a space to apply those practices in action research projects working with specific, real problems.
Action Research Methodology
The alchemic model relies heavily on an ‘action research’ approach to delivering youth-led interventions (Kemmis & McTaggert, 1981) that aims at creating change through ‘practice’, not just theoretical research. It applies a ‘bottom-up’ approach involving target groups as ‘co-creators’ of incremental change which have real-world impact, like improving a community’s recreational infrastructure. It involves iterative cycles of problem-setting, planning, acting, observing and reflecting, as shown in the graphic below.
The graphic shows:
● Stage 1 - Problem-definition focuses on working with the young people participating in each Community Lab to co-produce ideas on problems of interest - i.e. identifying 'what's broken', using co-creation workshops.
● Stage 2 - Taking the ideas developed in Stage 1 and putting them firstly to a process through which participating young people, and stakeholders, critically review and evaluate the ideas. Each idea is voted on and the best idea is selected for further development. A dedicated workshop is designed and delivered to produce an Action Plan for putting the idea into practice - i.e. 'how to fix what's broken'.
● Stage 3 - The Action Plan is put into practice. The Community Labs integrate and co-ordinate the resources available - host infrastructure; mentors, champions, social entrepreneurs, delivery team staff, community organisations, youth services - to deliver the Action Plan.
● Stage 4 - The plan is observed and evaluated in action. At Stage 3, evaluation systems and tools are put into place to track progress on the plan - for example observation protocols, diaries, blogs. These are then applied as the action research takes place. At the end of the action research experiment, this formative evaluation data is combined with ex-post evaluation data - collected, for example, through participant surveys, focus groups and interviews to participants and stakeholders to evaluate the success and outcomes of the experiments.
● Stage 5 – In the final stage, the learning from observing and evaluating the experiments is reflected on and triggers the next cycle of the action research 'spiral'. A Community Action Learning Set can be held to discuss how to promote the sustainability of the experiment in the future. This learning and these results can be then applied to ongoing improvement of the scale up and out of the programme.
Holding Environment
A successful programme needs to create a safe space in which creativity can be supported and risks can be taken without fear of ridicule or sanction. A key component of the alchemic model therefore draws on the concept of the ‘holding environment’ developed by Donald Winnicott (1965). In individual development, the holding environment is a ‘transitional space’ – an ‘in-between’ space which allows the child to move from an ‘idealised’ state in which the child’s identity is completely merged with that of its mother to a state in which the child gains its own psychic structure and concrete sense of autonomy. In the context of youth interventions, the holding environment becomes an organisational space that is receptive to the birth of new ideas that will eventually stimulate change and growth. In MYSTIC, the Immersive Residential Programme provides a physical space in which this organisational holding environment can be set up and maintained.
This requires:
● Physical spaces and working arrangements that promote democratic collaboration between members of the group
● Space and opportunities for creative risk-taking
● Rules and procedures that ensure the safety, security and emotional well-being of participants, so that everyone has a voice, and no-one is blamed or made to feel inadequate
● Clear boundary-setting and governance arrangements.
Delivery logistical tools
Programmes don’t deliver themselves. They require well thought-out infrastructure, systems and processes that provide clear informative guidelines on objectives and timelines; ensure the health and safety of participants and provide post-participation feedback. As an example, the MYSTIC project provided the following logistical tools for participants.
● Creating a Trustworthy and Supportive Environment: Establish a secure and all-embracing learning and development space that fosters participants' comfort in sharing their unique experiences and concerns. Encourage a culture of open dialogue and mutual respect, supporting connections not only among participants but also between participants and instructors. This is especially vital for individuals with cognitive disabilities or other particular needs, as creating a safe space is essential for their active participation and engagement.
● Balancing Online, Face-to-Face and Experiential Learning: MYSTIC showed how important it is to reinforce any online training and learning delivered with face-to-face engagement, ‘adventure’ experiences and interactive problem-solving activities. Getting the balance right is not easy. It’s also important to remember that experiential and intensive face-to-face activities are resource and time-consuming, and these costs need to be bult into the programme.
● Catering to Diverse Needs: Acknowledge the diverse range of learning and developmental needs that participants, including those with cognitive disabilities or specific requirements, may bring to the programme. Design the learning and development programme with inherent flexibility, enabling participants to customize their learning and development journey according to their individual pace, preferences, and capabilities.
● Avoiding Isolation: Counter feelings of isolation by cultivating a robust sense of community. Regular virtual meetups, peer study groups, and discussion forums serve as avenues for participants to interact, exchange experiences, and seek solace. This is particularly crucial for individuals with cognitive disabilities, who might face additional challenges in seeking assistance.
● Addressing Varying Levels of Technology Literacy: Recognize that participants possess varying degrees of tech-savviness, including those with cognitive disabilities who might experience unique digital challenges. Offer comprehensive technical support that spans a broad spectrum of IT and digital issues. This assistance should encompass basic troubleshooting as well as more advanced technological hurdles.
● Continuous Support Availability: Empower participants with the knowledge of available support channels tailored to their specific needs. Clearly outline how they can access help for not only technical difficulties but also content-related inquiries and personal concerns. This targeted support approach is particularly crucial for individuals with cognitive disabilities, who might require extra guidance (See Step 8).
● Monitoring and Evaluation: Regularly oversee participants progress throughout the learning and development journey. Gather pertinent data on engagement levels, completion rates, and participant feedback, with a heightened focus on individuals with cognitive disabilities and other particular needs. This data-driven approach informs the evaluation phase, ensuring that the programme continually evolves to better serve its diverse audience.
● It is fundamental that the people providing mentoring and support to young people in the programme (e.g. youth workers/youth work volunteers/social service personnel, trainers) are selected on the basis of having high emotional intelligence skills
● They must be able to understand the concept of "multiple belonging": youth belong to different communities (geographically or interest-based) and are influenced in their interests and behaviours all of them at the same time, hence they are often reconciling different influences and inputs. The MYSTIC Community Lab is one more community of belonging
● Mentors and support workers need to understand and respond effectively to the frequently changing mood of the group and evolving group dynamics
● They need to work effectively with "natural group leaders" within the target group in order to keep motivation high
● A key task in mentoring and support is to shift the perception of the learning offer from top-down to bottom-up by tailoring activities to young people’s expressed interests and/or no interests.
● It is important to make participants the "protagonists" of their learning experience, thus addressing the tendency of marginalised young people to resist ‘top-down’ learning
● The learning and development programme needs to adapt constantly to keep pace with the changing moods and needs of participants.
- Selection of programme instructors
- Training of programme instructors
- Team building between instructor and group and within group
- Trust building between trainer and group and within group
- Introduction to the programme as a community
- Identify problem & solution through "service design" methodology
- Identify group leaders and work with them
- Constant technical support on use of programme technology
- Constant pastoral support activities
Understanding the needs of the target group
This target group is characterised by multiple and very complex needs. However, the MYSTIC pilots in five different European countries (Cyprus, Italy, Portugal, Greece, Romania) found similar needs:
Young people show resistance to any kind of intervention which is offered to them in a top-down manner.
They tend to distrust adults they are not familiar with and need time and significant aware effort to build a level of trust that makes it possible to work with them in a meaningful way
They need to be encouraged to try new activities. In general, we found that young people on the margins require three types of support:
● Learning and developmental support: providing help to enable participating young people to tailor the learning/development programme to suit their background and needs, and to develop at their own pace
● Technical support: providing help regarding specific problems with a product or service, for example supporting them to access the interactive game in MYSTIC
● Pastoral Support: providing help and support to youth as well as providing information, advice and guidance. The aim of pastoral support in the context of your programme is to identify any concerns or issues at the earliest possible opportunity and remove any barriers to learning which may be preventing participants from participating fully to proposed activities. This fundamentally means developing positive relationships with them; responding as quickly as possible to any issues and ensuring that difficulties are discussed and resolved in the best possible way.
Providing learning support
Marginalised young people often have a negative experience of structured educational settings. They don’t adapt well to ‘school-type’ learning environments. They respond to learning that excites their interest, is relevant to their lives, is flexible and adaptive and which is ‘scaffolded’ to suit their own pace and development trajectory.
This requires considerable effort to be put into learning support.
In one of MYSTIC’s original pilot Community Labs in Perugia, Italy, these principles were put into practice through a ‘Community Reporting’ programme.
The group of participating young people were difficult to work with because they didn’t show much interest in any activities and it was particularly challenging to keep them interested, engaged and motivated. It was fundamental to work on team building and creation of trust within the group and between the group and the trainers. This required much time and dedicated effort, but the effort was necessary because this work is preparatory and forms the basis of any specific activity that could be developed for them.
Following meetings based on group creation, the core of the work that was developed focused on Community Reporting. The basic Community Reporting programme was based on 5 specific areas: Identify the problem and solution through "service design" methodology; Community reporting: telling a story; Community reporting: video making/editing; Community reporting: web radio; Community reporting: organising an event. The programme illustrates how learning support needs to reflect: co-creation and the active engagement of participating young people; the use of a ‘design thinking’ approach to encourage young people to ‘think out of the box’; the use of ‘action research’ to engage young people in practical problem-solving; continual monitoring and adaptation to the group’s developmental path.
In many projects working with young people, the paradox is that project partners work with young people who are generally technology-wise and have in-depth knowledge of social media tools such as photo and video-making, as well as technology-based social media. However, research – and experience – shows that marginalised young people tend to suffer ‘dual exclusion’ – their social exclusion is amplified by digital exclusion. This requires effort in providing technical support
Providing technical support
The digital exclusion of young people is typically expressed in three main – and inter-connected – ways: access (e.g. limited access to high-speed broadband and digital technologies); usage (e.g. limited digital and media competences to use digital technologies); quality of use (e.g. limited opportunities to apply digital tools and competences in everyday life or to enhance life opportunities).
Technical support is needed mostly because of lack of access to digital devices; lack of familiarity with email; lack of interest in uploading personal content online, with a strong preference to use their own social media; preference to upload content as "stories", with short lifetime (e.g. Instagram stories which have a duration of 24 hours).
In practice, technical support is often provided in the following ways: providing advice on accessing digital tools through a helpdesk; adapting digital platforms to suit the routine digital behaviours of the target group – for example switching to WhatsApp and Snapchat as modalities to support collaboration; including training on digital and media skills.
Providing pastoral support
Providing marginalised young people with pastoral support means providing help, guidance and support in the following areas:
∙ Behaviour support/management
∙ Emotional help (including anger management and counselling)
∙ Restorative Approaches
∙ Peer mediation
∙ Motivational groups
∙ Links to specialist external agencies when needed
● Difficulty of accessing the target group. This challenge is a general problem in working with marginalised young people, because of issues around trust, credibility, past negative experiences of participating in support programmes. It is sometimes compounded by ‘operational’ challenges – for example the resistance of parents in giving participation consent; strict rules and procedures involving young people who are ‘minors’ or who are ‘looked after’ by civic agencies.
● Significant work needs to put into building credibility and trust in the programme – for example by recruiting role models and mentors who are looked up to by participating young people. This also needs to be supported by networking - strengthening the delivery organisation's network of partners at local level and reassuring them about the "safe" nature of the activities proposed by the programme.
● Difficulty in engaging the target group. It’s one thing to recruit young people, but it’s another to retain their interest, motivation and active collaboration. Strategies to achieve this include: trust and Team Building activities; shifting the programme focus from a top-down training offer to a bottom-up learning opportunity; the use of a ‘design thinking’ and ‘service design’ methodology for problem and solution identification; constant and adaptive pastoral activities; the adoption of ‘community reporting’ as an engagement approach.



V. EVALUATE
- Impact assessment: analysing the results achieved.
- Sustainability and scalability: strategies to maintain and expand interventions.
● Evaluation should be used not just as a retrospective tool to assess performance at the end of the programme, but should be embedded within the programme process from the start to support a cycle of continuous learning and improvement
● This means that evaluation should be used for four main purposes: a developmental purpose - to support the programme design and implementation plan (ex-ante evaluation); an operational purpose - to help the programme keep track of how it is progressing (on-going or ‘formative’ evaluation); a summative purpose - to help the programme measure what is has achieved (ex-post evaluation); a sustainability purpose - to help key actors in the programme learn from their experience
● There are many different methods and tools for collecting and analyzing evaluation data. Each has different purposes and different resource and skills requirements. The evaluation design and plan should take into account ‘pragmatic’ considerations: the ‘object’ of the evaluation; the purposes of the evaluation; the resources available to carry it out; who the evaluation audience is and what are their expectations; what evaluation skills are available in the programme, or can be brought in from outside; how long is the timeframe for the evaluation and what is it likely to cost
● The evaluation should not just reflect the ‘expert’ view but should take a ‘participatory’ approach - trying to ensure that the voices of different stakeholders and their perspectives are represented – particularly those who have less power and whose voices are not often heard
● This means that as far as possible evaluation data should be drawn from different sources and from different perspectives, and compared against each other, through ‘triangulation’, so that the evaluation reflects a balanced viewpoint
● Youth-led interventions reflect an attempt to address an existing social problem – that is, to create some level of social change. The focus of evaluation should therefore be on assessing whether and how this change has occurred.
● The programme works – or not - by enabling participants to make different choices, so a key objective of evaluation is to capture how and why these choices are made.
- Identify the evaluation purposes, timeframe and modes of operation
- Decide on who the audiences are and what are their expectations
- List the evaluation questions the evaluation will answer
- Decide on the methods to collect and analyse the data
- Decide on the indicators to measure results
- Work out what resources you need to do the evaluation
- Produce a plan to carry out the evaluation and assign tasks and roles
Theory of Change
In Step 3 we demonstrated how Theory of Change can help to tell the story of your programme. Theory of change is also one of the most powerful tools to evaluate the programme because:
● It shows the expected programme change journey from the challenge it is presented with at the start of the journey to where it hopes to be at the end
● It sets out the programme inputs, outputs, outcomes and impacts and the connections between them
● It specifies the hypotheses and assumptions of the programme – in particular its expected ‘causal chains’ - if we take Action ‘X’, this will produce Output ‘Y’, which will then lead to Outcome ‘Z’.
Essentially, what evaluation does is to test this Theory of Change by gathering evaluation data over the life cycle of the programme, to see whether these expected hypotheses and assumptions work, and are supported by the evidence.
Using the Theory of Change, you can:
Work out which ‘modes’ of evaluation you need to apply and when (developmental, process, summative)
Identify which activities are critical for evaluation
Work out how you will measure outputs, outcomes and impacts
Periodically use the Theory of Change to monitor how far your programme is progressing in its ‘change journey’
Review the Theory of change at programme end to assess how far the programme has progressed
Evaluation Design Template
Evaluation has four main purposes. These correspond to different evaluation ‘modes’ and need to be applied at different stages in the programme. They are: a developmental purpose - to support the programme design and implementation plan (ex-ante evaluation mode); an operational purpose - to help the programme keep track of how it is progressing (on-going or ‘formative’ evaluation mode); a summative purpose - to help the programme measure what is has achieved (ex-post evaluation mode); a sustainability purpose - to help key actors in the programme learn from their experience (learning mode). These need to shape the evaluation design.
Just as the programme being evaluated has a life cycle and progresses through different stages, so does its evaluation, and the methods and tools appropriate for each stage of the evaluation differ.
The key stages of the life cycle of an evaluation are:
Stage 1: Mapping and planning
Stage 2: Implementation
Stage 3: Reporting and dissemination
Having developed an evaluation plan, the next stage of the evaluation will inevitably focus on carrying that plan out.
The main stages involved in implementation are:
● Establishing the evaluation criteria that need to be assessed
● Deciding on what methods and techniques are to be used for data capture
● Managing and co-ordinating data collection, including analysing the results
Dissemination should not be restricted to the circulation of a final report - especially in the case of ‘developmental’ evaluations.
Different stakeholders may require different communication approaches.
These might include:
● Short summaries of the evaluation, tailored to different audiences
● Journal articles for other researchers
● Topical articles in the trade press/social media/blogs
● Workshops for specific audiences
● Feedback seminars for key decision-makers.
Developing Indicators
Measures to evaluate impact require the careful creation of indicators. There are four main types of indicator:
● Critical Success Factors (CSFs);
● Key Results Indicators (KRIs)
● Immediate and Intermediate Outcomes (IMOs) and
● Key Performance Indicators (KPIs)
CSFs are the critical areas whose success is important and also the steps taken to succeed
KRIs measure the effects of these steps at the end of the programme (impacts)
IMOs measure the outcomes along the way
KPIs make the connection between the CSF’s and the IMOs. They track the actions between the CSF’s and the IMOs and assess progress towards final results
The programme evaluation needs to combine all four elements in order to assess the success of the project - looking at the big ‘wins’ at project end; the critical success factors that are needed to make these happen and the key performance indicators that can tell us how we are progressing on the journey towards achieving the desired project results. In between we need to measure two kinds of outcomes:
● Immediate Outcomes – changes in awareness, attitudes and knowledge
● Intermediate Outcomes – changes in behaviours and structures.
Process dashboard
The Process Dashboard has four purposes:
i) to enable monitoring of programme progress set against key progress indicators, or baselines
ii) to provide a picture of where the programme is in relation to the ‘change journey’ specified in the ‘Theory of Change’ (and also to review whether the underlying assumptions and hypotheses embedded in the project ToC hold true or need revision)
iii) to feed data into the overall summative (outcomes) evaluation of the programme
iv) to stimulate review and learning as the programme develops.
The Dashboard is composed of Key Progress Monitoring Indicators – a list of baseline core outputs defined as ‘evidence of success’, that together build up a snapshot at a point in time of the extent to which the programme is meeting its planned operational objectives. The dashboard and associated indicators are regularly monitored and updated in line with the programme and evaluation life cycle. An integrated spreadsheet containing the process monitoring data can be uploaded to a sharing platform like Google Docs. Data entry and updating enables a ‘snapshot analysis’ of progress to be carried out, which provides a set of time series assessments that ultimately feed into the overall summative evaluation of the programme. In addition it should include the KPIs developed for the evaluation.
An example of a Process Dashboard is shown in the Table below.
● Try not to be too ‘scientific’. Everyone likes ‘numbers’ – particularly programme funders who typically require evidence that their investment shows value for money. However, a programme for young people is not a new anti-inflammatory drug. It’s a complex social intervention. ‘Experimental’ evaluation methods – like randomized control trials – won’t work with this type of programme. Be pragmatic and realist. Use Theory of Change.
● Know your limitations – make sure you have included in your evaluation design and plan estimates of the resources and skills required to carry out the evaluation. Be aware that some evaluation techniques – like ethnographic work and case studies – are more resource-intensive than ‘cheap and cheerful’ methods like surveys.
● Avoid evaluation suspicion and resentment – many programme users and stakeholders see evaluation and performance assessment as the same thing. Make sure you explain to all involved that evaluation is about learning, not performance. Get people on board by using a ‘participatory evaluation’ approach so all user and stakeholders ‘own’ the evaluation.
● Don’t be afraid to measure shortcomings and to report on where the programme objectives fall short. Learning from failure is as important as learning from success.
● Choose data collection tools and design data collection instruments that will appeal to your evaluation participants. For example, if you survey young people, do it through a social media platform they’re familiar with.
● Be SMART – design indicators that are Specific, Measurable, Achievable, Relevant and Time-bound.
● Produce results that are relevant and usable – the main objective of evaluation is to learn. Make sure the learning from the evaluation feeds into ongoing programme monitoring – so you can take remedial steps if necessary – and into the sustainability plan for the programme.
● Replication means getting evidence to show that the programme doesn’t only work in one place or at one time. Although it may not be necessary to expand or transfer the programme, you may need to demonstrate that it can work for different groups of young people in other locations or in other contexts, for example to support further funding applications. This requires using the evaluation evidence from Step 9 to show what works, for whom under which circumstances.
● Stakeholders who may have an interest in providing resources to ensure future continuity of the programme need to be persuaded by strong evaluation evidence that it works and provides value.
● Sustainability means planning for this future continuity by analysing the economic, financial and social returns your programme can deliver and exploring potential sources of funding and other resources that can support the long-term implementation of the programme. ● Sustainability planning should start early on in the life cycle of the programme. Avoid bolting on a sustainability plan at the end. Planning for the future could cover identifying and networking with potential key stakeholders, ensuring robust evaluation is taking place and identifying possible future funders.
● Make sure you engage key stakeholders in the planning for replication and sustainability. The stakeholder mapping work covered in Step 4 will help you identify which stakeholders are likely to want to get involved in future programme expansion or replication.
● Sustainability planning requires skills in economic and financial analysis. You may need to bring these skills in from an external source.
- Produce evaluation report on evidence of programme effectiveness
- Decide on need for scaling up/out
- Review stakeholder map from Step 1 to identify potential partners
- Carry out replication analysis
- Carry out economic and financial analysis
- Produce replication and sustainability plan
Replication Analysis Tool
Replication can be understood as a process of taking a product, service, model or even information into a different setting (context) or to a different target group than the one it was originally developed for. This process is also sometimes referred to as ‘scaling out’ and is different from ‘scaling up’ which tends to involve increasing the volume of what is delivered.
Replication tends to be a three-stage process:
● Knowledge and awareness stage: To replicate or adopt an innovation, it needs to have been shown to meet needs, to be effective and to be known by those considering adopting it.
● Choice and decision stage: this involves relevant actors making choices about the replication destination, the process of doing this, and how it will be financed. Both of the first two stages benefit from the existence of evaluation and / or cost-benefit data.
● The final implementation stage involves taking the product, service or other innovation into one or several other contexts – e.g. adapting the programme for young migrants. This tool supports the first stage of this process, in order to inform the next stage – helping to inform your decisions about how to scale out your project. It provides a checklist of the questions that need to be answered to assess the ‘replication readiness’ of your programme.
Intervention (programme) features and design | Details |
---|---|
What is the nature of the programme? |
|
How much do you know about what the essential parts of your programme are that make it successful? |
|
Will your programme work in other contexts? |
|
What evidence do you have that your programme has an impact? |
|
Replication plans, strategies and structures | Details |
What is the main reason or motivation to replicate the intervention? |
|
What is your business model for replication? |
|
How are you planning to deliver the project in another context? |
|
Is there a clear owner of the replication project? |
|
What understanding and evidence do you have of the match between the social, economic and environmental needs of the local and replication contexts? |
|
What evidence do you have of the supply of people or organisations willing to deliver the programme? |
|
Organisational culture, capability, capacity | Details |
Are the functions and organisational values necessary for replication well defined and developed? |
|
What is the quality of staff involved in the replication effort? |
|
What is the seniority of staff involved in the replication effort? |
|
To what extent are organisational and programme technologies transferable to different contexts? |
|
What is the nature of communication patterns within the project and with external stakeholders? |
|
To what extent do staff and external stakeholders support replication? |
|
Is the brand understood and valued by your audience? |
|
Business Model Canvas
A business model describes how an organization creates, delivers and captures value. It’s a visual template that can be used to outline key elements of a business model. The most commonly used model – the ‘classic’ model - focuses essentially on financial aspects (Osterwalder, 2010). They show things like key partners, key activities, key resources, customers, costs and revenue streams. Other models are adapted for service delivery programmes (Jukka and Katri Ojasalo, 2015), whilst others reflect ‘social returns’ – i.e. benefits that may not have a purely financial value but can be ‘monetised’ in terms of the contribution they make to the social good including social and environmental impact (Joyce and Paquin, 2016).
● Over-ambition – many an enterprise has failed because it expanded to much and too soon. There’s no absolute reason why a successful programme deserves be scaled up and out. The case for replication should be carefully examined – by assessing level of demand and user need, stakeholder interest and the support available from the external environment – before going ahead with a replication strategy.
● Lack of evidence – you need to have credible and plausible evidence of value in order to develop a replication and or sustainability plan. This highlights the importance of developing and implementing an effective evaluation effort as an ongoing feature of the programme development and implementation process.
● Lack of interest – there are many worthy and effective programmes for young people out there, which makes for significant competition. Successful replication and sustainability strategies are those that have stakeholder buy-in – not only from funders put potential programme partners and users. You need to cultivate and work collaboratively with potential stakeholders early on.
● Lack of technical resources and expertise – assessing replication readiness and sustainability potential, as well as developing a business case for a programme, takes significant technical know-how and resources. You need to ensure these are in place – either in-house or imported – to develop successful replication and sustainability plans.



Empowering Youth to Shape the Future
With the tools provided by MYSTIC Project
Funded by the European Union. Views and opinions expressed are however those of the author(s) only
and do not necessarily reflect those of the European Union or EACEA. Neither
the European Union nor the granting authority can be held responsible for them.
