Data maturity self assessment

Dark-pink-background.jpg

HOW DATA SAVVY IS YOUR ORGANISATION?

Self Assessment TOOL

We’ve developed this free tool to help you assess how data savvy your organisation is and identify areas for improvement. The assessment is based on our data maturity framework.

You can choose between a short 5 minute survey or the full assessment, which takes about 20 minutes to complete.

ABOUT THE TOOL

Want to know how we got to this point? Read more about the background behind our research into data maturity and how the data maturity framework was developed.


What’s the purpose of the data maturity self-assessment tool?

For you, the users

The self-assessment tool is a diagnostic tool to help not-for-profit organisations understand how they are doing with data. Users can complete a 5 minute or 20 minute version of the assessment and the responses are used to generate a report on your organisation's data maturity.

The 5 minute version will give a quick ‘rough’ indication. The longer version will indicate strengths and weaknesses across the seven themes, enable comparison with a benchmark, and identify the next stages to work towards.

We hope it will stimulate thinking and dialogue about data and enable organisations to develop plans and priorities for the future. There are no right or wrong answers, it’s your self assessment and it’s completely confidential, so it’s best to be as honest as possible.

For us and our not-for-profit sector

We will ask a few questions about you, your role/s and your organisation. These will not be shared with anyone but will enable us to monitor how representative of the sector our respondents are.

Anonymised aggregated data from respondents will be analysed for research purposes to help build understanding about data maturity in the not-for-profit sector. We will use this to inform funders, policy makers, and other Data4Good partners about the needs of the sector and design appropriate support services.

Who should take the assessment?

In our experience ‘data’ means different things to different people and nobody knows everything there is to know about data in an organisation. Typically the answers to the assessment questions are in many people’s heads and you/they may not necessarily agree. See ‘Three ways for organisations to take the assessment’ below.

For those wanting to use the free self-assessment tool, or wanting to start with this, we recommend you complete the online assessment as a small group. In general, the more participants from your organisation involved, the more accurate the results will be. Realistically though, with more than six people it’s difficult to manage the level of discussion and opinion in a timely way. Ideally the group should include someone in a leadership position and the person in the most data-related role. Others may come from service delivery/planning, research and evaluation, digital technology, marketing and communications, HR, ICT, Fundraising/sales.

Many of Data Orchard’s clients seeking a serious strategic review of data, complete a ‘Whole Organisation’ assessment where large numbers (sometimes all) of the staff take part. This might also suit larger or geographically dispersed organisations where it’s difficult to get a group of people together in the same place at the same time.

If you’re completing the assessment as an individual we recommend you answer questions to the best of your knowledge based on your perception of the overall organisation, rather than you as an individual or views of a particular project or department.

So, it’s best for as many people to be involved as possible. Obviously the more diverse and representative the better. We recommend a diverse mix of seniority from frontline to senior leadership and across a range of teams/departments and projects.

Three ways for organisations to take the assessment

  • Option 1: Individually. People in organisations to take the online self-assessment individually and then might get together with their separate results to discuss. When completing the assessment, we ask that you answer questions based on your perception of the overall organisation, rather than you as an individual or views of a particular project or department. (FREE)

  • Option 2: Small group. Get up to six staff together and complete the online self-assessment as a group. You should probably allow at least an hour for discussion around the questions. Ideally the group should include someone in a leadership position and the person in the most data-related role. Others may come from service delivery/planning, research and evaluation, digital technology, marketing and communications, HR, ICT, Fundraising/sales. (FREE)

  • Option 3: Whole organisation. The whole organisation option is suitable for those taking a serious strategic review of data in their organisation. It might also suit larger or geographically dispersed organisations where it’s difficult to get a group of people together at the same time. Purchase a unique link to share with colleagues in your organisation, set your own time-frame for collecting responses, and get an organisation-wide data maturity assessment report. (FROM £500+VAT)

There are also options for a range of add-on features including facilitation of the process and engagement with an independent consultation. Contact Data Orchard for a no obligation chat about your requirements.

How does the scoring and benchmarking work?

Essentially the measurement in our data maturity tool works by: scoring questions; weighting questions; grouping the weighted scored questions by theme and taking an average of these to giving an overall theme score; and averaging the seven theme scores to give a data maturity stage (based on a scale of 0 to 5 for Unaware through to Mastering).

All questions are normalised so that before weighting they contribute equally to the relevant theme and themes are normalised so that they contribute equally to the overall score for an individual organisation.

We note that this current approach is fairly simple and different methodologies could be applied. We plan to review this in 2020 once we have a much larger dataset to work with.

Scoring in more detail

Each question has scoring assigned to it in the range 0 to 5. Most are on an ‘extent’ or ‘agreement’ range though some have bespoke scoring. ‘Don’t know’ and ‘not applicable’ responses are not included in the scoring. For some questions, where multiple responses are possible (tick all that apply), the total score is normalised by 5/maximum possible scores if all options are selected so that the score is in the range 0 to 5.

Where we arrange for multiple users of an organisation to take the assessment so we can get a more detailed overview, we sum the scores and divide by the number of responses (excluding don’t know/not applicable) to give an average score for that question.

Weighting in more detail

Each question is assigned a weight: standard, high, or very high based on how important responses will be in determining important features or behaviours of data maturity. In testing we have monitored both weighted and unweighted scores for themes.

Benchmarking in more detail

Benchmarking has been used as a key feature for enabling comparison between one organisation and a number of others. For benchmarking we use the same approach for the scoring, weighting and grouping of questions by theme. However we select ‘all respondents’ rather than one organisation’s response. To date, in testing, we have been able to verify participating organisations are authentic, and validate the tool’s assessment against our own observations.

Who built the tool?

We have a long list of people and organisations to acknowledge and thank for bringing this data maturity self assessment tool into being.

The Data Orchard Team

Sian Basker, Co CEO at Data Orchard has been the driving force and lead researcher developing it since way back in 2015. Madeleine Spinks (the other Co CEO), Tony Cramp (Senior Analyst), Ben Proctor (Associate Communications Manager for Data Evolution project); and Paul Ticher (Associate Data Protection Specialist) have been the other key Data Orchard people involved in its development.

Partners and specialist advisors

Partners: DataKind UK (notably Emma Prest) and the Project Board that supported the first pilot phase of the research in 2016 (Duncan Ross - DataKind UK/TES Global; Alexandra Rehak- DataKind UK/Ovum; Madeleine Thornton - Big Issue Invest; Gaia Marcus - Centrepoint; Ed Anderton - Access Social Investment). Tracey Gyateng, Data Science Manager, and their data science volunteers also tested the beta-phase assessment questions and provided feedback in 2019.

Online Data Maturity Assessment Tool Steering Group that supported the second phase of the research and development: Caitlin Milne and Anne Tregoning (Kindlemix Communications); Suraj Vagdama (CAST), Giselle Cory (DataKind UK), Ben Proctor (Satori Lab/ODI Cardiff).

Specialist advisors: We were enormously grateful to draw on the expertise of seven specialist advisors with experience of working with not-for-profits in building their data and digital capabilities. They kindly reviewed the beta stage assessment questions and advised on improvements to the first data maturity framework. These were: Megan Griffith Gray (NCVO); Ellie Harries (Freelancer); Tracey Gyateng (DataKind UK); Josh Cockcroft (Esmee Fairbairn Foundation); Ben Proctor (ODI Cardiff/Satori Lab); Kate White (Superhighways); and Alison Prince (Parkinson’s UK).

Special thanks to: Ellie Harries whose expertise has driven the development of some amazing data tools (Inspiring Impact; Audience Finder); Elizabeth Larkin (Epidemiologist/Data Scientist at Digital C in Cleveland US) who took our pilot Data Evolution assessment questions, adapted them, and surveyed hundreds of non-profits in the US, and kindly fed back their learning and experience to inform our next phase of development; Gaia Marcus, Alison Prince, and Jesse Mears from Parkinson’s UK who shared their experience of using the original data maturity framework and their experience of assessing and re-assessing data maturity.

User testing participants

User Testing #1 (May 2018 – July 2019): The following charities submitted themselves (in some cases ALL their staff and trustees) to the task of completing the beta phase online assessment and benchmarking (in some cases they did it twice – 12 months apart): The Cart Shed, Bracken Trust, The Key, Joseph Rowntree Foundation. In total the beta version tool was tested by 51 people from 4 organisations and we are enormously grateful to them for their feedback.

User Testing #2 (March 2019): Six not-for-profit organisations kindly took part in our user testing of different data/digital maturity tools. They were: Thomas Pocklington Trust; Founders and Coders CIC; FareShare; Parkinson’s UK; Jupiter Ltd; Into University.

User Testing #3 (July 2019): Five not-for-profit organisations took part in our third round of user testing of our online tool. They were: St John’s Ambulance; The British Science Association; Imperial Health Charity; Charity Finance Group; and ACDI/VOCA.

The web developers

Following a formal tendering process we were delighted to appoint Outlandish, a leading tech co-operative, to build the online tool. Their technical expertise both in terms of web development, user design, and in data was crucial to the project’s success. We also really liked that they were themselves part of the not-for-profit family!

Our funders

We are extremely grateful to our funding partners for their grant support towards the creation of this data maturity assessment tool. They are: