Science

Measurement matters: Part 3 – How we developed and tested the Unmind Index

Rhian Male, MSc

Example position

Unmind Product Spotlight: Calendar Reminders

TABLE OF CONTENTS

We’ve explained the how and why of the Unmind Index. But how do we know it works? This final blog unpacks how we developed the Index and why you can trust it’s doing its job properly.

How did we develop the Unmind Index?

Creating a new measure is a complex process, with many steps. 

We set out to develop a measure that would benefit everyone. It needed to capture the whole spectrum of MHWB – from key components of positive mental wellbeing, to emerging or existing symptoms of common mental health problems. 

The mission was to create a resource that was relevant for all Unmind users, that covered many aspects while making sure it was simple, accessible and effective.

So here’s what we did: 

  • We worked with six clinical psychologists specialising in adult mental health to create a large pool of potential questions relating to different aspects of mental health and wellbeing.
  • We asked over 1,000 research participants to answer these questions.
  • We analysed this data using a statistical method (known as exploratory factor analysis) to find groups of questions that seem to relate to the same areas of mental health and wellbeing.
  • We then selected the questions which best measured each of the most important areas.
  • We were left with 26 questions across seven areas of mental health and wellbeing, which make up the Unmind Index. These are Happiness, Calmness, Coping, Sleep, Health, Connection, and Fulfilment.

How did we ensure the Index is backed up by science?

To make sure we were on the right track, we needed the best and sharpest scientific minds. That’s why we called on the assistance of our world-leading collaborators at the University of Cambridge Psychometrics Centre.

To ensure the Index provides the most valuable and accurate insights and content recommendations to our users, we put it through a validation study. In simple terms, this was a scientific MOT – to check the Index was doing its job properly.

Once we’d decided on our final 26 questions, we asked another 1,000 research participants to complete the Index. 

Survey-takers were representative of the UK population in terms of age, gender, and ethnicity, to make our results relevant for our range of users in the UK. They also completed some other established gold-standard mental health measures. Some participants were invited back one week later to complete the Index again. 

The tests would answer two key Qs: is the Index valid? And is the Index reliable?

Is the Index valid?

Broadly speaking, a measure is valid if it’s appropriate for what you’re using it for, but we can break this down even further: 

  • Good concurrent validity is when the measure compares well to an established, gold-standard measure that measures the same thing. We found that questions in each of the seven areas of the Index correlated with related gold-standard measures. For example, Happiness scores negatively correlated with scores on the PHQ-9 – a measure of depression. 
  • Good discriminant validity is when the measure is not related to measures that aim to measure something different. This can happen if the questions are not specific enough. The Index was only weakly related to the TIPI, which is a measure of personality traits so shouldn’t be highly related to Index scores. 

So far, so good.

Is the Index reliable?

Reliability, on the other hand, is all about whether something is measured well. It means that other random factors don’t have too much influence on how someone answers the questions.

This also falls into two categories:

  • Good internal-consistency is when questions that belong to the same area or aspect of MHWB are strongly related to each other. This was excellent for all seven areas of the Index, and for the overall wellbeing score that the Index produces.
  • Good test-retest reliability is that participants’ scores are similar if they complete the measure twice, one week apart. Participants’ scores on the Index after one week were strongly correlated to their first set of scores.

This left one final hurdle to navigate: relevance.

Is the Index relevant?

The Unmind Index should be relevant for everyone. We tested to make sure that measurement doesn’t vary depending on gender or age, known as measurement invariance. We concluded that the Index is measuring the same underlying construct regardless of gender and age. 

In conclusion: we did it. 

The Unmind Index is a scientifically-validated psychological measure. 

It’s a valid and reliable measure of MHWB and the seven areas it targets, so it’s an appropriate and useful tool for you and other Unmind users. 

You can trust the Unmind Index will help you reflect on and monitor your MHWB. A clinical psychologist has put together the personalised recommendations that the app provides – based on your responses to the Index’s questions. 

What’s next for the Index?

If you’re interested in reading more about the development and testing of the Index in the UK sample, we’ve just published an academic paper for this study.

We’ve also been testing the Unmind Index in the US, Australia and New Zealand. This is important because we know that measurement can be different in other cultures and regions. Because Unmind has a global user base, we want to ensure the Index is doing its job properly across the globe.

For instance, the measure may be measuring different underlying constructs, because of differences in how people conceptualise mental health, or because people interpret the individual questions and wording differently. 

Happily, we found that the Unmind Index is also a valid, reliable and consistent measure of MHWB in these countries. 

The next step? To translate the Index into other languages. (We’re working on it.)

Finally, we’re including the Index as a measure in some of our efficacy studies. This will help us work out how well the Index can detect changes in mental health. We’ll see whether changes in existing measures like the GAD-7 predict changes in Unmind Index scores.

Thanks for reading. Over this three-part series, we hope to have:

  • Given you some insight into ways mental health and wellbeing can be measured.
  • Helped you understand why we created the Unmind Index.
  • Helped you appreciate the seven areas of MHWB the Index targets and the way we ask questions.
  • Explained how we can be sure the Unmind Index is valid and reliable, and that it’s doing its job properly.
  • Teased our exciting future plans for the Index.

Missed parts 1 or 2 of this blog series? Or so impressed you're busting to read it again? Don't fret: here's part 1 and part 2.

Find out more about Unmind

To learn more about nurturing the mental health of your organisation, click here to book a chat with one of our specialists.