Over the last year, we ran a series of elicitation workshops that aimed to quantity the factors that were needed to shape our Horizon 2035 model. Elicitation is the process of obtaining knowledge from one or more experts for uncertain futures.
For the purposes of Horizon 2035, elicitation is used to quantify critical model parameters with an unknown future value. There is no alternative to eliciting expert knowledge as the future is inherently undefined.
At the CfWI, we have decided to use the SHeffield ELicitation Framework (SHELF) (http://www.tonyohagan.co.uk/shelf/) developed by Professor Tony O’Hagan an Emeritus Professor, Department of Probability and Statistics (University of Sheffield).
This form of elicitation replaces our online Delphi panel exercises. This enhancement is a natural evolution of modernising our methodology to fit future needs. Limitations from the incumbent panel method used at the CfWI included the longer length of the process, no open debate, losing engagement of experts between the rounds, and rationales often being reduced to short sentences. The purpose of using SHELF is to improve on such concerns.
The SHELF method involves two rounds of debate on a particular question that is pertinent to an area under consideration. After the first round of debating, experts are asked for a series of values on the question. The median, which is the value the expert believes to be most probable, is determined first. This is followed by the upper and lower bounds which are values the experts believe are possible but extremely unlikely. Finally, the lower and upper quartiles are asked for, which are the twenty-fifth and seventy-fifth quartiles between the upper and lower bounds. These are used to obtain a probability distribution curve for all experts. An example of a probability distribution curve is shown below in Figure 1.
Figure 1: An example of a probability distribution curve. Green line showing the lower quartile, the blue line showing the upper quartile. The peak of the curve representing the median, the most likely value.
These graphs are then shown to the group and a second round of debate takes place in order for a consensus to be reached. For Horizon 2035, we have run three workshops considering physical long-term health conditions, mental long-term health conditions and workforce productivity. The workshops were one day long each, which gave the participants enough time to understand the SHELF approach and to answer our questions.
One lesson that we have learnt is that in order to have a successful elicitation, it is vital that you invite experts that are relevant to the discussion, and provide them with a comprehensive brief beforehand. This is particularly important if you want to save time on the day as there is a lot of content to get through. For our workshops, we sent out briefing packs which explained our project, the process and the questions we will be asking.
The structure of the day was broken down into three sections. The beginning of the day was devoted to explaining our project, what we were hoping to achieve from Horizon 2035, and how the experts could help us. Here we gave the experts a chance to ask questions on the project itself to develop a stronger sense of understanding of the project and its importance. This led to increased levels of engagement.
The second section of the workshop was spent on training the experts in the SHELF method, during the training we had ran through various examples. The experts were explained probability distributions at first. Once these concepts of SHELF had been grasped we were able to move on to the third section of the workshop.
In the third part of the workshop we moved on to asking the experts the questions we needed answering for our project. A running theme in all the workshops, which was interesting to note, was the discussion on whether the question being asked was of the right one. However, for each question being asked, only a few minor tweaks were made to them. For example in long-term mental health, we changed the scope to include alcohol and substance abuse. This showed the levels of engagement of the experts which was a positive sign. Once the questions were agreed by all experts to be the right questions we had started the SHELF process.
The most interesting part of the day was to see the experts debate the questions we had presented them. This showed their expertise in the matter and difference of opinion. For instance, in our workshops we were able to get a consensus on all questions except one where there was a split opinion. This was fine as it presented two differences in opinion on a certain future. In the workshops we may have wanted to consider extra time as asking three questions was challenging and in the last workshop we weren’t able to ask the third question due to time constraints. Looking back at the workshops we could even have spread them over two days to give us enough time for all our questions to be answered, though we recognise that this is a lot of time away from usual jobs for our participants and is not always possible.
All in all the SHELF method has proven to be a great way of gaining quantities which were not possible to gain elsewhere. Going forwards, we will try to improve the process ensuring learning from our previous workshops. All readers are welcome to contact the team with their thoughts or questions to email@example.com.