Reproducibility of Hedonic Data
Consumer benchmarking studies form a common part of competitor tracking and product quality management. A shift in hedonic ratings against competitors helps to determine if our product quality is stable and may also highlight quality improvements by the competition.
It follows that benchmarking assumes that a product’s hedonic ratings are stable to the effects of different sets of consumers (albeit selected to the same criteria); and to different test conditions such as time of year, weather and so on.
To investigate this assumption, we looked at data from two independent studies. Each study had looked at three chocolate products, two of which were the same in both with only the third changing. The first study was run on a very hot day in July 2017, the second on a typically cold and windy Spring evening in 2018.
The results for overall liking and liking for flavour and texture are shown in the chart. In this case the reproducibility of the test was excellent and no statistically significant differences were found. Whilst this can give us confidence both chocolates were highly liked – we are interested to investigate the stability of data for staples which tend to have lower scores.
One point of interest: we have observed on many occasions that national mood has a marked influence on stability of hedonic data. An iHUT in Italy a few years ago showed a sudden dip in ratings at one point during the four-week placement period. Investigation showed this aligned exactly with a national rail disaster in the country. A reminder that when looking at our results we need to remain alert to how external factors, and not just product, may be influencing our conclusions.