THE NEED FOR ROBUST APPRAISAL OF RESEARCH IN ARTS AND HEALTH: QUESTIONS OF VALUE, RELEVANCE AND USEFULNESS (GUEST BLOG)

A GUEST BLOG BY DR STEPHEN CLIFT

This is the third guest blog by Stephen Clift. Here he asks questions about the value, relevance and usefulness of some research into 'arts and health', and wonders how such examples were funded and why they garnered such favourable, unquestioning reviews.

Stephen Clift (BA, PhD, PFRSPH) is Professor Emeritus, Canterbury Christ Church University, and former Director of the Sidney De Haan Research Centre for Arts and Health. He is a Professorial Fellow of the Royal Society for Public Health (RSPH) and is also Visiting Professor in the International Centre for Community Music, York St John University.  Stephen has worked in the field of health promotion and public health for over thirty years, and has made contributions to research, practice and training on HIV/AIDS prevention, sex education, international travel and health and the health promoting school in Europe. His interests relate to arts and heath and particularly the potential value of group singing for health and wellbeing. Stephen is one of the founding editors of the journal Arts & Health: An international journal for research, policy and practice.  He was the founding Chair of the RSPH Special Interest Group for Arts, Health and Wellbeing, and a founding trustee of Arts Enterprise with a Social Purpose (AESOP).  He is also co-editor with Professor Paul Camic of the Oxford Public Health Textbook on Creative Arts, Health and Wellbeing published in November 2015. Currently, he is working on developing a series of provocations in arts and health research. a special collection of critical papers on arts and health with Frontiers in Psychology, and a special issue of the International Journal for Community Music on the impacts of the COVID-19 pandemic.

THE NEED FOR ROBUST APPRAISAL OF RESEARCH IN ARTS AND HEALTH: QUESTIONS OF VALUE, RELEVANCE AND USEFULNESS

Introduction

As the volume of research in ‘arts and health’ has increased over the last fifteen years, and new opportunities for peer-reviewed publications have developed, there have been repeated calls for greater robustness in research in this area.  This has been promoted, in part, by the growing number of ‘systematic’ reviews and even Cochrane reviews, which have drawn attention to the methodological shortcomings of the ‘arts and health’ research corpus, as well as academic discussions of the development of the field and priorities for future research (Clift and Camic, 2015; Stickley and Clift, 2017).

There have also been welcome initiatives to set standards for research and evaluation in the field (Fancourt and Joss, 2015; Daykin and Joss, 2016; Fancourt, 2017), which reflect wider developments in the medical and social sciences to enhance the rigour of research (e.g. MRC complex interventions guidance (Craig et al., 2006); PROSPERO international register for systematic reviews (Booth et al., 2012); CONSORT guidelines for reporting RCTs (Schultz et al., 2010), PRISMA guidelines on systematic reviewing (Shamseer et al. 2015 etc.), and COREQ criteria for reporting qualitative studies (Tong et al., 2007).

In addition to issues of methodological rigour in the conduct and reporting of research, however, I think there are other standards that research in ‘arts and health’ should meet. These include: the relevance, significance and originality of topics covered; clarity of theoretical underpinnings; the extent to which researchers engage effectively with patients and the public in designing studies; the ‘artistic’ quality of the ‘arts’ in ‘arts and health’ activities, the quality of co-production and co-research relationships between service users and researchers, and realistic attention to the relevance of findings to practice.

There is some evidence, sadly, that research in ‘arts and health’ is undertaken and published which fails to meet some of these essential substantive criteria, and which raise questions about value, relevance and usefulness.  Further questions are raised as to how proposals for such work were ever funded; how they achieved a favourable ethical review, and how they survived peer-review scrutiny by academic journals – even journals with a substantial reputation and good impact factors.

My concerns about these questions are highlighted by a paper recently published in the Royal Society of Public Health (RSPH) journal Perspectives in Public Health: ‘Happy Hookers: findings from an international study exploring the effects of crochet on wellbeing’ (Burns and Van Der Meer, 2020).  I will briefly outline the study and then detail my concerns about it. 

I want to make it clear at the outset that my intention is not to demean crochet! I have dear friends who are passionate about crochet and I have joined in crochet workshops myself. I fully appreciate how demanding and enjoyable this craft is and how much skill is involved in creating the funny, beautiful, stylish and useful artefacts illustrated below.  If people value and enjoy this craft activity; if they get together to share their enthusiasm; if they create gifts for family and friends – then that is all wonderful. But the activity is one thing – research on the activity is another.

Burns and Van Der Meer (2020) Summary

Burns and Van Der Meer conducted an online survey to explore the wellbeing benefits of crochet modelled closely on an earlier online survey of the benefits of knitting (Riley, Corkshill and Morris, 2013). The survey garnered responses from over 8,000 participants from 87 countries (but mainly the USA and UK).  Virtually all respondents were female ranging in ages from teens to 81+, and three-quarters had been crocheting from between 3 and 40+ years.  Data was gathered on reasons for crocheting and impacts on thoughts and feelings based on ratings of options provided.  In addition, participants were asked to rate their ‘mood’ before and after crocheting and had the opportunity to add any further information they wished in an open text box.

The main problems with this study

• The survey is based on a convenience sample, accessed online, of participants committed to the activity of crocheting. They are also people who have access to the internet and have the skills and motivation to participate in a survey.  The sample is very large and international, but this does not compensate for these obvious biases.
• The survey instrument consists of questions which are highly leading, particularly where respondents were asked to report on their feelings and thoughts when engaged in the activity. The precise format of the questions is not explained in the paper, but reference to Riley, Corkhill and Morris (2013) reveals that in the knitting survey, respondents were presented with ‘5,7 or 10-point scales (…) to rate their mood or feelings.’ (p.52). The findings, for example, that 82% of respondents reported feeling ‘happier’ when crocheting; that 71% felt it improved ‘concentration’ are so unsurprising as to be banal.
• The authors also report ‘significant’ associations between the amount of time spent on crocheting a range of positive feelings and thoughts which are both banal and trivial. In this paper, ‘significant’ association’ between time spent and feelings and thoughts are based on the use of the chi-squared test – so the scale of the differences are not apparent. 
• An insight into just how trivial these relationships are likely to be is provided by the earlier paper on knitting, where mean ratings on four ‘cognitive’ variables are reported in relation to frequency of knitting (Table 3, p.54) (It is not entirely clear what the rating scale was for these variables, but as the mean values in Table 3 are all between 3 and 4, the likelihood is that a five-point scale was used). In the earlier paper we find, for example, that ‘forget problems’ when knitting is more often reported by respondents who knit three or more times per week, than those who knit less than three time per week. A two-sample t-test is employed and a value of p<0.001 is reported. However, the means for ‘forget problems’ is 3.33 for participants knitting more frequently, and 3.16 for those who knit less frequently – a difference of a mere 0.17 on a five-point scale.  This is a clear example of a ‘statistically significant’ result due entirely to the large sample sizes involved (n=2562 vs. n=666), rather than the difference having any substantive meaning.
• The comparison reported between ratings of ‘mood’ on a seven-point scale before and after crocheting are not clearly reported and in any event the finding of a ‘significant difference’ in the assessments of mood is both banal and spurious, as the ratings are retrospective and made at the same time.
• There are also findings which are trivial, and comparisons are made with the previous survey of knitters, which emphasis this point. For example: ‘The impact of colour and texture was less for people who crochet (22.1% and 33.1%, respectively) than knitters (24.0% and 46.0% respectively). It is not clear why this difference occurred; however, it should be noted that we did not explore the impact of yarn content, for example, acrylic versus natural fibres on mood.” (p.5)
• To be fair to this study, the open responses in the survey did produce some interesting comments, but the reporting is highly inadequate, with little information on how qualitative data was processed, how often different themes were expressed and no indication of how quotations were selected for inclusion in the paper. There are repeated references to: ‘Many respondents chose to talk…,’ ‘Many respondents reported…,’ ‘Many respondents commented…’ followed by a selection of quotations.
• A final concern raised by this paper relates not to its limitations, but the fact that the editors of Perspectives in Public Health, on the advice no doubt of the peer-reviewers consulted, decided to publish such a weak and essentially uninteresting paper.

Summing upAs a general principle, when a person engages in any activity, which they value and enjoy, and in which they invest considerable time and some money – they are almost certain to agree with any positively-worded statements about it. Please, no more research of this kind!Challenges

• To the authors of the study reviewed: Do you accept the critique made of your work? If not, what defence would you give in response to the criticisms made?
• To the Ethics Committee that approved this study: In the course to considering the proposals for this study, were any of the criticisms made above anticipated by reviewers?  Given the criticisms made above, would you still give the study ethical approval?
• To the editors of the journal Perspectives in Public Health: Were any of the criticisms made above raised by reviewers and put to the authors? Given the criticisms made above, was it a good decision to publish this study?
• To researchers in arts and health: Do you agree with the judgements made in this critique that the findings reported in this study are ‘banal’, ‘trivial’ and ‘spurious’? Can you identify other studies which make little in the way of a contribution to knowledge and practice in the field of arts and health?

SourcesBooth, A., Clarke, M., Dooley, G., Ghersi, D., Moher. D., Petticrew, M. and Stewart. L. (2012) The nuts and bolts of PROSPERO: an international prospective register of systematic reviews, Systematic Reviews, 1, 2, 1-8. https://systematicreviewsjournal.biomedcentral.com/track/pdf/10.1186/2046-4053-1-2Burns, P. and Van Der Meyer, R. (2020) Happy Hookers: findings from an international study exploring the effects of crochet on wellbeing, Perspectives in Public Health, published online. https://journals.sagepub.com/doi/abs/10.1177/1757913920911961Clift, S. and Camic, P. (2015) (Eds.) The Oxford Public Health Textbook of Creative Arts, Health and Wellbeing. Oxford: Oxford University Press.Craig, P., Dieppe, P., Macintyre, S., Michies, S., Nazareth, I. and Petticrew, M. (2006) Developing and Evaluating Complex Interventions: New Guidance, London: Medical Research Council. https://mrc.ukri.org/documents/pdf/developing-and-evaluating-complex-interventions/Daykin, N. and Joss, T. (2016) Arts for Health and Wellbeing: An Evaluation Framework.  London: Public Health England. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/765496/PHE_Arts_and_Health_Evaluation_FINAL.pdfFancourt, D. (2017) Arts in Health: Designing and Researching Interventions. Oxford: Oxford University Press.Fancourt, D. and Joss, T. (2015) Aesop: A framework for developing and researching arts in health programmes, Arts & Health, 7, 1, 1-13. 10.1080/17533015.2014.924974Prospero International Prospective Register for Systematic Reviews: https://www.crd.york.ac.uk/prospero/

Riley, J., Corkhill, B. and Morris, C. (2013) The benefits of knitting for personal and social wellbeing in adulthood: findings from an international survey. British Journal of Occupational Therapy, 72, 2, 50-57. https://journals.sagepub.com/doi/abs/10.4276/030802213X13603244419077  

Shamseer, L., Moher, D., Clarke, M., Ghersi, D., et al. (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation, British Medical Journal, 349, https://doi.org/10.1136/bmj.g7647 Schulz, K.F., Altman, D.G. and Moher, D. for the CONSORT group (2010) CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials, British Medical Journal, 340, https://doi.org/10.1136/bmj.c332 Tong, A., Sainsbury, P. and Craig, J. (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups, International Journal for Quality in Health Care, 19, 6, 349–357, https://doi.org/10.1093/intqhc/mzm042