Article Text

Download PDFPDF

Vitruvian plot: a visualisation tool for multiple outcomes in network meta-analysis
  1. Edoardo Giuseppe Ostinelli1,2,3,
  2. Orestis Efthimiou1,4,5,
  3. Huseyin Naci6,
  4. Toshi A Furukawa7,
  5. Stefan Leucht8,
  6. Georgia Salanti5,
  7. Laurence Wainwright2,
  8. Caroline Zangani1,2,3,
  9. Franco De Crescenzo1,2,
  10. Katharine Smith1,2,3,
  11. Katherine Stevens2,
  12. Qiang Liu1,2,
  13. Andrea Cipriani1,2,3
  1. 1Department of Psychiatry, University of Oxford, Oxford, UK
  2. 2Oxford Precision Psychiatry Lab, NIHR Oxford Health Biomedical Research Centre, Oxford, UK
  3. 3Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, UK
  4. 4Institute of Primary Health Care (BIHAM), University of Bern, Bern, Switzerland
  5. 5Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
  6. 6Department of Health Policy, London School of Economics and Political Science, London, UK
  7. 7Department of Health Promotion and Human Behavior, School of Public Health, Kyoto, Japan
  8. 8Department of Psychiatry and Psychotherapy, Technical University of Munich, Munchen, Germany
  1. Correspondence to Professor Andrea Cipriani, Department of Psychiatry, University of Oxford, Oxford, UK; andrea.cipriani{at}psych.ox.ac.uk

Abstract

Objective A network meta-analysis (NMA) usually assesses multiple outcomes across several treatment comparisons. The Vitruvian plot aims to facilitate communication of multiple outcomes from NMAs to patients and clinicians.

Methods We developed this tool following the recommendations on the communication of benefit–risk information from the available literature. We collected and implemented feedback from researchers, statisticians, methodologists, clinicians and people with lived experience of physical and mental health issues.

Results We present the Vitruvian plot, which graphically presents absolute estimates and relative performance of competing interventions against a common comparator for several outcomes of interest. We use two alternative colour schemes to highlight either the strength of statistical evidence or the confidence in the evidence. Confidence in the evidence is evaluated across six domains (within-study bias, reporting bias, indirectness, imprecision, heterogeneity and incoherence) using the Confidence in Network Meta-Analysis (CINeMA) system.

Conclusions The Vitruvian plot allows reporting of multiple outcomes from NMAs, with colourings appropriate to inform credibility of the presented evidence.

  • PSYCHIATRY
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

The impact of scientific findings on real-world clinical practice is determined not only by their intrinsic value but also by how effectively they can be communicated to clinicians and patients.1 This has been the case for more than 150 years. In 1854, nurse and statistician Florence Nightingale arrived in Turkey with a group of 37 nurses to attend the wounded of the Crimean War. While there, she recorded more casualties from epidemic diseases, malnutrition, poor sanitation and other modifiable factors than from battlefield wounds.e1 Her initial attempt to communicate how a large proportion of these deaths was avoidable used complex statistics. Despite her efforts, her findings were viewed with scepticism and disregarded.e1 Realising that the main pitfall was the format used to communicate her findings, she developed the Diagram of the Causes of Mortality in the Army of the East, now known also as Nightingale’s rose diagram or coxcomb.2,e2 The success of this communication strategy promoted the implementation of preventative public health measures and ultimately contributed to the reduction of death rates in hospitals.e1

Available literature on behavioural decision research and communication on benefit–risk information indicates that the format in which findings are presented may significantly affect the comprehension and behaviour of end users.3–7,e3, e4 Well-designed patient decision aids—either visualisation tools or conversation aids—have been shown to materially improve patients’ understanding of the effects of outcomes and risks compared with control interventions.3, e4 First, bars and pictographs have been identified as the best strategies to visually communicate complex statistics and numerical results from multiple comparative alternatives.6, e3 Second, a common timeframe and a consistent reference class (denominator) and format should be employed when comparing the chance of occurrence of two or more independent events, favouring absolute risks, either percentages or frequencies, over relative ones.6 Third, 6textual and graphical communication should acknowledge the average numeracy and literacy skills of the target audience.5 6 Fourth, the implications of communicating the inherent uncertainty in the results should also be considered.6 7 Finally, available information should be accompanied by narrative statements to inform both gain and loss frames (ie, people experiencing and not experiencing the event of interest)6 7

Network meta-analysis (NMA) is a useful tool for summarising available evidence about the effects of three or more interventions for the same condition.8 Typically multiple outcomes are analysed separately, aiming to capture the comparative profile among treatment alternatives. The combination of multiple interventions and outcomes further increases the complexity of presenting benefit–risk findings from NMAs. Pooled estimates concerning an outcome are usually presented using forest plots or league tables.e5 These methods, however, are only fit to simultaneously show results regarding a small number of outcomes, limiting the overall interpretation of findings.9 10 Alternative methods to visualise multiple outcomes from NMAs have been proposed,11–14,e6-e10 with only limited consistency with good practice guidelines for communicating benefit–risk information.6 15–18 For instance, some tools prioritised the textual information over the graphical communication, with the latter limited to the use of numbers in coloured tables.12,e6-e8 A visual representation would allow patients and clinicians to quickly appraise and compare the available interventions across the considered outcome. On the other hand, other tools employed alternative visualisation display (ie, rank heat plot, bubble plot, spie chart, and scatter plot) to communicate risk information from advanced metrics.11 13,e9, e10

The development of an effective and user-friendly communication tool would be a critical step forward in the implementation of an evidence-based approach in real-world shared decision-making processes. We aimed to create a tool to facilitate communication of findings from multiple outcomes in NMA to patients and clinicians. We developed this tool following recommendations from the available literature and collecting feedback from researchers, statisticians, methodologists, clinicians and people with lived experiences of physical and mental health issues. We named it a ‘Vitruvian plot’ after Leonardo da Vinci’s Vitruvian Man, which simultaneously represents multiple human proportions. This article illustrates two alternative visualisation strategies to communicate results from multiple NMAs and the evaluation of confidence in these findings. The data sets, the R code and related instructions to further customise the script are freely available on GitHub (https://github.com/EGOstinelli/Vitruvian-plot). In the online supplemental material, we provide two competing visualisation methods using the same data sets.

Methods

How to build the plot

We developed the Vitruvian plot in R as a radial bar plot, which is a bar plot shifted from a cartesian coordinate system (where coordinates are defined as distances from two fixed perpendicular axes) to a polar one (where coordinates are defined as the distance from a polar point and an angle from a polar axis). Each Vitruvian plot corresponds to a single intervention and shows the NMA results for multiple outcomes.

For binary outcomes, the impact of each intervention is shown on an absolute event rate scale, which is easier to understand for patients and clinicians.6 In our examples, we used the ORs from the outcome-specific NMAs and a fixed event rate for placebo (control event rate, CER) to estimate the event rate of each intervention19:

Embedded Image

For continuous outcomes, we can follow a similar procedure if the NMA is performed on the natural scale, that is, using mean difference. In this case, we would simply use the mean difference for each intervention versus placebo estimated from the NMA, and a fixed mean outcome for placebo. If, however, the NMA is performed using the standardised mean difference (SMD), these can be converted in ORs as follow, for instance using the Hasselblad and Hedges’ method:20, e11

Embedded Image

When estimating CER from continuous data, the number of events in the population of patients allocated to the common comparator can be imputed using the baseline and endpoint continuous values and after setting a meaningful threshold (eg, 50% improvement from baseline).21 In all cases, the absolute estimate for the common comparator can either be derived from the data used in the NMA or supplied from an external source (ie, ‘real-world’ study).

In the following analyses, we used a mock data set to analyse five outcomes with corresponding NMAs for demonstration purposes. Figure 1 shows a plot where the overall profile of each active intervention and placebo across five outcomes is depicted as wedges and expressed as absolute estimates: response, remission, all-cause drop-out (acceptability), nausea and headache. Absolute estimates of both the intervention of interest and the common comparator are reported as grey squares and light blue circles, respectively, to contextualise the absolute estimates. The Vitruvian plots can then be merged in a synoptic chart to simultaneously visualise and navigate among all the available interventions.

Figure 1

Anatomy of the Vitruvian plot. The key elements of the Vitruvian plot and related terminology are hereby illustrated.

We applied the recommendations on the communication of benefit–risk information from the available literature.3–7,e3, e4 Each radial bar represents the magnitude of effect for an outcome and also displayed as a percentage, using both textual and graphical communication strategies. Uncertainty over the relative performance against the common comparator is reported as the strength of statistical evidence (figure 2) or as the confidence in the evidence (figure 3). Finally, each visualisation is accompanied by a supportive statement to facilitate the interpretation of the Vitruvian plots.

Figure 2

Synoptic chart showing strength of statistical evidence. The profiles of placebo and treatments 1 and 2 are compared across five domains: clinical response, clinical remission, acceptability (all-cause drop-out), nausea, and headache. A dark green colour indicates there is strong statistical evidence that the corresponding intervention performs better than placebo. A dark red colour indicates strong evidence that the intervention performs worse than placebo. Light green/red colours indicate weaker evidence of a beneficial/detrimental effect. Colours closer to yellow indicate an increasing lack of evidence on whether the intervention performs better or worse than placebo. The light blue colour identifies placebo as the common comparator. When presenting the Vitruvian plots to patients, we strongly suggest including both probabilities of events and non-events as well as the time frame. A suggested reporting statement for the ‘remission’ outcome and treatment 1 is: ‘Of 100 people with your clinical diagnosis, 15 may remit (1 more than with placebo) and 85 may not remit with treatment 2 over 8 weeks’.

Figure 3

Synoptic chart showing confidence in the evidence. The profiles of placebo and treatments 1 and 2 are compared across five domains: clinical response, clinical remission, acceptability (all-cause drop-out), nausea and headache. Darker hues of purple indicate increasing confidence in the evidence for the comparison of the corresponding treatment vs placebo (categorised in four levels, that is, very low, low, moderate, high quality). The light blue colour identifies placebo as the common comparator. When presenting the Vitruvian plots to patients, we strongly suggest including both probabilities of events and non-events as well as the time frame. A suggested reporting statement for the ‘nausea’ outcome and treatment 1 is: ‘Of 100 people with your clinical diagnosis, 14 may experience nausea (2 more than with placebo) and 86 may not experience nausea with treatment 2 over 8 weeks’.

How to colour the plot

We have implemented two alternative and fully customisable colour schemes: the strength of statistical evidence, considering uncertainty as done by Seo and colleagues,12 and the confidence in the evidence, using the Confidence in Network Meta-Analysis (CINeMA) ratings.22 The colour scheme can be easily personalised (eg, to a colour blind accessible scheme such as the Viridis or Okabe-Ito palettes or a custom one) by changing the hexadecimal colour values in the code (https://github.com/EGOstinelli/Vitruvian-plot).

Strength of statistical evidence

This approach indicates the relative performance of the intervention and the common comparator, taking into account uncertainty.12 Z-scores are calculated from estimated effect sizes and SEs from the NMAs, coherently with the direction of effect. In our example, these values are reported as colours: from green (the intervention is better than the comparator) to yellow (unclear whether the intervention performs better or worse than the comparator) and red (the intervention is worse than the comparator). The common comparator is coloured in light blue. Given that p values are more widely used to communicate findings, these are calculated from Z-scores and reported in the legend of the plot.

Confidence in the evidence

Approaches such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) and CINeMA rate how confident we are in the evidence supporting a certain finding.22 ,e12 Specifically, CINeMA evaluates the confidence in the results from NMAs across six domains: within-study bias, reporting bias, indirectness, imprecision, heterogeneity and incoherence. These domains are then considered jointly to express a summarising judgement (very low, low, moderate, high confidence), an approach widely used to communicate and summarise key findings.23 In our example, darker hues of purple indicate an increasing level of confidence in the evidence. The common comparator is coloured in light blue.

Involvement of members of the public and clinicians

We actively involved people with lived experiences of physical or mental illnesses, and clinicians in the conceptualisation of the Vitruvian plot to ensure that the final output was optimised for these users. As the tool was rapidly evolving, we intentionally collected their feedback throughout the development process via ad hoc multiple individual interviews. We evaluated the intelligibility of the Vitruvian plot with guided (ie, explanation provided by a member of the team) and unguided (ie, use of written captions only) interpretation tasks using fictional clinical scenarios. For instance, we asked users to comment and compare the fictional interventions depicted in the Vitruvian plots. This allowed us to highlight a number of issues with the early versions of the tool, and to use their feedback to guide the overall development process. Finally, we used the collected inputs to develop the suggested supporting statements.

Results

Visualisation of the strength of statistical evidence

Figure 2 shows clinical profiles of placebo and treatments 1 and 2 in terms of response, remission, all-cause drop-out (acceptability) and two adverse events, namely, nausea and headache. In this example, treatment 1 shows an overall higher probability of achieving response (35%, compared with 13% for placebo) and remission (15%, compared with 14% for placebo). The probability of dropping out due to any cause is lower compared with placebo (acceptability: 16% and 18%, respectively). Treatment 1 is associated with a higher probability of having headache and nausea (13% and 14%, respectively) compared with placebo (11% and 12%), with strong statistical evidence for response and nausea outcomes. Treatment 2 shows a promising overall pattern, but the probability to achieve response is only 20%. The statistical strength of the evidence is strong for all the outcomes except for nausea.

Visualisation of the confidence in the evidence

Figure 3 shows an alternative visualisation strategy for the same interventions and outcomes. In this example, it is evident how the confidence in the evidence for the comparison between treatment 2 and placebo is overall high, with the only exception of remission (low, 16%). On the other hand, ratings of the five outcomes are more heterogeneous for treatment 1, with moderate confidence for remission and nausea (15% and 14%, respectively), low confidence for response and acceptability (35% and 15%, respectively) and very low confidence for headache (13%).

Discussion

In this article, we have illustrated a novel way of presenting information for NMAs in a transparent, accessible and useful manner that can inform and support the decision-making process between patients and clinicians. This is instrumental to take full advantage of the available evidence .6 7 For instance, in our first example (figure 2), treatment 1 may be considered a strong candidate due to its response profile. Nonetheless, after considering the confidence in evidence (figure 3), treatment 2 might be a good candidate due to its high confidence ratings.

One of the strengths of the Vitruvian plot is the active involvement of people with lived experience of physical or mental diseases, and clinicians. This approach, combined with the involvement of multidisciplinary experts and recommendations from available literature, allowed us to overcome some limitations of the existing modalities to present results from NMAs. For instance, although the Vitruvian plot shares visual similarities with the spie chart with equally weighted outcomes (see section 3 of online supplemental file), the latter did not include a measure of confidence in the evidence when presenting the magnitude of effect, something that Daly and colleagues expressed interested in as a possible future development.11 The spie chart implemented ranking metrics, such as the surface under the cumulative ranking curve (SUCRA) values, as metrics of choice,11 similarly to the radar graph developed by Seide and colleagues to visually compare multiple drugs over an outcome.14 The feedback we received from patients and clinicians was instrumental to develop the Vitruvian plot. For instance, we implemented the level of confidence in the evidence and favoured absolute estimates over relative ones or ranking metrics. The use of absolute metrics is in line with currently available recommendations for the comparison of the chance of occurrence of two or more independent events.6 7

Absolute estimates have been used in the Kilim plot, together with the colour scheme we adopted in our first example (ie, strength of statistical evidence).12 Our approach differs from the Kilim plot in how we presented the metric of choice, combining textual information with corresponding visual elements. The interpretation of the magnitude of effect is conveyed by both textual and graphical formats, an optimal communication strategy when targeting populations with varying levels of numeracy and literacy skills.4–6 Although visual aids are not free from bias, such as an increase or reduction of risk-avoidant behaviour, well-developed visual formats have been proven effective to communicate health risks. For instance, Garcia-Retamero & Cokley found that adding visuals to written messages was more effective than written health information alone in improving accuracy in judgements and promoting changes in health behaviour related to sexually transmitted diseases in a high-risk population.24

Communication and understanding of uncertainty are ‘arguably essential for informed decision- making’.6 This is in line with the increasing support to avoid dichotomising findings according to the statistical significance (ie, statistically significant or not), especially for NMAs.25 26 Nonetheless, how to effectively implement this in the shared decision-making framework is a challenge yet to be solved. Only a small number of studies have evaluated the potential impact of communicating uncertainty in healthcare, suggesting that it might be associated with pessimistic risk perceptions and avoidance of reaching a decision (ie, ‘ambiguity aversion’).6 27 28,e13-e17 Communication of uncertainty may exert an effect on how people understand it, feel about it, and ultimately trust the information.29 A careful assessment of the impact on these domains is essential not only to prevent detrimental effects but also to reinforce and enhance behavioural changes and confidence with the tool. For instance, reporting uncertainty may be perceived as an example of transparent communication in research, thus increasing trustworthiness.30 Findings from a recent study on the impact of communicating uncertainty suggest that people adequately recognised and perceived uncertainty, with only a limited impact on the perceived reliability of numbers.30

The current version of the Vitruvian plot has some limitations. First, outcomes are not differentiated in terms of weight. Daly and colleagues explored non-equal weighting of outcomes.11 In their example, they used the regression coefficients from a multivariate regression model to tailor areas and ranks for each treatment to sex assigned at birth and age group. While this approach could be implemented in the current version of the Vitruvian plot, we aim at developing an interactive version in which outcomes can be optionally weighted according to the individual preferences of users. Second, the current version of the Vitruvian plot works best with a limited number of concurrent independent outcomes. Implementation of highly correlated outcomes is discouraged, as these will impact the overall profile of the intervention. Moreover, the number of outcomes presented should be carefully considered and balanced to present a comprehensive and informative synthesis of the available evidence in supporting practical decision-making. Third, continuous outcomes require conversion into dichotomous effect sizes when SMDs are used in the analysis. Finally, it is yet unclear whether this and competing methods of visualisation for multiple outcomes NMA provide a real practice benefit. The performance of these visualisation strategies in improving comprehension and better engaging users in decision-making should be carefully assessed.

As the interplay between patients, clinicians and evidence continues to rapidly evolve, the development of accessible formats to communicate critical information is of increasing importance. We hope that the implementation of the Vitruvian plots in real-world practice will support patients and clinicians in evidence-based shared decision-making. Future steps would be to develop an interactive version of the Vitruvian plot and to properly assess this new communication strategy to explore the relative performance and preferences of different components of the Vitruvian plots, such as format (ie, static vs interactive) and specific approaches (eg, statistical strength vs confidence in the evidence), across different fields of medicine and healthcare.

References e1–e17 are provided in online supplemental file.

Ethics statements

Patient consent for publication

Ethics approval

Not applicable.

Acknowledgments

We would like to acknowledge Mary Jane Attenburrow, Philip Cowen, Guy Goodwin, Orla MacDonald and Anneka Tomlinson for their interim feedback on the Vitruvian plot.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Twitter @EGOstinelli, @Toshi_FRKW, @And_Cipriani

  • Contributors EGO: conceptualisation, methodology, software, formal analysis, visualisation, writing—original draft, writing—review and editing. OE: methodology, formal analysis, writing—review and editing. HN, TAF, SL, GS: methodology, writing—review and editing. LW: writing—review and editing. CZ: investigation, visualisation, writing—review and editing. FDC: investigation, conceptualisation, writing—review and editing. KS, KSt, QL: investigation, writing—review and editing. AC: conceptualisation, methodology, supervision, writing—review and editing.

  • Funding This research was funded by the National Institute for Health Research (NIHR) Research Professorship to Professor AC (grant RP-2017–08-ST2-006). EGO and AC are supported by the National Institute for Health Research (NIHR) Research Professorship (grant RP-2017–08-ST2-006), by the Oxford Health Biomedical Research Centre (BRC-1215-20005), and by the National Institute for Health Research (NIHR) Applied Research Collaboration (ARC) Oxford and Thames Valley (grant NIHR200172). EGO, CZ, KS, and AC are supported by the National Institute for Health Research (NIHR) Oxford Cognitive Health Clinical Research Facility (award CRF-2016–10014). OE was supported by project grant number 180 083 from the Swiss National Science Foundation (SNSF). GS was supported by the Swiss National Science Foundation (grant/award number 179158).

  • Disclaimer The views expressed are those of the authors and not necessarily those of the United Kingdom National Health Service, the NIHR or the United Kingdom Department of Health.

  • Competing interests EGO reports personal fees from Angelini Pharma, outside the submitted work; and is a member of the International Early Career Researchers Advisory Board for the Evidence-Based Mental Health journal. HN reports grant support from The Health Foundation; research funding from the UK National Institute for Health Research; financial support from the Pharmaceutical Group of the European Union; paid advisory role at the BMJ—all outside of the submitted work. TAF is Deputy Editor for the Evidence-Based Mental Health journal. SL is Associate Editor for the Evidence-Based Mental Health journal. GS is Section Editor for the Evidence-Based Mental Health journal. FDC is a DPhil candidate at the University of Oxford, but he is also an employee of Boehringer Ingelheim International GmbH (from February 2022). AC reports personal fees from Italian Network for Paediatric Trials and CARIPLO Foundation; grants and personal fees from Angelini Pharma outside the submitted work; and is Editor for the Evidence-Based Mental Health journal.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.