Background All mental health professionals are encouraged to practise evidence-based medicine, but in an era of overwhelming research output, information management is key. Until now, no one has assessed the role of secondary journals, which aim to synthesise and present recent evidence, so as to promote evidence-based practice.
Objective We conducted a cross-sectional study via an online survey, to evaluate the quality of the content of Evidence-Based Mental Health (EBMH), as an example of a secondary journal, and the impact it has on evidence-based practice.
Methods We sent an online questionnaire to the commentators and the original study authors of all commentaries published in EBMH over the past 5 years (from 2011 to 2015, inclusive). The questions primarily concerned the quality of the included papers and their respective commentary, in addition to the ability of the commentaries to help disseminate research findings and promote evidence-based practice.
Findings We sent out 894 anonymous questionnaires and the overall response rate was 30%. The commentator and study author groups were largely homogeneous. Both groups were satisfied with the format and content of the commentaries, although over 60% of the authors were unaware of the commentary on their study before the survey. Notably, 80% of authors and 87% of commentators felt that the commentaries were useful in disseminating the findings of the original studies and implementing evidence-based practice.
Conclusions and clinical implications The commentators and original study authors view EBMH not as a vehicle for criticism, but instead as a trustworthy publication that crystallises important findings and presents them in digestible form with the aim of promoting key advances in mental health. Next, we aim to assess the extent to which the readership of this journal agrees.
Statistics from Altmetric.com
Evidence-based medicine (EBM) has been defined as the ‘conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients’.1 All mental health professionals have been encouraged to use EBM and move towards evidence-based practice (EBP).2 The implementation of EBP involves four steps: ‘first, framing a clear question based on a clinical problem; second, searching for relevant evidence in the literature; third, critically appraising the validity of contemporary research; and fourth, applying the findings to clinical decision-making’.3 ,4 This is no mean feat. A study, now over 30 years old, estimated that on average five clinical questions are raised at each bedside encounter with a patient.5 In addition, as new evidence emerges, healthcare workers are constantly required to update their knowledge. In 1999, clinicians spoke of surviving an ‘information flood’, as trends demonstrated that the number of new journals doubled every 10–15 years.6 In 2014, there were about 28 100 active scholarly peer-reviewed English language journals, collectively publishing 2.5 million articles a year, of which 30% were biomedical.7 Even restricting reading to high impact journals in a single field of interest, the number of articles is in the thousands. For this reason, healthcare workers chose to read so-called ‘secondary journals’, those which aim to highlight and summarise the recent evidence, methodological advances and possible clinical implications of research.8
There are three secondary journals in the field of mental health with large readerships: Current Opinion in Psychiatry, Harvard Review of Psychiatry and Evidence-Based Mental Health (EBMH), established in 1988, 1993 and 1998, respectively. Each adopts a systematic, comprehensive strategy to identify the best and most relevant new evidence for mental health workers, and incorporates a rigorous peer review process for their articles. However, there are differences between them. Current Opinion in Psychiatry and the Harvard Review of Psychiatry are both bimonthly publications aimed at psychiatrists, whereas EBMH is a quarterly journal and caters to all mental health workers. In terms of content, Current Opinion in Psychiatry presents commissioned articles with the views of experts in addition to recommended reading on the topic. The Harvard Review of Psychiatry contains reviews and expert perspectives, in addition to a clinical case with expert discussion. Up to 2014, EBMH similarly focused on providing accessible summaries and accompanying short commentaries on recent papers. However, under a new editorial team, EBMH now aims to arm readers with the skills to practise EBM in their own daily work. The commentaries remain, but have changed in structure. No longer including the abstract of the paper, there is more room for discussing the strengths, limitations and clinical significance of the paper. In addition, the commentaries are now accompanied by other types of articles with clear educational purpose. ‘Evidence-based case conferences’, for example, address a complex clinical question and illustrate how to attempt to solve it ‘pragmatically’, using an EBP approach.4 ‘Statistics in Practice’, on the other hand, helps readers acquire the statistical knowledge themselves required not just to understand, but also appraise the evidence, a necessary step in practising EBM.9 ,10
Previous studies have investigated the attitudes towards EBP among various healthcare professionals.3 However, to the best of our knowledge, no one has tried to determine whether a secondary journal, such as EBMH, has real potential to promote EBP. In this study, we aim to evaluate the quality of the content of this journal, and the impact it has on EBP, by collecting questionnaire data from the authors of the papers selected for the commentaries, and the commentators themselves.
Between September and December 2015, we sent an online questionnaire to the commentators and the original study authors of all commentaries published in EBMH over the past 5 years (from 2011 to 2015, inclusive). This time period was defined a priori, because we considered it a sufficient number of issues for an adequate sample of authors and commentators (on average, 90 commentaries are published every year). Moreover, with data from before and after 2014, we are able to investigate any differences following the change in commentary format. The study sample included all the corresponding authors of studies presented with a commentary in one of the recent issues of EBMH (the ‘authors group’) and the commentators of the same studies (the ‘commentators group’). Each group contained 447 individuals.
We developed a 9-item questionnaire for the authors’ group and an 11-item questionnaire for the commentators’ group (box 1). Both included questions about the responders' professional role and place of work. We defined a researcher as someone who has no clinical role, but acknowledge that psychiatrists and psychologists may also be active in research. In addition, both groups were asked about the structure of the commentaries and their ability to disseminate research findings. Questions specific to the commentators included those about the quality of the original study, their motivation for writing the respective commentary, rating of the peer review process and suggested improvements for the journal. Questions for the authors related to their prior awareness of the commentary on their study, and its quality. Some questions were rated using a Likert five-point scale (‘definitely yes’, ‘probably yes’, neutral, ‘probably no’ and ‘definitely no’), whereas others were open questions.
List of questions for the authors and commentators (from the online survey)
Original study authors’ questions:
1. Please indicate how you came across this commentary.
2. Did you read it?
3. How did you have access to this commentary?
4. Do you consider this commentary a proper summary/comment of your paper?
5. Do you think that the structure of the commentary is appropriate?
6. In your opinion, can this kind of commentary help spread the study among general mental health practitioners?
7. Which of the following best describes you?
8. Please indicate your highest degree.
9. Where do you work?
1. Would you have chosen to comment on the same study?
2. Do you believe that this paper has a real value in its field?
3. Why did you accept to write this commentary?
4. Did you disseminate/send the commentary to colleagues or other people interested?
5. Do you think that the structure of the commentary is appropriate?
6. In your opinion, can this kind of commentary help spread the study among general mental health practitioners?
7. Do you think that the peer-review process has helped to improve the quality of your commentary?
8. What else would you like to see in Evidence-Based Mental Health as commentator?
9. Which of the following best describes you?
10. Please indicate your highest degree.
11. Where do you work?
Procedure and analysis
The questionnaires were sent by email as an online survey (developed using the SurveyMonkey software package) accompanied by an introductory letter stating the purpose of the study and promising confidentiality. The authors’ group was sent the EBMH commentary of their paper as an attachment. Fulfilment of the survey was considered consent to participate in the study, and all responses were anonymous. The individuals were given 1 week to complete the questionnaire, with non-responders receiving a reminder and a week-long extension, followed by a final reminder. Data analysis was conducted using Microsoft Office Excel 2013 calculating frequency and corresponding percentages to describe the responses to the survey questions. The Likert scale was dichotomised for the purposes of the analysis. A self-rating report of either ‘definitely yes’ or ‘probably yes’ was regarded as a favourable answer, while ‘neutral’, ‘probably no’ or ‘definitely no’ was viewed as an unfavourable answer. Answers to open questions were grouped according to theme to allow for analysis. In order to assess the impact of the change in commentary format, the responses were divided into those relating to issues between February 2011 and August 2014 (old format), as compared with issues between November 2014 and November 2015 (new format).
The overall response rate was 30%, with 25% of authors and 35% of commentators completing the questionnaire. About one-third of responses from each group related to the new format (after November 2014). Table 1 describes the characteristics of the participants. The two groups were largely homogeneous, although there were proportionally more researchers in the authors’ group, and psychologists in the commentators’ group. Approximately 60% of all the participants had a PhD, and 50% only work in academia. Less than half of the authors were aware of the commentary on their study. A small number were told about it by the commentator, while others found out from colleagues or via Web of Science. Few, however, had read the commentary prior to receiving our email, and just 5% of authors reported reading EBMH regularly. Eighty-seven per cent of the commentators said they would probably or definitely choose to comment on the same study again. Indeed, some 30% gave promoting the paper as a reason for writing the commentary. Forty-seven per cent felt the paper definitely had real value in its field, and a further 38% felt it probably did. Reasons given (which could be multiple) included relevance and originality of the topic (47%), findings and implications (40%) and methodology (23%). Expertise of the author was rarely cited. Fifteen per cent of commentators gave an unfavourable response about the paper, with no consensus evident in the reasons given, and just 3% said they wrote the commentary to express criticism of the paper. Many other reasons for writing were given by the commentators. In addition to those already mentioned, 76% said they were interested in the topic and wished to add their point of view and 56% said they appreciate EBMH and were delighted to write a commentary. Just 7% did so because they feel obliged to participate in the evaluation and dissemination of new research, and 4% because they never say no to an offer of a publication. Only one-third of commentators had sent their commentary to colleagues or others, although a further one-fifth would like to. For questions that concerned the structure, content and role of the commentaries, we compared the responses for commentaries before November 2014 to those after. Table 2 includes the authors' responses to the question: ‘Do you consider this commentary a proper summary/analysis of your paper?’ Seventy-seven per cent of authors gave favourable responses for the old format, increased to 88% for the new format. Comparing professional roles, psychiatrists gave more unfavourable responses. Psychologists and researchers were more positive, especially for commentaries in the new format. Comparing places of work, those in academia only gave more positive responses for the new format. Table 3 contains the authors' and commentators' ratings of the structure of the commentaries. Overall, 89% and 94% of the authors and commentators, respectively, give a positive rating. There is little difference between the new and old formats in either group. Both groups were asked to what extent they feel the commentaries help disseminate the findings of the original paper, with the results given in table 4. Overall, 80% of authors and 87% of commentators felt that the commentaries were useful, and there was no substantive difference between the old and new formats. Finally, the commentators were asked whether they thought the peer-review process had helped improve the quality of the commentary. Eight-five per cent gave a favourable response; reasons given included adding rigour and the importance of feedback. A small number of commentators felt the process led to censoring and too many revisions. They were then asked: ‘What else would you like to see as an EBMH commentator?’. Thirty-five per cent of commentators answered this open question, with 40% of those saying ‘nothing more’. Of the remainder, many asked for the inclusion of specific topics, while general suggestions included open access, adding commentator's biographical data, increased focus on study impact and links to resources to help translate findings into practice.
To the best of our knowledge, this is the first systematic survey about the quality and impact of secondary journals in mental health. We were pleased with the overall response rate of 30% for an email-based questionnaire. Indeed, a response rate of one quarter from the authors, despite the majority being previously unaware of the commentary on their study, may reflect the personalisation of the email invitations.
We found that the characteristics of the participating authors and commentators were largely similar. Both groups had a majority of psychiatrists, but also included psychologists and researchers. As might be expected, researchers are more represented in the authors’ group than in the commentators’ group. However, we do not interpret these distinctions too definitively, because there is a high likelihood of overlapping roles among participants. In both groups, over 60% of responders had a PhD and half worked in academia only. It would be interesting to compare this to the readership of EBMH. Given the explicit aim of the journal to help readers with EBP in their daily work, we might expect fewer readers to work in academia only. The vast majority of commentators gave positive responses about the content of the original paper. Thus, we can conclude that EBMH is not primarily a vehicle for criticism, but instead aims to promote high-quality papers. Despite this, less than 40% of authors had prior knowledge of the commentary on their paper. A few had been informed by the commentator, and while this is not currently routine practice at EBMH, it may be beneficial in promoting the commentaries and inviting further comment and debate. In addition, just one-third of commentators shared their commentary with colleagues, but a further one-fifth wanted to. EBMH could do more to facilitate this, perhaps using social media, as increasing numbers of researchers and clinicians use this as a platform for information sharing and discussion.11 We were eager to see how the new format of the commentaries compared with the old. With 94% of commentators and 88% of authors giving a favourable response about the structure of the old format, there was little room for improvement. However, we were pleased to see that the overall ratings were as good if not better for the new style (94% positive for both groups). Similarly, over 80% of responders in both groups felt that the commentaries in the old format were useful in spreading the findings of the original paper, and this was maintained for the new format. We acknowledge the risk of a response bias, particularly in the commentators’ group, who are by definition a group willing to contribute to a secondary journal. However, this would be no different for the old and new formats. The authors of the original studies, who are at less risk of bias, were also asked to rate the content of the commentaries. Again, the old format was compared with the new format, and there was a 10% increase in positive rating (from 77% for the old format, to 88% for the new format). Comparing the responses by professional role and place of work, we found psychiatrists as a whole to be less positive about the ability of the commentaries to spread findings, as well as authors working only in the National Health Service. We speculate that this may be due to an increased burden of information ‘management’ for practising clinicians, and a sense of pessimism about the ability to keep up to date with new studies.
Two of the authors of this paper are also editors of EBMH. We have tried to avoid any bias in the collection, interpretation and presentation of the data. We have kept a copy of all the questionnaires and related correspondence, and hope that in the future our survey will be updated independently of EBMH. In the meanwhile, we are encouraged by the largely positive responses from the authors and commentators relating to the structure, content and usefulness of the commentaries. There was little room for the new format to improve on the old format, but it appears to be viewed at least as positively in most groups. Twenty per cent of commentators suggested improvements to the commentaries. Most were requests that a particular topic or subspecialty receive more coverage in the journal. Of course, individual preferences and interests are likely to vary among the contributors and readers of EBMH, but we want to ensure that the journal covers a wide range of mental health issues. As well as taking these suggestions on board, we will look at ways of adopting further suggestions in the future. The next step for the editorial board is to survey the readers of EBMH, to determine their views on the commentaries, and the real-world impact that a journal such as this can have on their EBP.
The authors would like to thank Professor Stefan Leucht and Professor Gin Malhi for handling the peer-review process of this paper.
Competing interests AC and TAF are the Editor-in-Chief and the Deputy Editor of Evidence-Based Mental Health, respectively.
Provenance and peer review Commissioned; internally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.