Yves here. Educated reader have been trained to treat peer reviewed papers with far more respect than presumed-to-be-not-verified research. Below, KLG explains why the peer research standard was never quite what it aspired to be and has deteriorated under money pressures at publishers and financial conflicts of interest of investigators.<\/p>\n
KLG\u2019s overview:<\/p>\n
Peer review is the \u201cgold standard\u201d (I really dislike that locution, almost as much as \u201cdeliverable\u201d) that proves the worth of scientific publication. It has never been perfect. Nor has it ever been without controversy. The original contributors to the first scientific journal, Philosophical Transactions of the Royal Society, took a while to get it right. But overall peer review has served science well.<\/p>\n
However, it has become strained over the past 25+ years as the business of scientific publishing has hypertrophied beyond all reason (except for making money). This post describes a few recent failures of peer review. My sense, from reading the literature and the CV\u2019s of scientists applying for jobs and promotion, is these examples are quite common.<\/p>\n
While there can be no single cure for the problem, bringing peer review out of the shadows of anonymity, where poor work and worse can hide, is the best solution currently imaginable. In the meantime, read the \u201cpeer reviewed\u201d scientific and other scholarly literature with care.<\/p>\n
One on my most difficult tasks in my day job is convincing medical students that just because something is published in a peer reviewed journal does not mean it presents the truth so far as we can know it. At times published papers approach Harry Frankfurt\u2019s conception of bullshit: The authors don\u2019t care if the article is true or not, only that it gets published. A few guidelines are included for the general reader.<\/p>\n
By KLG, who has held research and academic positions in three US medical schools since 1995 and is currently Professor of Biochemistry and Associate Dean. He has performed and directed research on protein structure, function, and evolution; cell adhesion and motility; the mechanism of viral fusion proteins; and assembly of the vertebrate heart. He has served on national review panels of both public and private funding agencies, and his research and that of his students has been funded by the American Heart Association, American Cancer Society, and National Institutes of Health<\/p>\n
The first question asked when discussing a scientific publication is, \u201cHas this paper been peer reviewed?\u201d\u00a0 If the answer is \u201cno,\u201d then true or not, right or wrong, the paper has no standing.\u00a0 During the first years of the HIV\/AIDS epidemic this was never a question.\u00a0 Those of us in the lab awaited every weekly issue of Nature and Science and biweekly issue of Cell to learn the latest.\u00a0 There were a few false starts and some backbiting and competition and the clash of titans about who discovered HIV, but the authority of these publications was seldom in doubt.\u00a0 It was a different era.<\/p>\n
Forty years later biomedical publishing has outgrown itself (i.e., 436,201 Covid publications in PubMed as of 27 August 2024 and no real signs the pandemic is \u201cresolved\u201d), especially as publishers new and old have taken advantage of the internet to \u201cpublish\u201d journals online across the scientific spectrum.\u00a0 On the one hand, online open access has been a boon to scientists and their readers with more outlets available.\u00a0 On the other, it has often become impossible to distinguish the wheat from the chaff, as all indications are that peer review has suffered as a concomitant of this growth.\u00a0 The general outlines of this change in the scientific literature were covered earlier this year.\u00a0 Here I want to illustrate how this manifestation of Gresham\u2019s Law has influenced peer review, which can be defined as the anonymous, reasoned criticism of a scientific manuscript or grant application by expert peers who are equipped to do so.\u00a0 I have been the reviewer and the reviewed since the mid-1980s, mostly with fair results in both directions (but there is one grant reviewer I still would like to talk to if given the chance).\u00a0 Now that my focus has changed, I do not miss it too much.<\/p>\n
One of the larger \u201cnew\u201d publishers with more than 400 titles is MDPI, which has had two names beneath the one abbreviation: First Molecular Diversity Preservation International that began as a chemical sample archive and now Multidisciplinary Digital Publishing Institute.\u00a0 MDPI journals cover all fields of science.\u00a0 Papers are reviewed rapidly and made available online quickly.\u00a0 The final \u201cproduct\u201d is indistinguishable in pdf from those of legacy journals that go back to the Philosophical Transactions of the Royal Society (1665).\u00a0 What follows is a summary of one scientist\u2019s experience as an MDPI reviewer.\u00a0 As we have discussed here in another context, sometimes n = 1 is enough.<\/p>\n
Rene Aquarius, PhD, is a postdoctoral scientist in the Department of Neurosurgery at Radboud University Medical Center in Nijmegen, The Netherlands.\u00a0 He recently described his experience as a reviewer for a special edition of the MDPI Journal of Clinical Medicine.\u00a0 One should note here this use of \u201cspecial edition.\u201d\u00a0 These expand the market, so to speak, for the publisher and its contributors, so they are common among many open access digital publishers.\u00a0 Titles used by these publishers also mimic those of legacy journals.\u00a0 In this case that would be the Journal of Clinical Investigation (1924), which has been the leading journal in clinical medicine for 100 years.\u00a0 It is published by the American Society for Clinical Investigation (ASCI).\u00a0 Arguments from authority are not necessarily valid as COVID-19 revealed early and often, but the ASCI has earned its authority in clinical research.<\/p>\n
In November 2023, upon reading the manuscript (a single-center retrospective analysis of clinical cases) Dr. Aquarius immediately noticed several problems, including discrepancies between the protocol and final study, a target sample size larger than what was used, and a difference in minimum age for patients between the protocol and the manuscript.\u00a0 The statistical analysis was faulty in that it created a high probability of Type I errors (false positives) and the study lacked a control group, \u201cwhich made it impossible to establish whether changes in a physiological parameter could really predict intolerance for a certain drug in a small subset of patients.\u201d\u00a0 Dr. Aquarius could not recommend publication.\u00a0 Reviewer 2 thought the paper should be accepted after minor revision.<\/p>\n
The editorial decision was \u201creject, with a possibility of resubmission after extensive revisions.\u201d\u00a0 These extensive revisions were returned only two days after the immediate rejection; my revisions, extensive or not, have usually required several weeks at a minimum.\u00a0 Before he could begin his review of the revised manuscript, Dr. Aquarius was notified that his review was no longer needed because the editorial office already had enough peer reviewers for this manuscript.<\/p>\n
But Dr. Aquarius reviewed the revision anyway and found it had indeed undergone extensive revisions in those two days.\u00a0 As he put it, the \u201cbiggest change\u2026was also the biggest red flag. Without any explanation the study had lost almost 20% of its participants.\u201d\u00a0 And none of the issues raised in his original review had been addressed.\u00a0 It turned out that another peer reviewer had rejected the manuscript with similar concerns.\u00a0 Two other reviewers accepted the manuscript with minor revisions.\u00a0 Still, the editor rejected the manuscript after fifteen days, from start to finish.<\/p>\n
But the story did not end there.\u00a0 A month later Dr. Aquarius received an invitation to review a manuscript for the MDPI journal Geriatrics.\u00a0 Someone in the editorial office apparently goofed by including him as a reviewer.\u00a0 It was the same manuscript that must have been shifted internally through the transfer service of MDPI.\u00a0 The manuscript had also reverted to its original form, although without the registered protocol and with an additional author.\u00a0 Analysis of patient data without formal approval by an Institutional Review Board or equivalent is never acceptable.\u00a0 Dr. Aquarius rejected the paper yet again, and shortly thereafter the editor decided to withdraw the manuscript.\u00a0 So far, so good.\u00a0 Peer review worked.\u00a0 And then in late January 2024, according to Dr. Aquarius, the manuscript was published in the MDPI journal Medicina.<\/p>\n
How and why?\u00a0 Well, the article processing charge (APC) for Journal of Clinical Medicine was 2600 Swiss Francs in 2023 (~$2600 in August 2024).\u00a0 The charges were CHF 1600 and CHF 2200 for Geriatrics and Medicina, respectively.\u00a0 Nice work if you can get it, for the authors who got a paper published and the publisher who collected several thousand dollars for their trouble.\u00a0 \u201cPixels\u201d are not free but they are a lot cheaper than paper and ink and postage.\u00a0 But what does this example, which as a general proposition is quite believable, say about MDPI as a scientific publisher?<\/p>\n
Another recent case of suspect peer review was made public immediately after publication earlier this year in the journal Frontiers in Cell and Developmental Biology.\u00a0 Frontiers is \u201cWhere Scientists Empower Society, Creating Solutions for Healthy Lives on a Healthy Planet.\u201d\u00a0 Frontiers currently publishes 232 journals, from Frontiers in Acoustics to Frontiers in Water.\u00a0 The paper in question was entitled \u201cCellular functions of spermatogonial stem cells in relation to JAK\/STAT signaling pathway,\u201d a recondite title for a general sudience but a topic of interest to any cell biologist working on stem cells or signal transduction.<\/p>\n
\nAs with the previous example, the time from submission to acceptance was relatively short, from 17 November to 28 December 2023.\u00a0 No more than two days after publication a firestorm erupted and the publication was withdrawn by the publisher soon after.\u00a0 It turned out the paper was very likely written using ChatGPT or equivalent Algorithmic Intelligence (AI).\u00a0 It was nothing but twelve pages of nonsense, with reasonable-sounding text at first glance but figures undoubtedly drawn by AI that were nothing but pure gibberish.\u00a0 The paper itself has vanished into the ether, apparently deleted without a trace by the publisher.\u00a0 In anticipation of this I saved a pdf and would share if there were an easy mechanism to do so.\u00a0 This link gives a general sense of the entire ridiculous event, with illustrations.\u00a0 The AI drawing of the rodent is nonsensical and NPSFW, not particularly safe for work.\u00a0 The thing is, this manuscript passed peer review, and the editor and peer reviewers are listed on the front page.\u00a0 They all agreed that this Figure 2 is legitimate science.\u00a0 The other figures are just as ridiculous.<\/p>\n
![\"\"](\"https:\/\/www.nakedcapitalism.com\/wp-content\/uploads\/2024\/08\/00-peer-review.png\")
<\/p>\n
How could this have gotten through anything resembling good editorial practice and functional peer review?\u00a0 The only answer is that it was passed through the process without a second look by the editor or either of the two reviewers.\u00a0 So, is n = 1 enough here, too, when it comes to Frontiers journals?\u00a0 No one will get any credit for \u201cCellular functions of spermatogonial stem cells in relation to JAK\/STAT signaling pathway\u201d because it has been scrubbed.\u00a0 But the paper reviewed and rejected for apparent good reason by Dr. Aquarius has been published.\u00a0 More importantly it will be counted.\u00a0 It will also still be faulty.<\/p>\n
Is there an answer to this crisis in peer review? [1]\u00a0 Yes, for peer review to function properly, it must be out in the open.\u00a0 Peer reviewers should not be anonymous.\u00a0 Members of the old guard will respond that younger (i.e., early career) peer reviewers will be reluctant to criticize their elders, who will undoubtedly have power by virtue of their positions.\u00a0 This is not untrue, but well-reasoned critiques that address strengths and weaknesses of a manuscript or a grant application will be appreciated by all, after what would be a short period of adaptation. This would also level the \u201cplaying field.\u201d<\/p>\n
Of course, this requires that success rates for grant applications rise above the10-20 percent that is the current range for unsolicited investigator-initiated applications to the National Institutes of Health (NIH).\u00a0 In a game of musical chairs with career implications, good will and disinterestedness cannot be assumed.\u00a0 In my long experience it has become clear that the top third of a pool of applications should get funded because there are no objective distinctions among them, while the middle third should get funded upon revision.\u00a0 The latter third will remain hopeless for the duration and are not reviewed by the full panel.\u00a0 In any case, the data are clear that NIH grants in the \u201ctop 20%\u201d are indistinguishable in impact measured by citation of the work produced and papers published.\u00a0 I anticipate that this would extend to 30% if the authors of this paper repeated their analysis with a more current dataset.\u00a0 For those interested in a comprehensive treatment of modern science, this book by the authors of this paper is quite good, but expensive.\u00a0 Ask your local library to get it for you!<\/p>\n
A recent article on non-anonymous peer review was written by Randy Robertson, an English Professor at Susquehanna University: Peer review will only do its job if referees are named and rated (registration required).\u00a0 Of all things, the paper that led him down this path was one that stated \u201cerect penile length increased 24 percent over the past 29 years.\u201d\u00a0I think Professor Robertson is correct on peer review, but he also inadvertently emphasized several deficiencies in the current scientific literature.<\/p>\n
What did Professor Robertson find when he read this \u201csystematic review and meta-analysis\u201d?\u00a0 The authors claim they included studies in which the investigators did the measurements, but three of the largest studies included papers in which the measurements were \u201cself-reported.\u201d\u00a0 Yes, I laughed at \u201cself-reported,\u201d too.\u00a0 Neither were the method(s) of measurement described.\u00a0 I don\u2019t even want to think about that.\u00a0 Robertson wrote to the editors and corresponding authors, who acknowledged the problems and stated they would revise the article.\u00a0 After months the correction has not been published.\u00a0 The World Journal of Men\u2019s Health, new to me, is published by the Korean Society for Sexual Medicine and Andrology, which may be organized only for the publication of this journal.\u00a0 The authors, however, are from universities in Italy, Milan and Rome (Sapienza), and Stanford University and Emory University in the United States.\u00a0 Heady company.<\/p>\n
This paper seems frivolous, but perhaps it is not.\u00a0 However, it does not support what it purports.\u00a0 Professor Robertson also misses the mark when he states that a meta-analysis is the \u201cgold standard\u201d in science.\u00a0 It may be the standard in Evidence-Based Medicine, but reviews and meta-analyses are by definition secondary, at least once removed from primary results.\u00a0 The question raised here is whether caveat lector must be our guide for reading the scientific literature, or any scholarly literature, when publish-or-perish along with publish-and-still-perish, rule? This is not tenable.\u00a0 If each reader must also be a peer reviewer, then peer review has no meaning.<\/p>\n
Robertson is correct when he states that effective review before publication is superior to post-publication \u201ccuration\u201d online, which will leave us \u201cawash in AI-generated pseudo-scholarship.\u201d\u00a0 See above for an egregious example. \u00a0Good refereeing is not skimming so you can get back to your own work or rejecting a submission because you do not like the result, or that it encroaches on your territory.\u00a0 Good refereeing means \u201cembracing the role of mentor\u201d and \u201cbeing generous and critical\u2026it is a form of teaching.\u201d\u00a0 This is certainly an academic and scholarly ideal but also worth remembering as the minimal requirement for legitimate publication of scientific research.<\/p>\n
The biggest problem with peer review, aside from the fact that it is unpaid labor without which scholarly publishing could not exist, is that reviewing is unrewarded professionally and will remain so as long as it is anonymous.\u00a0 The stakes must be raised for reviewers.\u00a0 Frontiers journals do identify reviewers and editors, but it didn\u2019t matter in the paper discussed above.\u00a0 When reviewers are known they will get credit, if not payment for services, and the entire process will become transparent and productive.\u00a0 It would also weed out the lazy, ineffective, and malicious.\u00a0 This would be a good thing.\u00a0 When high-quality reviews are recognized as the scholarship they are instead of mere \u201cservice\u201d to the profession they become \u201can integral part of scholarly production, if book reviews merit a distinct CV section, so do peer reviews.\u201d\u00a0 Would we be better off with a slightly slower science?\u00a0 The question answers itself.\u00a0 It is better to be right than first.<\/p>\n
Finally, what is a layperson to do when reading the peer-reviewed scientific and other scholarly literature?\u00a0 Several rules of thumb come to mind that can improve our \u201cspidey sense\u201d about published science\u201d:<\/p>\n
Read the acknowledgements. If the paper is biomedical or energy science, then how it was funded is critical.
\nNote the time from submission to publication. If this is less than 6-8 weeks, caveat lector, indeed.\u00a0 Good editing and good reviewing take time, as does the analysis of images for evidence of manipulation (See, for example, Lesn\u00e9, Sylvain).
\nIdentify the publisher. We do this all the time in our daily life.\u00a0 In These Times and the Wall Street Journal are predictable and useful.\u00a0 It is just as important in reading the scientific literature to know the underlying business model of the publication.\u00a0 Established legacy scientific publishers are not perfect, but they have survived for a reason.\u00a0 Journals published by widely recognized professional organizations are generally reliable.
\nDo not automatically reject recent open access publishers, but remember the business model, again. Many of them exist primarily to collect article processing fees from scientists whose promotion and tenure committees can do nothing but count.\u00a0 This matters.<\/p>\n
In the long run, either science for science\u2019s sake will return or we will continue wandering in this wilderness.\u00a0 We would make the most progress by de-Neoliberalizing science and its publication along with everything else that has Undone the Demos. \u00a0Suggestions are welcome.<\/p>\n
Notes<\/p>\n
[1] I am acutely aware that peer review has not always been fair.\u00a0 I have seen too much.\u00a0 But the breaches have been exceptions that prove the rule.\u00a0 Those who are found out are eventually ignored.\u00a0 I have reviewed for a dozen legacy journals (full disclosure: I have reviewed one paper for a Frontiers journal, which will be my last) and served on and chaired a review panel for a well-known non-governmental funding agency for more than ten years.\u00a0 I know that group cared deeply about being fair and constructive.\u00a0 Despite my frequent failures, I believe that most panels are fair.\u00a0 The problem is that careers sometimes perish before things even out.<\/p>\n
![\"Print](\"https:\/\/cdn.printfriendly.com\/buttons\/print-button-gray.png\")
<\/div>\n<\/div>\n