No, I Have Not Discredited Peer Review.

Peer-review is as strong as the peers and their review methods

Photo by Will van Wingerden on Unsplash

Today, I did a foolish thing and searched my own name on Twitter to see what the various versions of me that ideologues have created are up to now. Instead, I saw a number of people claiming that I, along with James Lindsay and Peter Boghossian, have thoroughly discredited peer review. This came as a result of Joe Rogan trying to get virologist, Peter Hotez, to have a debate with anti-vaccination campaigner, Robert. F. Kennedy Jr., after which epidemiologist, Elizabeth Jacobs, tweeted that Mr. Kennedy should publish his research in a peer-reviewed journal instead. I was tagged into this and responded that Dr Jacobs’ suggestion was not helpful or realistic, but neither was Mr. Rogan’s. I suggested the latter instead put the large sums of money he was offering into taking Dr. Hotez up on his offer to visit his laboratory and see what went on there. This would give Dr. Hotez the time and space to respond to vaccine skeptics’ main points of concern with the precision and detail required of a scientist in a visual format that could be more accessible to the layperson. It was later that I saw that, by many people, I am considered to have helped thoroughly discredit peer review.

I assure you, I have done nothing of the kind. This is a misconception of what we did and one that we asked people not to make at the time.

We managed to get seven shoddy, absurd, unethical and politically-biased papers into respectable journals in the fields of grievance studies. Does this show that academia is corrupt? Absolutely not. Does it show that all scholars and reviewers in humanities fields which study gender, race, sexuality and weight are corrupt? No. To claim either of those things would be to both overstate the significance of this project and miss its point. Some people will do this, and we would ask them not to. The majority of scholarship is sound and peer review is rigorous and it produces knowledge which benefits society.

We know that the peer-review system, which should filter out the biases that enable these problems to grow and gain influence, is inadequate within grievance studies. This isn’t so much a problem with peer review itself as a recognition that peer review can only be as unbiased as the aggregate body of peers being called upon to participate. The skeptical checks and balances that should characterize the scholarly process have been replaced with a steady breeze of confirmation bias that blows grievance studies scholarship ever further off course. This isn’t how research is supposed to work.

It is very important to understand that it is not that peer-review itself is a flawed system, but that it is a system that can develop flaws which need to be weeded out. This is because peers are humans and humans are flawed.It is essential not to allow the perfect to be the enemy of the good. No system within which humans produce knowledge can ever be perfect because humans are not perfect. We have to work things out as we go along and sometimes we make mistakes. A good academic system is not one in which scholars never make mistakes, but one which positively encourages them to look for and find each other’s mistakes and weed them out of the body of knowledge. That requires interdisciplinarity, viewpoint diversity, sincere attempts to mitigate bias and a collegial spirit with an expectation that one scholar will respond to another’s criticism or correction with appreciation or at least tolerance (one may fume internally) and never with any attempt to punish or silence the critic. This requires a multi-layered system of self-correction within disciplines. It requires peer review.

There is no other system that has the power to do this. Only humans can evaluate the scholarship of other humans. There is no academic excellence AI that can review scholarship objectively from some position of omniscience. Publishing work unreviewed would result in a proliferation of what would essentially be blogs with no fact-checking or methodological checks at all. It would be difficult enough for scholars to separate the wheat from the chaff and the layperson would have almost no hope at all. Without that review process, it is unclear how a body of knowledge could build upon itself and grow.

We also need peer review to be good, of course, and sometimes it is not good. This can be due to a number of factors. Sometimes it is because journals make poor choices for acceptance criteria by, for example, only publishing studies that have positive findings thus exacerbating the replication problem. Sometimes they are inconsistent in their criteria or too narrow in their scope. Sometimes, they have too few reviewers and the ones they have are over-worked and careless. These are logistical problems which can and be addressed and academics within those fields are already doing so.

Less forgiveably, some scholars can be dishonest, mostly to advance their own careers but sometimes for a political agenda and occasionally for money. This last one is particularly egregious but it gives every appearance of also being rare. Equally worrying is when a journal is overtaken by ideological bias or open to including it or afraid of publishing anything that goes against it. This is a real problem and the reason the Journal of Controversial Ideas now exists. If you are concerned about this, you are right to be and should join the effort to support scholars addressing ideological bias within their own discipline. If those disciplines are going to self-correct, it will need those rigorous scholars to be in positions of peer review, so they need all the support they can get to access them.

Paradoxically, it is when a discipline is self-correcting that it could look most like it is failing. This is counterintuitive but it is the same principle as when a new discovery is made in science that has ramifications for a particular theory and then many existing papers with hypotheses provisionally accepted as true have to be revised and updated in light of the new evidence. People who are inclined to be critical of science and have not grasped that this is a feature of its methodology, not a bug, may then say, “Look, science has been proved to be wrong again? Why should we trust it?” The fact that the scientific method has built into itself, not only a means by which it can discover itself to be wrong but a methodology based on trying to do so (falsifiability) is why you should trust it (provisionally, of course) and why it has been such a reliable source of knowledge production. In contrast, at the postmodern heart of so many Critical Social Theories is a rejection of objective truth and, consequently, many of its claims are unfalsifiable. If something cannot be falsified, it can never be found to be wrong but we also little reason to trust that it is right.

This is important because it reveals the problem with using our project within the realm of Critical Social Justice to assert that peer review has been thoroughly discredited within the discipline of epidemiology. There might be a problem with peer review within epidemiology, but, there is no reason to think so based on our exploration into identity and cultural studies. This is because the issue is not the system of peer review itself but the criteria upon which the peers within any system review scholarship.

What we found was that the criteria on which our papers were evaluated were various points of strict adherence to Critical Social Justice ideology. (The Journal of Poetry Therapy does not deserve inclusion here as its criteria were more artistic and open to a variety of approaches). As long as we had achieved that, it did not matter that our papers bore little resemblance to reality, were both silly and horrible, had no clear methodology, failed to provide any evidence or coherent data, made very little sense and denigrated large swathes of the population. Only one reviewer ever noticed that, in one of our papers that had bizarre data, that the data itself was highly implausible and that the conclusions we had drawn from it were not remotely indicated. This, then, was a very particular kind of problem specific to journals that published Critical Social Justice Scholarship. There is no reason to think that epidemiology is having this kind of problem, nor have any of the scientists who have contacted me to say CSJ has been creeping into their field been epidemiologists.

There have been problems with shoddy or fraudulent papers getting past peer review in science journals, but the causes of this have been different and so too have the responses of the journals. The most excellent microbiologist, Dr. Elisabeth Bik, has been using her remarkably sharp eyesight combined with technology to discover duplicated or manipulated images in published papers within her own field. So far, she has achieved 938 Retractions, 121 Expressions of Concern, and 957 Corrections and significantly improved the body of scholarship within microbiology. The reason Bik was able to bring about this tidy up, (apart from seemingly being some kind of awesome human microscope), is because microbiology journals not only accept that objective truth exists but have ‘being true’ as an essential criterion for publication. When presented with evidence that slides on a paper have been falsified, they will retract it, (although too many will be slow due to embarrassment). Attempting to present evidence to a journal publishing Critical Social Justice scholarship that any claim within it is not true is most likely to result in being told to decolonise your way of knowing and suspected of white supremacy. No retractions will be made.I have now heard from people working in two of the journals that published our rubbish that, far from incentivising them to be more rigorous in their scholarship,, they have instead become more rigorous in identifying people submitting papers.

The motivations for shoddy scholarship are different too. The main motivation for the prolific petty fraud in microbiology appears to be pressure on junior academics to publish frequently in a highly competitive space. Bik’s main proposed solution is adding image analysts to peer review. No such practical solution presents itself in the case of Critical Social Justice, because our papers did not get in by mistake. The journals truly thought they were good scholarship and still would if they had been submitted by somebody sincere. The problem is ideological capture and the solution is to continue working to decrease the prestige of this ideology.

I hope the difference here is clear? The problems in microbiology are caused by a peer review system which is rigorous and concerned with truth in principle, but in practice, lacks the standard image analysis which would enable it to detect fraud. Thus, it produces papers that are not true. Critical Social Justice scholarship produces papers which are not true because truth was never a criterion in the first place. They will not retract unless somebody like us says that we didn’t really mean it. That is a truly terrible reason to retract a paper. Can you imagine if a creationist tried to reproduce our methods and succeeded in publishing a paper which followed conscientiously all the scientific methods of evolutionary biology? If he then said, “Ha! Fooled you! I don’t believe a word of it!” the journal would have no reason to be embarrassed or retract the paper because its soundness would be not at all impacted by whether its author believed it or not.

Of course, the fact that problems in peer review have different causes does not mean that we should be less concerned about any of them. Unreliable knowledge production is always a problem in any field and academics within that field must always be trying to detect them, address them and weed them out. Usually, they are.

The fact that peer review in Critical Social Justice is guided by ideological bias and therefore, largely worthless, does not at all indicate that peer review in virology or epidemiology are. In fact, a look through the most recent publications on Covid vaccines today indicates the contrary, in my limited experience. I have no qualifications in science, but I do have some experience of what ideological group-think in academic publishing looks like. It looks like people building on each other’s work without ever checking its foundation in reality. It looks like constant escalation and racing off in all directions. It has a tone of narrative-reinforcing competitiveness that says “She’s right but I’m even more right or right in a more interesting or radical way.” It does not look like people checking each other’s premises and correcting or qualifying each other or themselves accompanied by a list of limitations. It does not have a tone of cautious understatement that says, “These findings suggest X but there’s a question over Y to be resolved and more research needed into Z.” It certainly does not look like every 4th paper being a correction of an earlier paper based on new evidence. That pattern does not look like ideologues trying to reinforce a narrative but scientists trying not to be wrong.

I will now be accused of being naive at best and of being ideologically captured at worst. I will probably be accused of being both an establishment liberal elite or a radical “wokeist.” That happens a lot lately, although my views have not changed. I have already been accused of just wanting to “Protect THE SCIENCE TM” today. I have no idea what that means. I certainly have no wish to protect any scientific claims that are not true or any scientist who is knowingly perpetuating them. If there are any virologists or epidemiologists who are publishing fraudulent papers due to ideological capture, personal promotion or for pay by the shadowy “Big Pharma,” this would be utterly despicable and I would want them removed and penalised. That is what would genuinely protect science. If there is a problem with peer review in that area, please show it to me and I will pass it on. Scientists within those fields who are aware of fraud, please get in touch with me. I can connect you with people who will take this very seriously indeed and can help you address it.

I, however, have very little wish to get into any discussions about Covid or vaccines. It is neither my area of expertise nor a subject I have much interest in. I would just ask anybody claiming that the Grievance Studies project discredited the peer review system to stop doing so, because it is not remotely true.