Hello, limitations! The paradoxical power of limits in scientific writing

Sanjay Singh

 Department of Dermatology and Venereology, Institute of Medical Sciences, Banaras Hindu University, Varanasi, India

Correspondence Address:
Sanjay Singh
Department of Dermatology and Venereology, Institute of Medical Sciences, Banaras Hindu University, Varanasi - 221 005
India

Be a light unto yourself.

~The Buddha

I keep six honest serving-men

(They taught me all I know):

Their names are What and Why and When

And How and Where and Who.

~Rudyard Kipling, The Elephant′s Child

(Just So Stories, 1902)

From the January-February 2015 issue, Indian Journal of Dermatology, Venereology and Leprology has introduced a new requirement for its esteemed authors. The authors are now required to write the limitations of their study in a section of structured abstracts and in the discussion section of their Original Articles/Studies. The present article looks from different angles into the value of mentioning the limitations of a scientific study and makes some suggestions about how to discover them.

Why Bother About Limitations?

Inductive reasoning is often used in science to advance our knowledge. In this process, a hypothesis is formed and experiments are conducted to test the hypothesis. On the basis of observations, a hypothesis is either "proved" or disproved. But the fact remains that no amount of research can prove a hypothesis as absolutely true because a single new piece of contrary evidence can disprove it. Perils of the observation-based certainty are beautifully illustrated by the story of a turkey who concludes from 1000 days of being well fed before the thanksgiving that humans are friendly, only to be proved dead (literally) wrong on the 1001 ^st day. ^[1] Thus, all conclusions in science are provisional and falsifiable, attesting to the naivety and incorrectness of belief in perfection of research. In addition to this in-built limitation of science, being the fallible human beings all of us are, our work may not conform to the highest standards of science and it is desirable that we as researchers accept and mention this. So limitations are normal, it is statistically abnormal to do a limitations-free study. Furthermore, by explicitly stating the limitations of our work, we are really doing a service as such an act stimulates new research and provides us a chance to gain a better understanding of the world, which is the only aim of science.

Why We Sometimes Do Not Mention Limitations of Our Research

Although mentioning limitations of our work is an admirable goal, barriers exit in the way of doing so. These barriers may (of course, provisionally) be classified into three categories: (a) Real, (b) Psychological, and (c) Imaginary. Unfortunately, these categories are not mutually exclusive, but tend to travel in groups of varying combinations. The real reason for not mentioning the limitations is a true unawareness of the limitations of the work. Here I don′t know that I don′t know. Strangely, this real barrier is probably the easiest to overcome. We will find plenty of suggestions below to transcend this unawareness.

Let us now meet the father of all fallacies (or the mother of all misconceptions); the psychological barrier to seeing the whole picture, its name is the confirmation bias. ^[2] We undeliberately (subconsciously) tend to look for information that favors our cherished beliefs and disregard the information, which is contrary to it (the disconfirming evidence). Confirmation bias is stronger in emotion-laden domains and when we have invested a lot of time or effort in a given issue, ^[3] making us, in effect, partially blind in relation to the work. This bias thrives in the discussion part of the article. And suppose I have spent a year doing a study and the results are psychologically appealing to me, the study acquires a rosy glow, a sort of halo, and the subconscious bias obliterates my desire to even think about its drawbacks, forget about writing them. Fortunately knowledge is power here; by merely knowing that this bias beast exists I, in an important first step, can partially self-liberate myself from its clutches. Following suggestions are designed to take care of the rest of the problem.

Finally, we turn our attention to the imaginary barriers. These are imaginary in the sense that they exist more in the mind than in the real world. I may believe that if I write about the limitations of my study, it will negatively impact referees′ assessment of the work. But this is an incorrect assumption and any good referee or editor, knowing that no study is perfect, will welcome the mention of limitations. Thus we see that being defensive or trying to minimally mention drawbacks of the study is both scientifically incorrect and unproductive or it may even be counter-productive. When I am writing an article and there is some confusion about whether or not to mention limitations of the work, I may ask myself a few vital questions, which together may be called the Confusion Clarifying Test: "If I was the referee or editor, wouldn′t I prefer a clearly written article which mentions its strengths as well as the weaknesses over the one which leaves out the negative details?" "What will help more the doctors (and consequently the patients) who read my article and make clinical decisions on its basis, giving them partial or no information about its drawbacks or showing them the whole picture?"

Having mentioned the virtues of revealing the limitations of a study, it goes without saying that when a study has serious drawbacks, revealing these is not likely to salvage it from rejection (chances of such a study getting published are slim otherwise also).

Will It Work? Embracing Pragmatism

Good referees will almost certainly find out the limitations of a study and journal editors take great care to send the manuscripts to such referees. So the efforts to avoid mentioning the drawbacks or to partially mention them are most probably doomed to failure. Even if due to some extra piece of good luck I escape this filter unhurt, readers are going to discover the limitations of my work. Therefore, it is practically beneficial to be open about the limitations up front.

Discovering The Drawbacks

Find a devil′s advocate

Find someone who frequently disagrees with you or a colleague who was opposed to the idea of your research from the beginning and remained so. Treat him or her nicely and give a copy of your manuscript for comments. Ask the devil′s advocate probing questions such as "What else you think could be wrong with my work?" or "Could you be a little more critical please?"

Play the devil′s advocate yourself

Imagine and really believe that your article was written by someone who you are not very fond of. An old but now unsympathetic friend, whom you may have started slightly disliking, will nicely fill this space. Then question yourself at every step of the study: "Why could this be wrong?" or "What could have been better?"

The Tolerant Friend

A friend who can spare some, or (if you are blessed) a lot of time to critically read and comment on your manuscript will be a great asset.

The Checklist Choice

Checklists are a great way to find out what is to be done or which steps to follow to achieve a desired objective, or if something has already been done to know what has been unwittingly left out. Obviously it is preferable to consult them before a project is begun. Ex post facto they humbly, but being impersonal unforgivingly, remind us about the limitations of our work.

So the first natural step would be to find a checklist that is appropriate to your work. Fortunately many such good checklists have seen the light of day, which are freely downloadable from the Internet. For cohort, case-control, and cross-sectional studies, you have STROBE checklists. ^[4] STROBE stands for "STrengthening the Reporting of OBservational studies in Epidemiology". In STROBE, you will find separate checklists for the three types of studies and also a combo version. For clinical trials, the checklist to be consulted is the 25-item CONSORT checklist, ^[5] where CONSORT stands for "Consolidated Standards of Reporting Trials."

The Wait Test or Distancing

Write your article and then make it invisible by hiding it somewhere, or more effectively, ask someone to secretly hide it from you. The minimum recommended Manuscript-Hiding Period is one week. During this period, don′t think about it, if the desire to think about it or see it arises, as it will, let it float by like a cloud in the sky, don′t succumb to it. Wait. This unique combination of physical, cognitive, and temporal distancing from your manuscript will give you some perspective to find faults in it when you return to it later.

The Notice Board Strategy

Bravely put your manuscript on the departmental notice board. Treat people kindly and repeatedly request them to put their anonymous typed comments below it at odd hours when no one is watching. Such completely anonymous out of the blue comments have a greater likelihood of being honest and helpful. This strategy may also be called the Bulletin Board Bravery.

That-Which-Could-Not-Be-Named

Respectfully invite comments on your manuscript from people who have published in the area of your work. I couldn′t think of a fancy name for this strategy (however, for acronym lovers, it could be TWCNBN).

The above-mentioned strategies, especially when used in combination, are likely to brightly illuminate the entire landscape of our research work.

To conclude, although I tried to make a strong case for revealing limitations of the research work, is this piece of writing a tour de force? No, truly far from it, it has several, well… limitations.

ACKNOWLEDGMENTS

Part of the title of this article is inspired by Hayes SC. Hello, darkness: Discovering our values by confronting our fears. Psychotherapy Networker 2007; 31: 46-52. "Distancing" is inspired by use of this word (in a different context) in Hayes SC, Smith S. Get out of your mind and into your life. Oakland: New Harbinger Publications; 2005.

1.	Taleb NN. One thousand and one days, or how not to be a sucker. In: Taleb NN, author. The black swan: The impact of the highly improbable. 2 nd ed. New York: Random House Trade Paperbacks; 2010. p. 38-50. [Google Scholar]
2.	Dobelli R, author. The art of thinking clearly. 1 st ed. London: Hodder and Stoughton; 2013. [Google Scholar]
3.	Heath C, Heath D. Consider the opposite. In: Heath C, Heath D, authors. Decisive: How to make better decisions. 1 st ed. London: Random House Books; 2013. p. 92-114. [Google Scholar]
4.	STROBE group. STROBE checklists. Available from: http://www.strobe-statement.org/pdf/index.php?id=available-checklists [Last accessed on 2014 Oct 31]. [Google Scholar]
5.	CONSORT Group. CONSORT checklist. Available from: http://www.consort-statement.org/download/Media/Default/Downloads/CONSORT%202010%20Checklist.doc. [Last accessed on 2014 Oct 31]. [Google Scholar]