Earlier this year, my scientific article titled "traditional indicators inflate some countries’ scientific impact over 10 times" was published in the Scientometrics journal. For anyone interested in the field, the full text available at https://rdcu.be/b0ZF0. Here, however, rather than discussing the subject matter of this particular research, I want to analyze its broader implications.
The title itself is rather telling. Leaving the subject matter aside, it implies that certain indicators were in use in the professional community, which are, in some cases, 10 times off. At least that is what I claim in the article, and the arguments appeared sufficiently solid to be published by a leading periodical in the field, issued by the Springer company.
An indicator which can be 10 times off - what does it really mean? Is it a big deal? Yes, it is. Imagine someone estimated Shaquille O'Neal's height to be 22 centimeters, rather than 216. That's what 10 times off means. Yet, such indicators were - and still are - in use in the scientific community.
I happened to find such cases in the very first field where I tried: scientometrics aka the science of science - the field dedicated to quantitative assessment of scientific activity. The indicators in question aren't some semi-professional tools only used on the fringes of the field, either. They have been used by successful, reasonable, well-meaning researchers for decades. I myself used them until I became involved in the practical side of scientometrics - that is, became a scientific development advisor at the University of Georgia, Tbilisi, and gradually realized, what substantial effects certain methodological practices had.
Does the possibility of such extreme imprecision in scientific research imply science is useless? As misleading as, say, fortune-telling? Certainly not. Yes, the use of the indicators which "inflated some countries' scientific impact over 10 times" was a part of the scientific process. But finding ways to replace or improve these indicators is also a part of the same process. We can never be sure that the estimates given by scientists are precise. Yet we can't obtain more precise estimates without science. That might be frustrating, but that's the way it is.
Neither is this bittersweet reality of science limited to, say, social sciences - after all, a key premise of Einstein's breakthrough with his theory of relativity was that Newton's mechanics was terribly wrong in certain situations. Obviously, I'm not drawing a parallel between my relatively minor research and Einstein's monumental discovery: that's just the first example coming to mind.
But really, entire history of science is about correction of the mistakes of one's scientific predecessors, often even one's own. Yet in the public discourse science somehow remains largely associated with precise, definitive answers. That can be dangerous both for the public and for science itself. Once certain scientific "truths" are found to be wrong, this can lead to disillusionment.
Rather than telling people they should blindly believe anything scientists say, we should be explaining what science is - and what it is not. It is: skepticism, rational quest for knowledge, gradual improvement of estimates. It is: superior to mere guesswork, fortune-telling, or literal interpretations of ancient myths. It is not: a set of dogmatic certainties which have to be followed no matter what.