Chemistry Blog

«

»

Jun 18

The Good, The Bad, and The Ugly

by Jeremy | Categories: chemical education, opinion, science policy | (53629 Views)

Does anyone else have a difficult time trying to separate “good science” from “bad science”?  I’m a very black and white person.  I love facts and truths and logic, and that drives most of my family crazy.  Perhaps that’s why I struggle with identifying bad science; there’s seemingly no clear-cut, concise way of identifying junk that ends up published.  To be clear, I’m not talking about retractions for blatant disregard for scientific ethics.  I’d classify these situations (e.g., the Xenobe controversy, Sames’ retractions, Bell Labs, etc.) as “ugly.”  I’m particularly concerned with cases where during a presentation everyone sort of looks at each other, raises his/her eyebrows, frowns, and collectively mumbles, “Hmm.”

It seems the term “junk science” has been in use in the legal profession since the 1980’s.  Yet, despite its existence, “junk science” is actually an ambiguous concept.  In 1998, legal experts Edmond and Mercer attempted to conquer this beast by identifying “good science,” then considering outlying cases “bad.”  Here’s what they considered “the good”:

“’Good science’ is usually described as dependent upon qualities such as falsifiable hypotheses, replication, verification, peer-review and publication, general acceptance, consensus, communalism, universalism, organized skepticism, neutrality, experiment/empiricism, objectivity, dispassionate observation, naturalistic explanation, and use of the scientific method.”

Does this list really mean that everything else is considered “junk”?  I can think of a few brilliant studies that used trial and error methods in lieu of the scientific method.  Conversely, I’m aware of peer-reviewers who simply check the “publish” box without actually reading the manuscript.  As is argued on several other blogs, identifying “junk science” is a very gray area.

Perhaps one way to define junk science is to take the Jacobellis v. Ohio approach.  In a 1964 US Supreme Court case involving obscenity, Justice Stewart Potter wrote in his opinion, “I shall not today attempt to define the kinds of material I understand to be [pornography]…but I know it when I see it.”  Clearly the same frame of thought can be applied to junk science.  I am less inclined to accept the Jacobellis approach because it offers nothing tangble.

There must be some empirical qualities that set the good from the bad.  Despite all the skills I’ve learned with a mere decade of lab experience, I am disheartened to admit that I honestly never perfected the skill of detecting bad science.  So, like a responsible, up-and-coming assistant professor of chemistry, I went crawling through the literature to determine what separates the good from the bad.  Below is a list of a few things I learned.

In the spirit of Jeff Foxworthy, science might be “junk” if…

Researchers are more concerned with holding press conferences than publishing results in reputable, peer-reviewed journals. One might assume that “breakthroughs” ought to be showcased in the most prestigious journals after being subjected to a rigorous peer review process.  Fast tracking all the way to the press conference phase certainly raises some flags about credibility.  I’ve seen this phenomenon happen first-hand, and when the science is questionable, the ensuing public announcement can get really ugly (and entertaining, for that matter).

Something about the research seems off kilter. If you think something doesn’t feel right, you might be correct.  Although going with your gut will only get you so far, analysis guides such as “Tipsheet: For Reporting on Drugs, Devices and Medical Technologies” help identify specific areas for journalists to consider when examining the veracity of medical therapies.  Cook and co-workers suggested that similar checklists might likewise serve the general scientific community when evaluating the credibility of reported work.

Conflicts of interest are not explicitly disclosed. In these cases, scientific integrity might be compromised for financial, political, or other external motivations.  In developing this article, I encountered journals, funding agencies, and governing bodies that require authors to declare any potential conflicts of interest while publishing or applying for grants.  Although editors and referees try to uphold strict transparency policies, authors can still fail to report external influences and biasing.  These cases essentially touch every facet of research–cancer, testing pesticides (Berkley Scientif. J. 2009, 13, 32-34), and even drug development.  The onus is put on the audience to look into the author’s sources of funding.

The flow of logic doesn’t make any sense. Junk science may have gaping holes in experimental descriptions or proposed models.  Fortunately, overly simplistic and inaccurate scientific explanations usually evoke sharp criticism from the scientific experts.  Credible “debunkers” often attack the logic of an issue by (for example) discrediting cited authoritative opinions, identifying assumptions, and/or offering overlooked hypotheses.

Colleagues in the field are widely skeptical of the work. Mix it up with your cohorts.  A simple, “Hey, what did you think about the most recent (insert name of researcher here) article in JOC,” can shed some light on the context of published or presented findings.  “[He] hasn’t published anything reproducible in the past 20 years,” my PI once said.  “I sincerely doubt that this latest paper is anything new.”

15 comments

Skip to comment form

  1. Noel

    Quoting the great Ralph Nuzzo: “Bad science is like pornography: I know it when I see it.”

  2. Breanna

    Very nice synopsis of the bad science. It gave a crystallized definition that I was able to forward to my husband in order to explain why he gets so annoyed with my criticisms when we are watching tv documentaries.

    1. Jeremy

      I’m glad to see I’m not alone ;-)…
      Anyways, back to Mythbusters (groan)

  3. Unstable Isotope

    Great post, but I do have a concern with this statement: Something about the research seems off kilter. I think this is how politicians dismiss science results they don’t like. I agree with the idea of having checklists. I just think that journalists, in general, need to have more scientists in their lists of sources and less journalists, spokespeople and politicians.

  4. Anneliese Amacher

    I’ve seen a paper in which the authors integrated the solvent peaks in their NMR, so that their integrated area contained not only the peak they wanted, but also the solvent peak! That was very bad science, and it was a JACS paper, too.

    1. Chemjobber

      Anneliese, do you think that there is a difference between “bad science” and a legitimate, but embarrassing error? Just askin’.

    2. Unstable Isotope

      I once saw a paper (when I was an undergrad) where the scientists had spent a lot of time interpreting the spinning side bands of a solid state NMR.

  5. John Spevacek

    Keep in mind Sturgeon’s Law: “90% of everything is junk”, only he didn’t say “junk” – insert your adjective for waste matter.

    With time and wide reading, your BS detector will become increasingly sophisticated, as the wide reading will allow you to build up an internal and self-consistent network of the universe, making it more difficult to fit square pegs into round holes.

    But also keep in mind what Feynman said: “It’s easy to fool people and the easiest person to fool is yourself.”

  6. Juan R. González-Álvarez

    Life is not black and white, it is full of color! And this also applies to science.

    You claim to be particularly concerned with cases where during a presentation everyone sort of looks at each other, raises his/her eyebrows, frowns, and collectively mumbles, “Hmm.” This resembles Pocono conference, where Feynman went to explain his new approach to quantum theory and the many important physicists therein said “Hmm”. An American Physical Society site reports the Hmm:

    A frustrated Feynman, however, failed during his talk to convince the attending physicists of the soundness of his methods. Elder statesmen such as Paul Dirac and Niels Bohr concluded that the young American simply did not understand quantum mechanics.

    Years after, Feynman wrote about this: it didn’t make me angry, it just made me realize that [… they …] didn’t know what I was talking about, and it was hopeless to try to explain it further.

    See also:

    Freeman Dyson: The seminar series: convincing Oppenheimer

    Regarding your Colleagues in the field are widely skeptical of the work, this can be also useless. In my work Science in the 21st century: social, political, and economic issues I have given dozens of examples where colleagues were widely skeptical and finally showed to be, all of them, wrong.

    There is more cases that I have not listed therein. For instance, Ahmed Zewail has reported how his early work was considered by his chemical colleagues as unimportant and of no interested for chemistry (now he is a Nobel laureate). In recent times I also discovered to Murray Gell-Mann saying what was the initial receipt of his quark theory: A lot of people thought the quarks were a crank idea

    I have read your definition of good science and I would like to ask: Is superstring theory good science for you?

    1. mitch

      Any theory that can be tested is a good theory.

      1. Juan R. González-Álvarez

        Agree, but the problem is that many string theorists do not accept their favorite theory to be bad and then [book cited below]…

        its practitioners have become so desperate, says Woit, that they’re willing to redefine what doing science means in order to justify their labors.

        I have read to Witten claiming that string theory predicts gravity (see the interview with John Horgan in the End of Science). There Witten uses the term prediction as if was a class of synonym of “post-diction”.

        Others want to redefine prediction in another way. For instance, they consider a theory good if it predicts something and the contrary. They imagine a multiverse where each universe has one of the possibles values. Of course, their theory does not explain what value correspond to our universe, but that is a minor detail for them. I agree with David Gross that this is all nonsense and unscientific but many string theorists disagree with us

        Proponents and opponents of the idea

        Moreover, one also find the fraudulent papers where the results are falsified. The Schön scandal is well-known and I think Jeremy referred to it when he alluded to “Bell Labs”.

        I wait my original question to be answered by Jeremy: Is superstring theory good science for him?

        Further reading:

        The Trouble With Physics: The Rise of String Theory, The Fall of a Science, and What Comes Next

        Not Even Wrong: The Failure of String Theory and the Search for Unity in Physical Law for Unity in Physical Law

        1. mitch

          Is Math good science? It may have a use one day in a universe far-far away.

          1. Juan R. González-Álvarez

            Using “science” in the sense used above as “natural science”, then your question has not an unambiguous answer. Mathematically you could built a model where molecules have the size of Earth, there is nothing mathematically wrong with that, but I would not consider that “good chemistry” or “good science”… but just bad.

            The question here is not if some people is doing mathematical work for spacetimes of 26 or 10 dimensions and all that, but if their work has any validity as a physical model of universe and if it is fair to provide them funding and support, or if the last 40 years were enough and it is time to explore alternatives.

        2. Jeremy

          I purposely have not provided you (my audience) with any definition of “good science.” Perhaps with more exploration and study of the subject, I’ll one day compile my thoughts into a book you can purchase on most e-readers for a mere $9.95.

          Second, I am not an expert in physics by any stretch of the imagination. I’ll be the first to admit that Morgan Freeman knows much more about the subject than I do. That said, I attempted to brush up a little on quantum mechanics in an attempt to articulate an answer to your question. Namely, is superstring theory good science?

          After exploring the issue a bit, I’ve come to realize that I think your question is loaded. Conceptually SST is largely theoretical, and its constraint is the lack of tangible, experimental evidence. So, is the science good, bad, or ugly? Honestly, I don’t know. What do you think?

          Here are some facts I’ve uncovered about the subject:

          1. A large portion of the theory is, in fact, still theoretical.
          2. Several key players (i.e., elementary particles) in the superstring theory have yet to be directly observed through experimentation. They are currently the best rationalization for the effects experimentalists observe. This fact alone doesn’t discredit the theory; scientists have never seen a hydrogen ion, yet the idea of Bronsted acidity is fairly solid.
          3. The concept of supersymmetry has yet to be well understood (this is a fact, not an opinion). If supersymmetry is well understood, it would be a relatively bulletproof concept, and not the subject of debate. Although, in a quick GoogleScholar search, there are some pieces of experimental evidence to support the concept of supersymmetry, it has its critics. In any case, future planned LHC experiments will further probe the issue, though some bloggers believe that these experiments will result in another nut that is even more difficult to crack.

          I think the question over “good/bad/ugly science” aught to be explored fundamentally. For example, are the experiments supporting the concept of suppersymmetry considered good science?

          1. Juan R. González-Álvarez

            The problem of SST is not the lack of experimental evidence. That was the main problem 20 or 30 years ago. The problem today is that SST even cannot describe the known physics described by the Standard Model and General Relativity, and that we know that it cannot do it. It is now claimed by an increasing number of scientists that SST may be a theory of nothing (TON). Some believers (former string theorists) have started the search for a M-theory…

            Some bored physicists have written parodies of string theory

            Regarding supersymmetry, the models (e.g. supersymmetric extensions of the standard model) and the attempts to detect it (e.g. at LHC) are good (hypotetical) science, but this has little to see with the fiasco of string theory, and all the sociological issues (including the hype in media, unfair reports, blacklist of articles going against string theory, etc.) reported in the books cited above.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>