The Good, The Bad, and The Ugly

Does anyone else have a difficult time trying to separate “good science” from “bad science”?  I’m a very black and white person.  I love facts and truths and logic, and that drives most of my family crazy.  Perhaps that’s why I struggle with identifying bad science; there’s seemingly no clear-cut, concise way of identifying junk that ends up published.  To be clear, I’m not talking about retractions for blatant disregard for scientific ethics.  I’d classify these situations (e.g., the Xenobe controversy, Sames’ retractions, Bell Labs, etc.) as “ugly.”  I’m particularly concerned with cases where during a presentation everyone sort of looks at each other, raises his/her eyebrows, frowns, and collectively mumbles, “Hmm.”

It seems the term “junk science” has been in use in the legal profession since the 1980’s.  Yet, despite its existence, “junk science” is actually an ambiguous concept.  In 1998, legal experts Edmond and Mercer attempted to conquer this beast by identifying “good science,” then considering outlying cases “bad.”  Here’s what they considered “the good”:

“’Good science’ is usually described as dependent upon qualities such as falsifiable hypotheses, replication, verification, peer-review and publication, general acceptance, consensus, communalism, universalism, organized skepticism, neutrality, experiment/empiricism, objectivity, dispassionate observation, naturalistic explanation, and use of the scientific method.”

Does this list really mean that everything else is considered “junk”?  I can think of a few brilliant studies that used trial and error methods in lieu of the scientific method.  Conversely, I’m aware of peer-reviewers who simply check the “publish” box without actually reading the manuscript.  As is argued on several other blogs, identifying “junk science” is a very gray area.

Perhaps one way to define junk science is to take the Jacobellis v. Ohio approach.  In a 1964 US Supreme Court case involving obscenity, Justice Stewart Potter wrote in his opinion, “I shall not today attempt to define the kinds of material I understand to be [pornography]…but I know it when I see it.”  Clearly the same frame of thought can be applied to junk science.  I am less inclined to accept the Jacobellis approach because it offers nothing tangble.

There must be some empirical qualities that set the good from the bad.  Despite all the skills I’ve learned with a mere decade of lab experience, I am disheartened to admit that I honestly never perfected the skill of detecting bad science.  So, like a responsible, up-and-coming assistant professor of chemistry, I went crawling through the literature to determine what separates the good from the bad.  Below is a list of a few things I learned.

In the spirit of Jeff Foxworthy, science might be “junk” if…

Researchers are more concerned with holding press conferences than publishing results in reputable, peer-reviewed journals. One might assume that “breakthroughs” ought to be showcased in the most prestigious journals after being subjected to a rigorous peer review process.  Fast tracking all the way to the press conference phase certainly raises some flags about credibility.  I’ve seen this phenomenon happen first-hand, and when the science is questionable, the ensuing public announcement can get really ugly (and entertaining, for that matter).

Something about the research seems off kilter. If you think something doesn’t feel right, you might be correct.  Although going with your gut will only get you so far, analysis guides such as “Tipsheet: For Reporting on Drugs, Devices and Medical Technologies” help identify specific areas for journalists to consider when examining the veracity of medical therapies.  Cook and co-workers suggested that similar checklists might likewise serve the general scientific community when evaluating the credibility of reported work.

Conflicts of interest are not explicitly disclosed. In these cases, scientific integrity might be compromised for financial, political, or other external motivations.  In developing this article, I encountered journals, funding agencies, and governing bodies that require authors to declare any potential conflicts of interest while publishing or applying for grants.  Although editors and referees try to uphold strict transparency policies, authors can still fail to report external influences and biasing.  These cases essentially touch every facet of research--cancer, testing pesticides (Berkley Scientif. J. 2009, 13, 32-34), and even drug development.  The onus is put on the audience to look into the author’s sources of funding.

The flow of logic doesn’t make any sense. Junk science may have gaping holes in experimental descriptions or proposed models.  Fortunately, overly simplistic and inaccurate scientific explanations usually evoke sharp criticism from the scientific experts.  Credible “debunkers” often attack the logic of an issue by (for example) discrediting cited authoritative opinions, identifying assumptions, and/or offering overlooked hypotheses.

Colleagues in the field are widely skeptical of the work. Mix it up with your cohorts.  A simple, “Hey, what did you think about the most recent (insert name of researcher here) article in JOC,” can shed some light on the context of published or presented findings.  “[He] hasn’t published anything reproducible in the past 20 years,” my PI once said.  “I sincerely doubt that this latest paper is anything new.”

Subscribe to the comments for this post

NSF Reauthorization

The bill that will reauthorize the NSF had a markup by the subcommittee on Research and Science Education. Since what happens in the policy world can have repercussions in the science world here is a list of policy changes to NSF that caught my eye.

The Bill: NSF Reauthorization 2010
• 5% of the NSF research budget has to be used for high-risk high-reward proposals. (SEC. 201. SUPPORT FOR POTENTIALLY TRANSFORMATIVE RESEARCH)

The Amendments: NSF Reauthorization Amendments 2010

Daniel Lipinski (D-IL)

• Wants NSF to give cash prizes to high-risk, high-reward research challenges. (SEC. 207. PRIZE REWARDS)
• The prizes will range from 1 million to 3 million

I don't care what the topic turns out to be, but for that much money Chemistry Blog will organize and field an open team for the competition. How the NSF goes about and implements these contests will be interesting to see.

The NSF Reauthorization will be wrapped into the America COMPETES Act and likely will be voted on by the full House of Representatives before May 31st. The America COMPETES Act shouldn't suffer any major hurdles for passage.

Mitch

Subscribe to the comments for this post

Geoengineering Scientists and Congress

(From Left to Right)
Dr. David Keith
Dr. Philip Rasch
Dr. Klaus Lackner
Dr. Robert Jackson

Geoengineering is a wonderful example of taboo science. Most people would fall within 2 camps. Camp 1 considers geoengineering with disdain as it mucks with the natural environment. Camp 2 probably wouldn't want their government involved in planetary climate control. With those entrenched camps where do scientists fit in?

Scientists were called as witnesses before The House Subcommittee on Energy & Environment last week in regards to geoengineering. The witnesses invited were...

• Klaus Lackner (Geophysics,
  Earth and Environmental Engineering): Covering CO₂ sequestration
• Robert Jackson (Biology): Covering Biological and Land Strategies to lower CO₂
• Philip Rasch(Atmospheric Science but a chemist by training): Calling for a Manhattan project type approach to researching geoengineering
• David Keith (Chemical and Petroleum Engineering): Mainly advocating that some sort of global policy towards geoengineering needs to be developed. The most sane and coherent witness; scientists don't usually fair well before politicians.

So why care about taboo science? The simple matter is that it would cost 1-2 billion a year to return the planet to pre-industrial levels of temperature, assuming they use cheap sulphates to do the job. This means any number of nations, frankly any wealthy cohort of industrialists, can take climate control into their own hands.

Since geoengineering is a delicate subject to broach to the public, transparency is crucial and wasn't loss on the chairman Brian Baird (D-WA). Congressman Baird mentions how some citizens believe their government is placing psychotropic drugs in jet fuels, the so called chemtrails and remarked "...legitimate scientific research [in geoengineering] must not get tied up in these kind of things."

However, all the scientists were taken aback by Randy Neugebauer (R-TX), my favorite exchange was the following.

Randy Neugebauer (R-TX)

Randy Neugebauer, "What percent of the atmosphere is CO₂?"
Scientist, "390 parts per million".
Randy Neugebauer, "Less than one tenth of one percent...This tiny minuscule amount...[can't] be more important factor in our climate than solar activity".

I'm not even sure where to begin to broach such a deep misunderstanding of climate change. I would have mentioned to Mr. Neugebauer that he would be dead if that minuscule amount of CO₂ was removed from the atmosphere, as all plants would die followed by animals in short order. The concept of small amounts having huge impacts in large dynamic systems is an important lesson to teach, even more so to do it dexterously. These types of exchanges went on for some time. I'm left wondering why Randy Neugebauer is even on the Subcommittee on Energy & Environment in the first place.

The ranking Republican, Bob Inglis (R-SC), had this to say in his last remarks, "I believe in a basic role of government is to do basic research, its an important function that we do." It is nice to know that basic science research is appreciated by both sides, even though there is always a rogue member in every committee.

Press Release: Subcommittee Examines Geoengineering Strategies and Hazards

Mitch

Subscribe to the comments for this post

ARPA-E Gets a Congressional Hearing

(From L to R)
Dr. Arun Majumdar
Dr. Chuck Vest
Dr. Anthony Atti
Mr. John Denniston
Dr. John Pierce

ARPA-E is one of the newest funding programs at the Department of Energy. It was authorized in 2007 with the passage of the America COMPETES Act, but was only funded when The American Recovery and Reinvestment Act of 2009 was passed with an initial $400 million. ARPA-E is unique in that its first Funding Opportunity Announcement (FOA) was kept broad and only asked for 8-page proposals for high-risk but high-reward "transformational" technologies. Yesterday (Wednesday) in the House Committee on Science and Technology ARPA-E was examined, the chairman for the hearing was Bart Gordon (D-TN).

Arun Majumdar, the current director for ARPA-E, gave several examples why federal funding is necessary for energy research and used a graph on worldwide shipments of solar photovoltaic cells to make his point that America is losing its edge in energy technologies.

Arun also gave some metrics on the ARPA-E awards. 3,700 concept papers were received. Only 340 were invited to write a full proposal. 37 projects were selected and $151 million was pegged for those projects. 45% went to small business, 35% went to educational institutions, and 20% to large industries. Also mentioned was the start of their Fellows Program for recent PhD students interested in energy/policy.

John Pierce, the vice president of DuPont Applied BioSciences, gave a statement that called for "external advisory panels" to guide the perspective of the ARPA-E agenda. Which sounds like something industry would want.

Link to more information on the hearing: Program to Foster Innovation in Energy Technologies Is Off to a Promising Start

Link to ARPA-E: Advanced Research Projects Agency - Energy

Mitch

Subscribe to the comments for this post

Obama & Science Education

Student science geeks will be going to the White House, that is what caught my attention from Obama's remarks on his "Education To Innovate" campaign delivered Monday. Student winners in national competitions in science, technology, and robotics will be eligible to display their know-how to Obama at a White House science fair. The exact dates and times were not announced, but it is nice to know that White House visits will not be limited to athletes.

A list of some of his other initiatives is given below.

• National Lab Day: Is an attempt to get scientists into local schools to help with demos and fieldtrips. I just signed up and would suggest you take a look at it too. [Link]

• Connect a Million Minds: Is a project running through Time Warner. The website is very vague and seems to mention robotics and has a sign up sheet, but detailed specifics on what they are planning is not forthcoming from their webdesign. The picture of a potato-grape molecule is interesting though. [Link, ie only apparently]
• STEM Video Game Design Competitions: A competition to make the best STEM video game. [Link]

Mitch

Subscribe to the comments for this post

Maz Goes Politician

Ever since the budget crisis began here in California, Mitch and I have debated how we would fix the problem if we were in positions of power. While we had some pretty great, and pretty terrible ideas, we soon stopped wondering what we would do in hypothetical situations and began to wonder how we could actually make a difference. Well, we decided to begin stepping into the world of politics; hoping to influence policy decisions that affect scientists and chemists for a start.

Enter ACR 88, a bill introduced by assembly members Torlakson (D-Martinez) and Furutani (D-Carson) in California.

The bill creates the California Task Force on Science, Technology, Engineering, and Mathematics Education (Task Force) to promote the improvement of science, technology, engineering, and mathematics (STEM) education across the state. The task force would generate discussion on policy that would improve the teaching of those subject areas for California's K-12 students. It has no fiscal impact (the task force members are not paid).

You see, currently a full third of the 4th graders and a fifth of the 8th graders in the nation can't preform basic computational math, and US high school seniors recently tested below the international average (out of 21 countries) in math and science.

Out of this poor group, take the fact that California ranked 46th (against other states) in math proficiency and 42nd in science proficiency on recent 8th grade National Assessment of Educational Progress (NAEP) tests.

And it still gets better. More than 50 percent of California 4th and 8th grade students scored below the basic level in science and 40% ranked below basic in math as determined by NAEP.

Given these statistics, it becomes obvious that California needs to drastically rethink it's teaching methods and policies for K-12 math and science. As energy production, global warming, water purification and other scientific issues become more common to the 10 o'clock news, and therefore more salient in the public mind, we need to also focus on preparing the coming generations for the problems we are going to leave them. Also, the United States Department of Labor has recently shown that math or science preparation will be crucial to successfully competing for a job in 15/20 of the fastest growing occupations right now.

The President is also focusing national attention on scientific research, innovation, and math and science education. In a speech at the National Academies on April 27th, President Obama promised to make U.S. students the international benchmarks in the next decade by doubling budgets at certain science and technology agencies, policy change to enhance math and science education, and beginning to allot more than 3% of the nation's Gross Domestic Product (GDP) to research and development. Obama said he wants to involve everyone from governors to parents to students to help increase support for science and technology and the quality of teaching. Supposedly 5 billion dollars is available in federal funds to help states improve their math and science teaching.

This is well and good, but throwing money aimlessly at the issue won't solve anything. Bills like ACR 88, creating task forces to investigate effective policy change and inform the legislators, are the correct first step to tackling science and math education reform.

If any of you readers live in California, I urge you to write to your assemblyperson telling them that you believe we need science and math education reform and that you want them to support ACR 88. For any of you that are ACS members, they made it supremely easy for you. Simply go to http://www.act4chemistry.org/action/STEMtaskforceca/ and enter the relevant information. They will automatically send it to the correct representative for your district depending on your address. In fact, they even wrote the letter for you too!

Comments PLEASE. As a California public school survivor for my entire academic life, I have been through (and seen the failings) of the system first hand. I have some ideas on how to fix the issue, but I want to hear from ppl not in California too. Leave your two cents on what needs to be done to improve K-12 science and math education. Move calculus to required at 10th grade? do away with optional general physical sciences and the like? make everybody take biology followed by chemistry and then physics? in that order? what about elementary school? when to start teaching the scientific method? If 5th graders get sex. ed., should they also get newton ed.?

Lets see your ideas.

Subscribe to the comments for this post

Mitch Lobbies the US Senate

I had the opportunity to visit senate offices and speak with staffers at the recent ACS conference. The visits were sponsored through the ACS Office of Public Affairs and organized through local sections government affairs liaisons. Since this was my first visit to the Hart Senate Office Building, I thought it would be of interest to share my impressions of the experience lobbying aides and pushing policy points.

The senator's staff are held up in a rather nondescript looking office building.^[Pic] As you walk into the building, there is a security guard and metal detector setup. You place your items to be x-rayed, like at an airport except these guards are much more friendly. Oddly, you don't need to be in the books, anyone can just walk in. The lobby of the building is imposing and there are no places to sit! There are also no water fountains! If you're thirsty you will have to beg for water from whichever senator's staff you are there to visit.

As I'm a Californian, my task was to speak with staffers for Barbara Boxer and Diane Feinstein. First up was Barbara Boxer. The California contingent consisted of 10 chemists that spanned the breadth of the state and ranged from academics, to industry, to government employees. When you walk into the office area, I admired how well everybody dressed. It was not like how a Chemist dresses "fancy" (i.e. dark dress shirt and light-colored pants, whether the shoes match is a craps shoot). All the male workers wore nice suits that matched with itself, the women were dressed in a formal professional attire that I have never seen before. First was the exchange of small pieces of paper with the receptionist. Unbeknown to me, it is strongly recommended that you have a business card to hand the receptionist. The receptionist will run off and photocopy all of them and give the copy to the aide so they are in a better position to know who they are talking to. The waiting area is decorated with US flags and pictures of the state. Much of Boxer's front office was actually covered with the names of Californians who have died in Iraq and Afghanistan. We were then moved to a conference room within the office and finally got to meet the staffer.

The staffer in this case was a legislative assistant. From what I could gather they are on the 3rd rung of the hierarchy, if we assign the 1st rung to the senators. They report the topics of the meetings to their supervisor and the senator. As we get introductions out of the way it is clear the staffer is very confused and not quite sure how to handle us. I think she was expecting smooth talking, hard core lobbyists, trying to make a hard sales pitch. What she found was a collection of 10 chemists sitting in a room having as much a good time telling stories to her and each other. The purpose of the visit was to explain to her how important stable science funding is to research and the goal was to ask for an increase in a k-12 spending bill from 175 million to 450 million. At one point in the meeting a couple of chemists became really animated and started raising their voices with each other and discussing how California state bureaucrats were making things worse back home. The staffer seemed a bit frightened about this, and claimed she has no control over how the state legislature does their business. From my perspective it was normal chemist behavior. We will often raise our voice, start speaking fast, and become animated when we find something worth arguing about. Apparently lobbyists arguing with each other is a strange and foreign concept and does not happen in the senator's office (mental note). I think by the end the staffer finally understood she wasn't dealing with real lobbyists in as much as a cross-section of concerned citizenry.

Next up was Feinstein's office. We regrouped and restrategized. It was decided that it would be bad form to argue with each other again and tell stories. This time we were actually successful in finishing going around the room and introducing ourselves. We spent 2-3 minutes each discussing how science funding has helped us develop professionally. We were so successful we even made it to the pitch. We asked whether Feinstein would support an increase in funding for so-so-bill in fiscal year 2011. At this point we were asked what the additional funding would be used for. Unfortunately, none of us knew and we said we would email the sheet in a follow-up email. We all forgot to bring that sheet of paper. D'oh! At any rate, just asking why there needed to be more education funding opened up a can of worms amongst the chemists that lasted the rest of the meeting. From any independent viewpoint it is obvious that science education in the United States is slipping. Some argued this through certain reports by the RAND corporation, some argued this through the drop in science papers from the US, others just used anecdotal evidence.

Overall,  I would venture to say our performance helped the cause more than it hurt, but it is clear to me that chemists make bad lobbyists.

Epilogue: I extracted the humor from my visit to give you a more interesting story to read. However, it would still behoove chemists to become comfortable communicating in the language of policy. The most valuable lesson I took from the experience was that I have more then a passing interest in policy, and the next time I pitch to a senator's aide they better be ready for a well versed policy exchange with a chemist.

Mitch

Subscribe to the comments for this post