The Curiosity Kerfuffle: the big (and increasing) difference between data and discovery

I'm in San Francisco, reporting from the American Geophysical Union meeting. This morning, there was a much-anticipated press briefing featuring the latest results from Curiosity. The news is simultaneously exciting and dull.

What's exciting: Curiosity's incredibly sophisticated SAM instrument is returning good-looking data; repeated analyses of the same material are producing the same beautiful results. It's the last of Curiosity's science instruments to be fully checked out, and with the exception of one damaged sensor on one instrument (a wind speed sensor), the whole scientific instrument package is working absolutely perfectly. When the scientists finally get a chance to start selecting interesting materials to study, the data set is going to be very rich, providing fodder for years and years of work.

What's kind of dull: Their analyses were of a "typical, ordinary" Martian soil, and in general, the Rocknest soil does appear to be typical and ordinary (keeping in mind, of course, that we're talking about what's "ordinary" on the surface of another freaking planet). There are tantalizing hints of some interesting chemistry, but it's too early to be certain if the organic compounds they detected got their carbon from Mars, from meteorites, or from Earth. Stay tuned. I will explain what SAM did find in a later post. But there's something else I need to get off my chest first.

Everybody involved in the Curiosity mission is glad that this morning's press briefing is behind them. It has been a nutty couple of weeks, because of an unguarded comment by John Grotzinger in front of NPR science correspondent Joe Palca. Right before the interview began, Grotzinger had been reviewing the data from SAM's second analysis of Martian soil, and seeing for the first time how well it matched the first analysis. He knew at that moment that Curiosity would be able to do all the science that he had been dreaming of for so many years of hard work developing the mission. That was the context for his comment to Palca that "This data is going to be one for the history books. It’s looking really good." Not that the specific data that he was looking at contained any surprising discovery; just that the quality of the data demonstrated that the capacity for discovery is there. Curiosity's capacity for discovery is greater than any mission ever sent to the surface of Mars. It's going to be a fun few years. We're not there yet though.

People heard "one for the history books" and immediately jumped to the conclusion that the mission had a major discovery on their hands. Palca fueled the flames by describing what Grotzinger was seeing as "earth-shaking," a phrase that has since quietly been edited out of the NPR story and replaced with the weaker word "exciting." He said that Curiosity had clearly made some major discovery, but that Grotzinger was being coy about it.

The Internet went crazy with speculation. SAM has many capabilities. Among them is the ability to detect and characterize organics. What could be "earth-shaking" (again, Palca's words, not Grotzinger's) other than organics, or even life? Even as NASA and JPL attempted to downplay the results, angry commenters abused NASA for sitting on major news, or blamed NASA for creating the hype in the first place. As if NASA had issued a formal statement of some sort teasing this news, rather than one scientist's unguarded comment from the apparent safety of his office.

I have been trying to figure out why and how this snowballed out of control, because I didn't see it coming. I've decided that the problem here is that most people don't understand the difference between "data" and "discovery."

At times, especially in our initial reconnaissance of the solar system, the two can be synonymous. One of the major discoveries made during my professional life was the discovery of Earth-like river systems on Titan, revealed for the first time in images returned from the Huygens probe. It took only one glimpse of those images (the data) to make that truly groundbreaking discovery obvious.

But science doesn't usually work that way. And as NASA's scientific investigation of the solar system is getting more rigorous, the science is getting more sophisticated. With an instrument as complex as SAM, the data do not yield instant science discoveries the way that the first images from a new world can. People seem to think that with SAM you turn a crank and a computer displays "LIFE" or "NOT LIFE," or even "EARTHLIKE ORGANIC CHEMICALS" or not. That's not how it works.

The instrument spits out data like: how many times a molecule of a specific molecular weight hit a detector in each 0.1-second bin. For 5,000 different-sized molecules. Over the course of several hours. Those molecules are gases that were made by cooking solid materials, decomposing big molecules into smaller ones. You have to work backward from those gases to get to the original soil composition, looking at the temperatures at which they appeared. There's a lot of ambiguity in the data. For just one very small instance, a detection at a molecular weight of 28 Daltons could be molecular nitrogen or carbon monoxide. So you have to repeat the run after running some of the gases through a chromatograph to separate them, and see if you can reduce that ambiguity. There are so many factors that could affect the results: Chemical reactions among the compounds that developed during heating. Local environmental conditions. Presence of Earthly contamination. Problems with the instrument -- noise, or malfunction of some detectors, or electrical problems. Mistakes in calibration. Incorrectly set instrument parameters.

In this context, knowing that you have an instrument that is working and producing clean-looking (not noisy) data that looks the same when you analyze the same material three different times, is, as Grotzinger called it during today's press briefing, a "hootin' and hollerin' moment." Knowing that you have good data is very exciting, even if you have no idea what it means yet, because you know that the instrument won't be holding you back from making discoveries.

Should Grotzinger have been more circumspect in front of Palca? Perhaps. Maybe it was a poor decision to react to brand-new data while Palca was recording. But before you criticize him too much, think about what the consequences of that criticism could be.

I'm a scientist, so I know how giddy scientists get about good data. I know first-hand that moments like that are what many scientists live for -- it's what makes all the hard work worth it. But that excitement doesn't often get transmitted to the public. When NASA has something to announce, you usually see exactly what you did today if you watched the briefing: a panel of scientists mostly not even cracking a smile as they make carefully measured and often prepared statements about what they can and cannot conclude from their data. (As much as I like Ken Edgett and Paul Mahaffy and John Grotzinger, I'm not very happy that the panel was all-male, either. Doesn't make a great impression.) I'm sure that, behind the closed doors of Curiosity mission operations, these guys were jumping up and down in their enthusiasm. On the panel, it was just a bit dry. I know how excited I am about the results, but I felt a little gloomy about how it was going to be playing to the seven television cameras that were set up in the back of the room. (To have any TV coverage of an AGU briefing is pretty rare.)

Space exploration is exciting. Learning new things about other worlds is thrilling. Knowing that a thing that you have spent more than a decade crafting and worrying over actually works is worth "hootin' and hollerin'" about. Rather than telling Grotzinger to be less excited, we need to work to explain to people how science actually works. That you don't often get discoveries handed to you on a platter. That there isn't going to be any single measurement from any instrument that can tell us there was life or wasn't life on Mars. That the journey to discovery is often as fun as the discoveries themselves.

NASA has a choice here. They can prevent kerfuffles like this by keeping the doors closed. By not permitting the public to look inside and see the messy process of doing science across 150 million kilometers of space. By only presenting results neatly packaged on a platter as if that's the way they arrived on Earth. Or they can let us in, so that we can enjoy the ride. NASA, to its credit, is one of the most open of federal agencies. That openness presents opportunities for snafus like this week's. But it also makes them one of the few branches of the United States government that many people actually like.

I do see scope for hope in the midst of the Internet insanity. The fact is, millions of people were excited about the prospect of a scientific discovery from Mars, absent any information of what that discovery might be. People care about space exploration, and they are paying close attention to a mission that hasn't even really had a chance to discover anything yet. That is all good, and while we have their attention we can do some education not just about space science, but also about the scientific process in general. This is education that the public badly needs, and those of us who are paying attention and can explain what's going on should be talking to as many people as we can right now.

That's my job, of course, but if you're reading this blog, it's your job too. You probably have a pretty good idea of how science works, and of what the Curiosity mission is capable of. So it's on you. You need to be prepared to use this as a teaching moment. Go forth and educate!

My day at AGU isn't over yet; the Curiosity scientific sessions begin soon. I hope to learn much more about what exactly Curiosity detected before the day is over. So stay tuned for a report on the science!