New Horizons "back in action" after safe mode event, ready to resume encounter science

NASA held a press briefing today to explain the nature and cause of the spacecraft anomaly that halted science on New Horizons for four days as it was on its terminal approach to Pluto. As of the moment that I write this post, New Horizons is not yet performing science observations, but it will begin them tomorrow. Here are the high points from the briefing:

• The anomaly related to the main computer being asked to do two computationally intensive tasks at once, and they were more than the computer could handle, so New Horizons switched to the backup computer, entered safe mode, stopped science, and called for help from Earth.
• On Earth, engineers quickly understood the problem.
• It is not a problem that can happen during the encounter.
• Mission leadership chose to suspend science activities to focus on recovery efforts.
• Science activities will resume on July 7 at 9:45 PT / 12:45 ET / 16:45 UT, Earth received time (so, about 12:15 UT, spacecraft event time).
• 30 planned science observations were lost between July 3 and 7, none of them required for the top-level science goals of the mission.
• The anomaly is no reason to doubt that New Horizons will perform its encounter science as planned.

The briefing panel consisted of Jim Green from NASA Headquarters; Alan Stern, New Horizons' principal investigator (representing the science team); and Glen Fountain, New Horizons' project manager (representing the engineering team). In the interest of getting a post up quickly, I'm just posting here my notes from the briefing, only lightly edited. I attempted to transcribe as they spoke, but in some cases I paraphrased, and I added some notes of my own in brackets []. I've interleaved the panelists' responses to questions in with their primary statements.

Jim Green:

At 2pm [I presume Eastern time] on Saturday, July 4, we lost the signal. Within 90 minutes signal was reacquired, and the spacecraft found in safe mode. We are just delighted with how New Horizons team has responded.

Glen Fountain on the nature of the anomaly and the recovery:

On July 3 we were preparing for the main event -- the command load that takes us through the flyby, which starts on the 7th. [A command load is a sequence of commands for the spacecraft, a program that it will execute autonomously.] All the observations from the 7th of July through the encounter on the 14th and on through the 16th are in a single command load. On the 3rd we loaded it onto the backup computer, and on the 4th we were loading it on to the primary computer.

At the same time, we were taking all the data we wanted to get down to the ground but wanted to save, and were compressing it into a single location on the [solid-state] data recorder. We were doing multiple things on the spacecraft at the same time, and what occurred is as we were doing the compression of all the data to save it for later download, we were burning to the flash memory the command load into the primary computer.

The computer was trying to do two things at the same time, and the two were more than the processor could handle at the same time, so the processor overloaded. The system autonomously switched to the backup computer and since it did that and we were not in encounter mode, we went to safe mode. At that point we lost the downlink from the primary side of the spacecraft because it had switched to the secondary side. [So note that while it was compressing data and loading software, New Horizons was in the middle of doing a third thing, downlinking data.]

We knew it would take about an hour for the spacecraft to transmit to Earth from the backup computer. We started looking for signal on backup side, and found it when expected. We looked at data, figured out what was happening, and started to put a plan in place to recover. The command load has now been put on the primary side so that when the 7th rolls around, we will go into the core load as planned.

Some of Fountain's responses to questions:

• We won't do another burn to flash during encounter.
• This was the first time that that specific fault has occurred; all the previous safe holds have occurred on the primary computer [without a switch to the backup computer].
• In encounter mode, a safe mode event like this cannot occur.
• The core load will begin on July 7 at 12:24 pm Eastern time.
• We designed the core load so that every observation that gets us group 1 data [highest-priority science results] is redundant, so that should a reset occur during the flyby, when we come back online, that observation will be repeated so that we still get the data. That's the kind of thoroughness with which we plan observation sequence around closest approach.
• Once you're in encounter mode, the process will reboot and come back to the timeline, but recovery takes on the order of 7 minutes, so that each of these critical observations is all separated by 7 minutes.

Alan Stern on the science impact:

New Horizons is operating flawlessly and on course. So are all the instruments in the payload. In the few days prior to this event we've been receiving tremendous data and I couldn't be happier with what we're seeing already from this great distance. Every day when the science team meets, the room is full of smiles.

Science was suspended, which is normal to do under these circumstances, making sure spacecraft was healthy, getting it configured the way it should be. What we lost was some of the science on Saturday, and then all the science for Sunday and Monday. And that was a command decision which I made and the team was in complete agreement with, that it's much more important to focus on getting ready for the flyby than to collect science 8 or 9 million miles from the target.

Precisely what we gave up:

• We lost 16 LORRI images, 3 of which were for navigation.
• We lost 4 Ralph color observations.
• 1 Ralph composition spectroscopy measurement.
• 4 Alice atmospheric observations -- but the Alice team does not expect to detect the Pluto system until July 12, these were just being taken for due diligence.
• We lost a "plasma roll" in which SWAP, PEPSSI and radio science collaborate to measure planetary environment.
• We lost 3 days of SWAP, PEPSSI, and SDC background monitoring. We don't have any evidence that any of these have detected the Pluto system yet anyway.

That's 30 observations out of a total of 496 to be made between July 4 and end of close approach observations. So about 6% of observations by count, but scientifically we weight observations by how close they are to the planet. These are not nearly as important as the ones we'll make from 100 times closer. While we'd prefer this hadn't occurred, it's a speed bump in terms of total return.

Some of Alan's responses to questions:

• The team was able to restart the compression process and save all the uncompressed data so we can downlink it after the flyby.
• The spacecraft is very effective at collecting a large amount of data. We're going to have 99% of the data on the spacecraft and only 1% back on the ground by the time we finish this 9-day load.
• The spacecraft is not currently collecting any data at all. Our objective, following the safe mode event, was to get to prime data collection, which is the flyby load. Distinctly secondary was the remaining observations that would have taken place before the main load begins. We made the decision to sacrifice all of those instead to focus entirely on spacecraft recovery, instead getting the flyby load up and verified, get the checksums back on the ground, erase some solid state recorder segments, finish that data compression. The first science that we will collect will be some plasma measurements that will begin shortly before the load engages. First imaging observations will begin after load engages tomorrow morning.
• On this spacecraft, there's a combined data storage capability of 128 Gigabits. By terrestrial standards that's small, but by spacecraft standards, that's large. By coincidence the flyby is the 50th anniversary of Mariner 4 mission, the first flown successfully to Mars. We will collect approximately 5000 times as much data as Mariner 4. We're going to knock your socks off, and we're going to continue doing it through 2015 and 2016.

Immediately after the briefing they shared some images composed of some of the last data that New Horizons returned to Earth before the anomaly on July 3.

A questioner asked about the deconvolution being done to this data, and whether some of the features in the images are artifacts of that process. Stern replied (I paraphrase here):

I would be careful about talking about impact craters at this point, we don't have any strongly confirmed impact craters. We are still very far away. The deconvolution procedures we are using are very, very mature. There is no hocus-pocus about it. We have multiple methods and intercompare them, and look for evidence of the same features in the raw data, and we look for them to be consistent from observation to observation. Things that aren't rotating with the disk are candidates for being artifacts. We are much less subject to artifacts now that we are closer because the target is so much larger.

The last photo of Pluto that New Horizons returned was taken at 23:25 UT on July 3. The next one that it will take (if I understand the plan correctly) is at 22:41 on July 7. In that time, New Horizons will have closed more than a third of the remaining distance, increasing Pluto's apparent diameter by a factor of 1.6 and the number of pixels on Pluto by more than three times. We're so close!