A play-by-play of the Cassini scare on Mayday

Last week's Cassini Significant Events Report included a detailed play-by-play of the frightening morning of Mayday. I thought it was a very interesting read on how a mission deals with an "anomaly." In this case, the anomaly began very frighteningly but turned out -- as it usually does -- to be no big deal, though events like this are probably stressful enough to shorten the lives of everybody involved. Here's an excerpt from the update. One definition beforehand: "downlink" refers to data being sent from the spacecraft; "uplink" refers to data being sent to the spacecraft, just as with the words "download" and "upload" in reference to getting data from and sending data to a server.

Monday, May 1 (DOY 121)

Monday was a busy day. Cassini had a DSN pass over Goldstone on Saturday, nothing on Sunday as planned, and our next tracks were the Madrid array on Monday, May 1. What follows is a mini diary of a spacecraft anomaly. The following has been extracted from various e-mails jetting through the project today. Read o

Sequence Lead
he planned downlink from the spacecraft (S/C) did not go as expected this morning. From the beginning of the track, we have been unable to see any signal from the orbiter. The uplink was swept multiple times, and a signal from the low-gain antenna has been looked for with no effect. I am calling an anomaly meeting for this morning at 9:00 PDT.

Spacecraft Operations Office (SCO)
loss of downlink anomaly occurred on May 1, 2006. No one-way data was received at the beginning of track at around 6:30 a.m. PDT. We started the uplink drive on time at 6:40 a.m., but stopped after 50 minutes when the no-downlink anomaly continued. After looking for all combinations of spacecraft safing with DSS 63, DSS 55, and Radio Science Open Loop receivers, SCO decided to wait for nominal lockup at the two-way time.

The "two-way time" means the time it takes for the uplink signal from Earth to travel to the spacecraft, and for the spacecraft to respond with a return signal. The downlink should have begun with Cassini automatically sending data back at a previously prescribed time. That data not having been received at all, ground controllers waited (impatiently) to see if the spacecraft would at least reply to the command they had sent from the ground at the start of the planned communication session. Two-way light time between Earth and Saturn would have been roughly 2.5 hours on May 1.

Sequence Lead
Good news! At the time we were scheduled to go 2-way with the spacecraft we regained downlink. Data is now coming in as expected. Telemetry indicates there was a Solid State Power Switch (SSPS) trip on the unit that powers the Ultra Stable Oscillator (USO), and the USO is currently powered off. This means that a 2-way light time after the transmitter was turned back off, during the recovery efforts, we will go back to 1-way, and thus will lose the downlink again, for approximately 2 hours. Since this will almost definitely impact the SAR data, we will be sending up the contingency command to zero out data policing during this pass, and zero out everything except VIMS during the following observation period. The SAR data will then be played back during tomorrow's pass.

An oscillator is a necessary piece of radio equipment for generating a signal of the exact frequency that a receiving antenna expects. With it switched off, Cassini went through all the motions of sending its data back to Earth, but it didn't generate any kind of signal that the Deep Space Network could hear. As for "data policing," Cassini carries an onboard table that prescribes how much data each instrument is permitted to write to its recorders -- this prevents any one instrument from filling up the recorder. Typically, the instrument teams plan out their observations so as not to run up against the caps written in the data policing table, but in the case of an anomaly like this they don't want Cassini to start overwriting data that hasn't been received on Earth; by "zeroing out" the table they can stop recording (and overwriting) of the precious data from the close flyby.

SCO is currently planning to investigate whether the SSPS trip was merely a cosmic ray hit or whether there is something actually wrong with the USO, and we should know that by the end of the current pass. I will forward that info as soon as I have it. We are trying to get our DSS-15 track tomorrow extended to our rise time, so that we can go 2-way as early as possible, in case the USO is damaged.

What this means is that Cassini was going to be above the horizon at DSN antenna 15 for a certain length of time, but the original plan was only to communicate for a part of that time. The project sent an emergency request to the DSN to please let them have more time -- the entire time that Cassini was above the antenna's local horizon -- so as to exercise all possible contingency plans during this upcoming pass.

Spacecraft Operations Office
It turns out that the SSPS for the USO was tripped, causing no one-way downlink carrier or data. This tripped switch condition is consistent with ones seen in the past. This is the sixteenth trip seen to date, the third trip of a switch that was ON at the time, and the second trip this year. The previous trip occurred very recently on March 2, 2006. They are predicted to occur at a rate of about two per year, and are most likely caused by Galactic Cosmic Rays.

Commands were built and uplinked to cycle the SSPS OFF to clear the trip, then commanded it back ON. The spacecraft is operating nominally following this activity.

Sequence Lead
The file to zero out data policing during the current pass and for all but VIMS on the next observation period has been sent with a bit-1 time of 121T17:34:55. We will confirm that this file reached the spacecraft at approximately 20:00.

The DSN was able to extend tomorrow's track for Cassini, starting it at station rise at 19:15 rather than at our allocated time of 21:10. This will allow us to do an early uplink, and go 2-way sooner than we would otherwise, which will be a boon, if cycling the SSPS does not clear the issue up.

Instrument teams have been reporting nominal operations for their instruments. All are awaiting playbacks to determine what data was overwritten.

I will keep you posted on any further development.

Tuesday, May 2 (DOY 122)

Spacecraft Operations Office
Telemetry indicates that the command file to move the pointers to replay Titan data has executed. The pointers were moved to the appropriate locations discussed in yesterday's meeting with CDS, Science Planning, Mission Planning, Uplink Operations, and RADAR. Now we just wait for the data...

I had to ask Dave Seal what pointers meant. He said that they refer to, well, pointers on Cassini's solid-state recorders that instruct the spacecraft where to start and stop playback. The recorders are like an infinite loop of tape. There is a record pointer and a playback pointer. When the recorder is empty, the record pointer is located immediately after the playback pointer. When there is some data on the recorder, the spacecraft begins downlinking from the playback pointer until it catches up to the record pointer -- they chase each other around the recorder. If the recorder fills up, that means the record pointer has caught up from behind to the playback pointer; to continue recording, it switches to the second solid-state recorder (Cassini has two).

So if, for whatever reason, the downlink is not received on Earth as expected, the pointers need to be moved back to tell Cassini from when to when it should repeat the playback of its data. When there's just a small data gap, it's quite easy to figure out where the pointers should go to pick up the data in the gap; but when there's a big anomaly, there can be some guesswork involved in figuring out where to move the pointers to in order to recover everything.

RADAR Instrument Team
Just received the first RADAR packet on the ground. The time lines up at about -12 minutes from closest approach. So looks like the first ~90 Megabits got overwritten as predicted. Hopefully the end point is correct. Looks good and as expected so far!!

UVIS Instrument Team
The UVIS T13 occultation data is on the ground. The data looks good except for an unfortunate 1-minute data gap just when the signal starts to decrease. It is assumed that this is due to the S/C going 2-way or to a telemetry rate change. In any case nice recovery work! Thanks to all involved in this data recovery.

RADAR Instrument Team
Great pointer work!!
The end point was also correct!
We got all our data from about -12 minutes up to where we received data yesterday. There was a small overlap, so there are no gaps in the data that we do have. The only missing data is the -20 to -12 minutes of SAR that was overwritten. So basically ~600 out of the expected 700 Megabits was recovered. Great work!

So, there you have it, the anatomy of an anomaly and a nearly full recovery. Thanks again to Dave for the explanation.