Spirit Remains in "Intensive Care," But Continues to Radio Home

Spirit 'phoned home' this morning and returned some good engineering data, but the Mars rover remains in "intensive care," as members of a newly formed anomaly team scramble to find out what caused the glitch two days ago -- and how to fix it.

"We are still critical," Mars Exploration Rover Project Manager Pete Theisinger announced at the press briefing this morning. "We do not know to what extent we can restore functionality to the system, because we don't know what's broke," he added.

However, Theisinger made it clear that at this point that neither the team nor Spirit is not confronting a life-or-death situation and for now the rover can continue on. "We believe, based on everything we know now, we can sustain the current state of the spacecraft from a health standpoint for an indefinite amount of time," he said. That will give the team time to work on a fix for whatever is troubling the rover.

"We are a long, long way from being done here, but we do have serious problems and our ability to eventually work around them is unknown. Do not expect a big sea change in either knowledge or theory in the next several days, because this is a very complex problem and we have very limited visibility right now," he advised the roomful of reporters.

The next scheduled communication between the ground team and Spirit is scheduled to take place around 4:30 a.m. Pacific Standard Time, between around 10:30 or 11 a.m. local Mars time tomorrow - and it will be through her direct-to-Earth X-band system, via one of the Deep Space Network (DSN) dishes.

The robot field geologist's computer appears stuck in a reboot loop, and Spirit, the team now believes, hasn't gotten much -- if any -- sleep during the last couple of nights, which could run her batteries down as early as tonight.

The good news is that the rover did successfully transmit data home to the Jet Propulsion Laboratory (JPL) today for the first time since early Wednesday -- and it arrived in three sessions lasting 10 minutes, 20 minutes and 15 minutes respectively. "We have been able to command Spirit and we have gotten limited data in return, said Theisinger.

While the data came in at a snail's pace of 120 bits a second, compared the 30,000-some bits per second it had been sending earlier this week, the data was "good data," Theisinger confirmed.

"This morning, we sent an early beep to the spacecraft and did not get a response, and, as we were preparing to send a second, the spacecraft talked to us," he recounted. "We got very fractional frames and moved quickly to asking [Spirit] to speak to us at 120 bits per second for 30 minutes. We got 20 minutes of transmission in that occasion, which was a single frame of engineering data repeated, and then we repeated that whole sequence of events and got about 15 minutes of engineering data at 120 bits per second, where the frames were updated for the 15 minutes, and then for the second 15 minutes we had nothing but fill data."

Not surprisingly, there was no science data returned today, just much-needed engineering information that could offer up clues as to what went wrong. As Theisinger put it: "We've got it broke and want to find out as much as possible - so as much as I like Steve Squyres [the lead scientist for the science part of the mission] he loses in this situation."

Sleepless in Gusev -- and Pasadena

Spirit was humming along at least until around 1 p.m. local Mars time Wednesday. "We did know [then] that the spacecraft did not believe itself to be in a fault condition, although it could have been having problems," offered Theisinger. They know that, he said, because Spirit had returned data at a rate "that's not at a rate we would expect if the spacecraft thought it was a fault condition."

Then, several hours before Spirit was due to call it a day, while she was testing a motor that moves a mirror for the rover's Mini-Thermal Emission Spectrometer -- known as Mini-TES -- the show stopped. Mini-TES is the instrument records the thermal spectra of various rocks and soils to determine the types and amounts of minerals that they contain.

"The reason [the motor] was being exercised is because the Mini-TES wants to take observations at night, at cold temperature, and at cold temperature you need to increase the current to the motor to get it to have the same performance you would get during warmer temperature," Theisinger explained. "We were going through a set of current limit steps - increasing the current, then moving the motor, then increasing the current, moving the motor again -- to characterize its performance in that situation. That was a sequence of events and I'm not exactly sure when in that sequence it stopped, but the last command of that sequence did not execute. That's what we know. That sequence of commands did not run to completion. That is the first place where we can say it started doing something unexpected."

Since then, the flight software "has not been behaving normally," Theisinger said. In fact, the software appears to have rebooted Spirit's computer more than 60 times since Wednesday. "The spacecraft has been in a processor-reset loop of some type where the processor wakes up, loads the flight software, and uncovers a condition that would cause it to reset," he explained. "But the processor doesn't do that immediately. It waits for a period of time -- at the beginning of the day it waits for 15 minutes twice, and for the rest of the day it waits for an hour -- then it resets, and comes back up."

The most puzzling aspect of this is that on at least two occasions whatever caused the computer to reboot was not the same thing same thing. "We are confused by that, but those are the facts as we presume them to be right now," Theisinger said.

The A-Team

Although there just isn't enough information to begin pinpointing exactly what happened or even the starting point at which things cascaded, that's exactly what a specially formed Anomaly Team is charged with doing.

This team is "completely separate from the Opportunity Team," Theisinger noted. "They will be working a schedule that will look like 0500 local Mars time to about 1500 local Mars Time, so they're going to synch up with Spirit today."

The A-Team will report to work at midnight PST, and spend the first part of their 'day' examining and analyzing the data recently returned and then figuring out the "go-forward" plan for the day, Theisinger said. The team will then spend most of their day on the consoles, after which they will work on theories. I expect this to go in this mode for several days - talking to the spacecraft gathering more data, winnowing out theories, testing those theories against spacecraft observables, and continuing that process," he offered.

Knowing what started this would be most helpful to getting Spirit back on the road. And whether or not her system suffered from one error or glitch or a host of glitches. But that's exactly what they don't know. "We don't know what started this chain of events -- and I believe personally that it is a sequence of things -- and, therefore, we don't know the consequences of that," Theisinger said. "We have a long way to go here with the patient in intensive care. But we have been able to establish that we can command [Spirit]. And we have been able to establish that it can give us information, and we have been able to establish the power system is good and that thermally, the rover is okay - and those are all very, very important pieces of information."

The data Spirit returned this morning may offer up more clues. "It helps us a great deal," Theisinger said. "We get a set of error messages for what the software believes the current state is and we do get engineering on the principle subsystems and the engineers are looking over that now to ascertain" what's happening. "It gives us pointers for where to ask for more information tomorrow. As the theories begin to winnow out because they're not supported by the observables, then we can begin to work on what the root cause and [determine] what the corrective actions might be."

The most recent analyzed data, for example, " tells us the software is in X-band fault mode" -- a sort of emergency mode said Theisinger. "That could be caused by a large number of things. We surmise it got there because of some problem with the high gain antenna (HGA) pointing and that is why the second HGA pass on Wednesday did not work. It gives us a little bit of the telltale for what's going on with the processor now." But just a little bit.

Given that the two times the team has and communicated with the rover's system they got different flight software behaviors, "we do not have assurance that the next time we go and ask it that we'll get either one of those behaviors or a third behavior."

The A-Team is currently analyzing those responses to determine why the flight software is doing what it's doing -- "and how come when we boot up it doesn't get better," said Theisinger. "Those are the kinds of questions we're raising at the moment. We have quite a bit of information but not as much as we would like." At the same time, they have learned what information they don't need to know and so they are moving forward with "a more focused view" for where they should be looking for answers. Unfortunately, until they uncover what started it all, their ground models at JPL are not of much help.

"There are two kinds of things that can get you into this trouble," Theisinger said. "One is a hardware system that breaks in a certain way and impresses that on the flight software somehow and you've got to figure out what that is and cause that same situation to test it against the testbed, So first you've got to find the root cause. The second thing is software. [Spirit's] software is a very complex software package with a lot of modules talking to each other," Theisinger continued. "One of reasons you test software so much is there are always probabilities of when things will happen and times don't always line up and you can have contention issues where pieces of software kind of butt into each other. Those are very hard to reproduce. The only way is to put a lot of time on system so that in a probability sense, you get one. So we need a starting place to begin to set up the testbed in a way that can mimic the situation. It's very hard to duplicate a situation like this until you know the root cause."

The A-Team's primary approach for now is to continue talking with Spirit and investigate the possibilities that it is in some part a hardware problem with the mini-TES motor or perhaps HGA. Right now, Theisinger said, the Spirit has an "inability to move" the high gain antenna.

"I think that we should expect that we will not be restoring functionality to Spirit for a significant period of time -- for many days -- perhaps a couple of weeks, even in the best of circumstances for what we see today," Theisinger said. At the moment, he added, "we need to get the software situation to get behaved so that we can then get more direct access to the functionality of the system and see what we have that's good.

Bottom Lines

For now, Spirit is holding her own, despite the lack of sleep. "We believe, based on everything we know right now, that we can sustain the current state of the spacecraft, at least from a health standpoint, for a substantial, perhaps indefinite period of time," Theisinger assured. "There's no indication that we have an imminent power problem or a thermal issue," he reiterated.

If Spirit stays sleepless in Gusev for too much longer though, her batteries will run down, and in that sense the robot field geologist may be more human that we would like to believe. But, Theisinger informed, "it is not something that would end the life of the rover, because we can in fact wake up on solar arrays only."

If the batteries do go low, the computer will command itself into low power fault, which in turn takes the batteries offline, among other things, making the situation "more complex," Theisinger said. "There is a chance that would happen. We attempted at the end of today to shut down the vehicle to basically conserve, allow it to recharge and to get the thermal situation a little cooler. When I left -- with about 30 minutes to go this morning -- that attempt had not been successful -- so there is a low, unknown chance we could go to low power tonight," he admitted.

Still, he pointed out, that's "not necessarily" a bad thing. It makes things more complex, but it does clean up a bunch of other stuff as well. Its not a duration threat to the vehicle. The batteries can go to zero state of charge and be recharged just fine."

When asked how concerned he is about Spirit, Theisinger bottom lined it: "I've been asked that question a lot in last 24 hours and I don't view the situation that way. We have a serious problem. The fact that we have a vehicle that we believe is stable for an extensive period of time will give us time to work that problem. We can command [Spirit] to talk to us and even though we get perhaps limited information, we do get good information and that helps us work the problem and that give the engineers a problem to chew on and they're off chewing.

"I expect we will get functionality back out of this rover and I think that the chances that it will be perfect again are not good," he contended. "The chances that it will not work at all are also low I think. We're somewhere in that broad middle and we need to understand the problem to find out exactly where we are."

Opportunity Cruising To an On-Time Landing

While the A-Team works on restoring Spirit, other members of the MER team have split off to prepare for the landing of the second Mars rover, Opportunity, scheduled for 9:05 p.m. Saturday night PST, 12:05 am Eastern Standard Time Sunday, and 5:05 a.m. Greenwich Mean Time.

Opportunity will land on the opposite side of Mars from Spirit, in a vast plains area called Meridiani Planum. The rover's target lies within an Oklahoma-sized outcropping of gray hematite, a mineral that usually forms in the presence of water. Scientists plan to use the research instruments on this rover to determine whether the gray hematite layer comes from sediments of a long-gone ocean, from volcanic deposits altered by hot water, or from other ancient environmental conditions.