Mars Rover Spirit Upgraded from "Critical" to "Serious"

The condition of Mars Exploration Rover Spirit “has been upgraded from ‘critical’ to ‘stable,’” announced MER Project Manager Pete Theisinger today, and the robot field geologist is now sleeping on Mars.

The specially formed Anomaly Team working overnight to unravel the mystery of what happened last Wednesday that caused the rover’s computer to fall into a reboot loop and stop functioning properly – had by this morning come up with a working hypothesis. There is, the A-Team has discovered, some kind of glitch or corruption in the flash memory and the software that is used to communicate with that memory.

That, at least is the “working hypothesis” the team is currently pursuing. “[That] is consistent with many of the observables and with operations performed on vehicle last night,” Theisinger explained.

The flash memory is one of three different kinds of memory in Spirit’s processor, in addition to Random Access Memory (RAM) and EEPROM, electrically erasable programmable memory.

The RAM is much like the memory on your computer. Spirit’s processor uses it in real time mode. This memory, however, is fleeting or ‘volatile,’ meaning whatever data is collects it must return promptly because when the rover is turned off at night, all memory is lost. EEPROM, electrically programmable memory, is used to store part of software image.

Flash memory, which similar to the memory in a digital camera and a type of EEPROM, can be read to and written from easily and is stored in the system, even when the rover is powered down. For the most part, the rover normally uses the flash memory primarily for storage and retrieval of engineering and science data –that’s how all those PanCam images were stored for sending home. In order to send that data home, however the flash memory has to ‘communicate’ with software, which is responsible for opening files and establishing directories, among other things. “That process has to work correctly,” said Theisigner, for the system to operate seamlessly. And apparently it was not last Wednesday.

While the A-Team continues working on coming up with a resolution to the problem, in the worst case scenario, Theisinger said, “we are capable of operating the vehicle without going to the flash memory, in what we call a ‘cripple’ mode -- a term, he stressed, the team had just created (“don’t read too much into it”).

“Basically, [cripple mode] tells the flight software when it boots up that it should operate with the file directory out of the RAM memory rather than the flash memory. That would avoid any issues we would have either with flash or the software,” Theisinger said.

As expected, Spirit failed to shut down last night, continued on for another sleepless night and went to low power. “That was confirmed by a later UHF session with the Odyssey pass where we got 73 megabits of data, mostly garbage data, although some fault data, some corrupt, some data that was 14 hours old.”

By the time the 9:30 a.m. local Mars time window for communications opened, Spirit was “not there,” which indicated, Theisinger said, “we had gone to low power.”

The team commanded Spirit to respond and acknowledge that during the 11 a.m., communications session. “We timed it so when the 11 o’clock session would start, we would begin to get that session at 10 bits per seconds indicating we had gone to low power.

Then they commanded Spirit to reset her computer to come up in ‘cripple’ mode. ”And that is what happened,” said Theisinger, adding that they then received one hour’s worth of data transmitted via the rover's low gain antenna at a slow, but steady, 120 bits per second. “That happened as planned,” he said. Moreover, the rebooting that had been happening every hour did not occur.

That sequence of exchanges led them to deduce the current hypothesis that “something involved in flight software that talks to flash software is causing difficulty,” Theisinger explained.

After data return earlier today, the team decided to shut down vehicle to replenish batteries. “We commanded a shutdown just before end of session, so we would know [because the] session ended early, and that happened,” Theisinger reported. In addition, they sent “two post shutdown” messages to which –if Spirit had shut down and was sleeping –would go unanswered. They did not hear anymore from Spirit, “confirming that the vehicle is now sleeping on Mars.”

The UHF passes to relay data through the orbiters have been cancelled for tonight to give the golf-cart-sized rover time to recharge.

The best news, perhaps, is that Spirit is stable in terms of power and its thermal condition is ‘nominal.’

The current ‘go-forward’ plan – is to continue communicating with Spirit and “move forward with diagnosis and recovery based on what we find,” Theisinger said.

At this point, however, they still do not know what happened when the command sequence initiated to test the motor than operates the mirror on the mini- thermal emission spectrometer – Mini-TES – stopped.

“The mission consequences are uncertain at present time, but we have more capability than in the worst couple of scenarios . . . and we still have a couple of weeks to get back," Theisinger concluded. He estimated Spirit might have to wait as long as three weeks before driving again.

For now, there is no plan to make any drastic changes on Opportunity as a result of Spirit’s troubles. “We’ll play this out,” Theisinger said, “one day at a time."