A.J.S. RaylApr 30, 2009

Mars Exploration Rovers Update: Spirit Endures Reboots and Amnesia, Opportunity Races Forward

Mars Exploration Rovers Mars Exploration Rovers
Credit: NASA / JPL-Caltech

The Mars Exploration Rovers challenged their ground crews with an April full of high drama, a little suspense, and a lot of energy. While Spirit lived through a kind of robot soap opera, complete with bewildering reboots and bouts of amnesia, Opportunity roved forward and back into the fast lane on a restored front wheel, slowing down for a brief visit to a series of small, intriguing craters and an unplanned close encounter with a pesky little purgatoid.

At Gusev Crater, Spirit began the month finishing up a close-up examination of another light-toned patch of soil that turned out to be rich with opaline silica and iron sulfate. It then carried on with its trek along the western side of Home Plate, heading for von Braun and Goddard, its next big attractions to the south. Despite slipping repeatedly in the fluffy, powdery Martian soil, Spirit was persevering as usual. Then mysteriously, on the morning of April 9, it overslept, missed three wake-up calls and three planned events, and lost a whole sol in the process. Its alarm clock finally woke it up 27 hours later and for unknown reasons the rover decided to reset itself.

Spirit, in effect, suffered a “fault,” according to Bill Nelson, engineering-team chief for the MER project, perhaps in the high gain antenna subsystem (HGA) toward the end of the communications session where ground control uplinks commands to the rover. It might have been the software that keeps track of the antenna direction may have somehow lost track and marked its positions 'unknown.’ In a situation like that, the rover is instructed to not move its antenna and declare a fault, he explained. But that’s just a guess. They don’t really know what happened.

The very next sol, Spirit unexpectedly suffered a “fatal” event on its system – much like a hard disc crash on a personal computer, Nelson explained. That caused the rover to reboot. What made these episodes bizarre was that the rover also suffered a couple of bouts of robot amnesia around the same time. In rover terms, that means Spirit failed to record any data in its flash memory, which is where the MERs store and preserve engineering and science data for later downlinking to Earth; therefore, after these events, its memory was blank. The absence of any data left the engineers at the Jet Propulsion Laboratory (JPL) – where the rovers were designed, built, and are being managed – mystified.

Since Spirit put itself into auto mode and later communicated with crewmembers, informing them that it was stable, the events were not disasters. In fact, it appears that the rover woke up after its “fatal” crash event in the “crippled mode,” Nelson said. In that mode, the rover automatically assumes its flash memory isn’t working, so it stores its data onto its random access memory (RAM). The only problem was that Spirit went back to sleep and the data was lost, because RAM is a volatile form of memory, which means it isn’t saved – as it would be on flash – when the rovers shut down.

During the week that followed, Spirit seemed fine and the team engineers allowed it to continue in auto mode, essentially waking up, communicating, and going right back to sleep, while they tried to figure out what happened.

Radiant ripple Radiant ripple
Opportunity took this beautiful image of a banded ripple crest with its panoramic camera (Pancam) on Sol 1858 (April 15, 2009) as it passed through by a series of shallow craters this month. The Pancam team at Cornell rendered here in stunning, false color. This ripple is covered with the little hematite concretions, known as "blueberries," and the bluest part of the ripple is virutally dominated by them. The redder, browner parts are dominated by basaltic sands mixed with the "blueberries," according to Rob Sullivan, a MER science team member. "The banding you can see in the blue part of the ripple on the right are what we think is the internal structure exposed by a little bit of erosion," he pointed out. Covered as it is with the hematite concretions, this ripple isn't going anywhere, Sullivan added, and it's become quite indurated or hardened, suitable for driving over.
Credit: NASA / JPL-Caltech / Cornell

Then, on April 17, Spirit suffered another bout of amnesia and the sol after that, it rebooted again, reliving a similar scenario to that of the previous weekend. Three bouts of unexplained amnesia, along with the reset and reboots – it was downright odd.

The MER team convened its Council of Experts, as Nelson put it, some of the original crew members who knew these rovers when they were just bits and pieces and parts. But they too were “baffled,” said Steve Squyres, the principal investigator for rover science, of Cornell University.

New R9.3 software had been uploaded onto Spirit in March, but that hardly seemed suspect since it was really just a patch to prevent a possible Y2K type of problem with dates. Moreover, Opportunity has been functioning with the same software even longer without any kind of problem. “The interesting thing is that it's an intermittent problem, the hardest kind of problem to troubleshoot, because they don't happen all the time,” Squyres pointed out.

Worse, the engineers had absolutely no data – zilch – nothing to look at, making the task of troubleshooting almost impossible. They don’t even know if the reboots had anything to do with the bouts of amnesia. “It's like the 18-minute gap in the [Nixon] White House tapes,” said Squyres. [Note to whippersnappers: See Frost-Nixon for more information.]

With Jake Matijevic, chief rover engineer and a member of the MER Council of Experts, at the helm, the JPL engineers dug in and began doing everything they could to figure it out. Plenty of speculations have been laid on the table – everything from a problem with the flash memory hardware to a corrupted file to issues with defragged files to some kind of problem stemming from the intimate relationship between hardware and software. “We just need grist for the mill,” said Nelson.

“People are concerned, but during all this time Spirit kept positive power, was never really out of control in the sense that it always got itself into a well-defined state, was willing to communicate with us, and it kept itself healthy," reminded Bruce Banerdt, MER project scientist, of JPL. "That’s a good sign.”

The council advised the team to change the rover’s nap time, so that if it displays this behavior again and saves its data to the random access memory (RAM) system, then at least it can send data to Odyssey for downlinking to Earth before it goes to sleep. That would give them something to go on. They also decided Spirit should keep on keepin' on, something the MERs have always done well.

Tracking the ripples Tracking the ripples
This Pancam image, taken by Opportunity on Sol 1858 (April 15, 2009) and quickly mosaicked especially for this report just last night by the Pancam team at Cornell, shows in glorious false color the rippled terrain through which the rover has been driving this month. "I believe the largest ripples here are pretty old and durable," said Rob Sullivan, a MER science team member. "There's one in the upper left corner, another along the right margin, and a smaller example that the right track obliterated. When we look closely at these, we find they are more or less covered with little hematite concretions, known as "blueberries.'. The bluer areas of the ripple crests are dominated by the hematite concretion 'blueberries,' while the redder, browner parts are dominated by basaltic sands mixed with the blueberries. We think the banding you can see in the blue part of the ripple on the right is the internal structure exposed by a little bit of erosion." Laden as they are with hematite concretions, these ripples, he added, haven’t moved recently and are indurated or hardened and suitable for driving over. "The brownish part of the ripple is wind-blown debris that has moved more recently down the troughs between the main ripples, " he said. "The bedrock in the trough floor is exposed," he added, " there's no regolith there.”
Credit: NASA / JPL-Caltech / Cornell University

One week ago, on April 23, Spirit powered up and roved onward, for the first time since April 8. It didn't get terribly far, because of the slippery terrain, but it did make some progress and it did everything it was commanded to do.

Spirit drove again this past Monday and Wednesday, making only a few centimeters worth of progress each sol, but at least it was moving and following commands. “We're keeping our fingers crossed because the next two weeks will be critical in terms of getting through the southern part of West Valley and into South Valley,” said Ray Arvidson, deputy principal investigator for rover science, of Washington University St. Louis.

The reboot-amnesia drama remains a mystery for now. Some mission followers wonder if Spirit isn’t just displaying a little jealousy, like it did right before Opportunity was to land way back in January 2004. And, to be fair, once again, this rover is left to deal with the harder road – and a completely broken front driving wheel – while its twin, Opportunity, is roving right along once again, without a seeming care in the solar system. “Maybe she has been throwing a tantrum,” considered Squyres.

Despite the frustrations, “the team is doing great actually,” said Squyres. “This has been an interesting puzzle. Certainly we don't like problems, but at the same time when problems and anomalies occur, it makes us use parts of our brain that we don't use otherwise and this is a group of people that kind of enjoys that in a way.”

“Our people here are always of the attitude 'we'll find a way,'"Nelson agreed. "If we can’t go over it, we'll go around it. We'll do anything except give up.”  Not surprising, really. It’s the team, after all, that has imbued the rovers with their can-do, never-say-never principles, something that is sometimes forgotten given the rovers' vibrant personalities.

Rayleigh Crater Rayleigh Crater
This is another false color image that Opportunity took with its panoramic camera on Sol 1854 (April 11, 2009) of a crater called Rayleigh and which was quickly mosaicked for this report by the Pancam team at Cornell. "The reason we took this picture is that if you look at the far wall you see some faint tiger stripes here and they're dipping downward to the right, similar to the dip angles you would expect to find inside a ripple on Earth," Sullivan said. [The stripes are very, very subtle and may take some serious scrutiny.] "We've seen these internal structures of ripples before, but very rarely, in just 2 or 3 images in all this time, and none as clearly as this – even though they're subtle. It's the nicest exposure we've gotten so far and it is consistent with the few other picturess we have that indicate the large ripples have moved in all cases east-to-west." Meaning the winds have blown blown east to west.
Credit: NASA / JPL-Caltech / Cornell University

It’s now been 12 going on 13 sols since Spirit has exhibited any strange behavior. The team, however, does expect the drama will continue. Actually, Nelson said they’re hoping it will. “We’re very anxious to try and catch this amnesia event and reboot episode in our event records and log file,” he explained. “We're hopeful for a clue as to how and why it’s doing this.”

Over on the other side of the planet, at Meridiani Planum, Opportunity after several weeks of a stationary in situ science campaign on targets in an area of exposed bedrock called Cook Islands, was finally given the greenlight to rove on – and the rambunctious rover did just that. The trouble it had with its right front wheel overheating, an issue that began in February, had forced it to slow down. Its handlers had ordered it to take it easy, drive little and almost entirely backwards and take rests. That prescription served the purpose of allowing the lubricant inside the wheel assembly to redistribute.

Cook Islands had been a perfect place to take a good long rest, because it also allowed Opportunity the chance to study some bedrock and soil for the systematic science campaign it’s conducting en route to Endeavour Crater. Once it finished its close-up examination of Penrhyn, a target on Cook Islands, the ground crew let the rover go.

Opportunity took off like Danica Patrick, zipping along to make up time, as well as some serious tracks on its long journey to Endeavour still a long, long ways away. “We wrapped up the Penrhyn investigation and are proceeding at a high rate of speed once again to the south,” said Squyres.

Actually, April was all about driving with a little bit of crater hopping. Based on orbital images, the team has chosen a route that takes Opportunity by a series of small, shallow craters and around a group of gnarly puragatoids, large dangerous sand ripples in which the rover could get stuck. These large ripples were named for Purgatory Dune, which snared the rover for some 5 weeks back in 2005.

Opportunity pulled up to the series of small, young craters, which the team named after Captain James Cook's ships, other than the Endeavour and cruised along, taking breaks for drive-by shootings – of pictures that is, with its panoramic camera (Pancam) of Discovery, Resolution, Adventure, Rayleigh, and Marquis. “We chose the path that would take us by as many of them as we could, because we wanted to get a good sampling of this crater field,” said Banerdt.

“This is one of the first times we've wandered into a cluster of craters that look like they might be genetically related to one another,” said Squyres, and that made them particularly inviting. Opportunity photographed about a half-dozen craters, though there were many more in the immediate area.

Resolution Resolution
This is an image of Resolution Crater, near which Opportunity was parked from Sols 1826-1844 (March 14-April 1, 2009). It is part of a panorama that is due out soon. "There are some issues to deal with because the Opportunity Pancam front sapphire windows are still somewhat dusty, but we're doing our best to generate the best looking 360-degree view from when the rover," said Pancam lead scientist JimBell, who is also the presidnet of The Planetary Society. This is an "exclusive" on the red-blue anaglyph, which only covers part of the 360 degree [the Pancam team didn't have time to finish the full right-eye 360, just the left eye, before our deadline]. " It's a beautiful view of this fresh, young crater on the plains of Meridiani," said Bell. Sit tight, the full panorama is due out soon.
Credit: NASA / JPL-Caltech / Cornell University

Some of the craters were right on top of the ripples. "In this area, it's pretty hard not to hit a ripple," noted Rob Sullivan, of Cornell University, MER science team member.

Interestingly, they are probably the youngest craters observed by either rover to date and “likely formed in the past 100,000 years,” according to Matt Golombek, a senior research scientist at JPL, rover science team member, and Mars Exploration Program landing scientist. Even more interesting, they are peppered with very small pebbles that are either meteoritic or, if the craters are secondary craters, formed by ejecta from the impact of another larger crater, pieces of Mars from elsewhere.

So, in a “new tweak” to the science agenda, as Banerdt put it, the rover will now be on the lookout for some of these pebbles that are large enough to examine with the instruments on its instrument deployment device (IDD) or robotic arm.

Once it finished photographing the craters, Opportunity hit the road again. It was obvious this rover was back in the fast lane and happy about it, a little too happy maybe, because it soon roved right into a closer than wanted encounter with a purgatoid. But this one was small, said Banerdt. “It was more like a – purgatina.” In any case, it wasn’t about to stop this rover, no way, no how. “We know how to get out of these things now,” said Squyres, “so we just popped right out,”

The MER mission crossed an anticipatory sort of milestone in April -- the "minus 1st anniversary” of Spirit breaking the duration record for longest-operational spacecraft on Mars, as Jim Bell, principal investigator for the Pancam and Planetary Society President, put it. Less than one year from now, on April 26, 2010, Spirit will pass Viking Lander 1's duration record of 2244 sols on Mars. Opportunity will break that record three weeks later.  “Can't wait,” said Bell.

Meanwhile, the team has kept its collective eye on the weather. On April 21, Mars moved into the closest point it will come to the Sun in its 23-month, elliptical orbit and in about three weeks the planet’s equinox will mark the start of summer in Mars’ southern hemisphere. This atmospheric-warming combination makes this the most likely time of the Martian year for dust storms severe enough to minimize or thwart the activities of the rovers. “Since the rovers are solar powered, the dust in the atmosphere is obviously extremely important to us,” said Nelson.

Dust storm to the south Dust storm to the south
This nearly global mosaic of observations made by the Mars Color Imager (MARCI) on MRO, on April 2, 2009, shows billowing clouds of dust being lifted into the atmosphere by a storm near the edge of the seasonal polar cap of southern Mars. Atmospheric haze due to suspended dust from recent storm activity is evident elsewhere on the planet, including the skies over Opportunity (MER-B) and Spirit (MER-A). Black areas in the mosaic are the result of data drops or high angle roll maneuvers by the orbiter that limit the camera's view of the planet. Equally-spaced blurry areas that run from south-to-north (bottom-to-top) result from the high off-nadir viewing geometry, a product of the spacecraft's low-orbit. Malin Space Science Systems, San Diego, provided and operates MARCI. Credit: NASA / JPL-Caltech / MSSS

While Spirit and Opportunity continue to point their cameras toward the Sun to check the clarity of the atmosphere almost every day, cameras onboard the Mars Reconnaissance Orbiter (MRO) and Mars Odyssey also make regular observations of the Martian atmosphere that have dramatically supplemented the rover’s own skywatch. The Mars Climate Sounder (MCS), the Mars Color Imager (MARCI) camera on MRO, and the Thermal Emission Imaging System (THEMIS) on Odyssey monitor changes in airborne dust or dust-related temperatures in Mars’ upper atmosphere.

MARCI actually observes the entire planet every day at resolutions comparable to weather satellites around Earth. “We can identify where dust is rising into the atmosphere and where it is moving from day to day,” said Michael Malin, of Malin Space Science Systems, principal investigator for camera. “Our historical baseline of observing Martian weather, including data from the Mars Global Surveyor mission, helps us know what to expect. Weather on Mars is more repetitive from year to year than weather on Earth. Global dust events do not occur every Mars year, but if they do occur, they are at this time of year.”

While widespread haze had reduced Spirit’s daily energy supply by about 20 percent and Opportunity’s by about 30 percent in March, the storm that had been brewing to the far south did not whip up into a major event. Weather reports from Bruce Cantor of Malin’s MARCI team confirmed that the March increase in haziness was not the front edge of a bad storm. “Bruce’s weather reports have let us be more aggressive about using the rovers,” noted Mark Lemmon, a rover-team atmospheric scientist at Texas A&M University. “There have been fewer false alarms. Earlier in the mission, we backed off a lot on operations whenever we saw a small increase in dust. Now, we have enough information to know whether there’s really a significant dust storm headed our way.”

Having these global perspectives on Mars' weather and advance notice of dust events  when they do whip up enables the rover team to take steps, if necessary, to conserve energy on their robot field geologists. They can reduce the length of time their charges will be active, for example, or can shorten or delete some communication events as they did back during the global storm of 2007 that threatened Opportunity and the November 2008 regional storm that dumped on Spirit.

This time, the dust event “kind of flared up, then went away,” said Banerdt. The skies over both Gusev Crater and Meridiani Planum cleared pretty quickly and the taus – or dust levels – at both sites came back down to more or less previous levels. “Right now, we're hanging in at a tau just below 1, which is pretty normal,” he added. But the season isn’t over, so the team is keeping its collective eye on the sky and relying on a little help from their friends on MRO and Odyssey. “We're watching all the time,” Banerdt confirmed. “Right now we're in good shape, but there’s still a strong possibility for storms."

Winds that can lift dust into the air also blow dust off the rovers’ solar panels as this mission has witnessed repeatedly. In fact back in the beginning of the mission no one expected the rovers to last much past five months, because the assumption had been the solar arrays would become covered in dust and thus put the rovers' lights out. The beneficial winds changed everything and are a prime reason Spirit and Opportunity have been able to rove long and prosper. This month both rovers got lucky.

Home Plate Home Plate
Spirit acquired this full resolution mosaic of Home Plate on Sol 1886 (April 23, 2009). The image was taken before a drive, as the over made its way around the western edge of the circular volcanic formation. The field of view spans about 90°and this mosaic, in approximate true color, was created from Pancam's 753-nanometer, 535-nm, and 432-nm filters.
Credit: NASA / JPL-Caltech / Cornell University

Amidst the darkness, there was light, sunlight for Spirit, and a lot of it – and that was the best news from Gusev Crater this month. The dust-laden rover was finally gifted with two solidwinds gusts that blew enough dust off its arrays to boost its power level from 240 watt-hours to a whopping 370 watt-hours, energy this rover hasn’t had in more than an Earth year. An estimated 42% of the sunlight hitting the arrays is now getting through and being converted to power. “Every little bit helps for Spirit, so that was very good news,” said Squyres. Indeed, the news made some MER followers positively giddy.

Opportunity, meanwhile, experienced a cleaning event in early April, jumping to a high of 515 watt-hours of power. It now ends the month at around 500 watt-hours, while still converting an estimated 62+% of the sunlight on its solar panels to power.

Both Spirit and Opportunity passed their monthly physicals with flying colors. “Spirit kind of got weird on us for a while there and I think it would be optimistic to think it's not going to happen again, so we'll just prepare for it,” summed up Squyres. “Opportunity's in excellent health and doing well.”

So, as the month comes to a close, the rovers are wrapping this April – their sixth on Mars -- with happy Hollywood-style endings, raring to rove on into a Martian summer sequel. The agenda now, Squyres said, is simple: “Press hard to the south with both vehicles.”

Spirit From Gusev Crater

The objective for Spirit in April remained the same as it was in March – make tracks around the western edge and to the south of Home Plate, where its next big attractions await. Von Braun, an intriguing mound, and Goddard, a pit in the ground that team members suspect is a volcanic vent – named for the two leading rocket pioneers – have been on Spirit's science wish list now for more than two Earth years and at long last the rover is closing in.

Spirit route map Spirit route map
This image taken by the HiRISE camera onboard the Mars Reconnaissance Orbiter has been labeled by Eduardo Tesheiner, an active participant on UnmannedSpaceflight.com, to show Spirit's route from its arrival in the Home Plate area on Sol 743 through Sol 1871 (April 8, 2009), when the rover began suffering a series of bouts with 'amnesia.' Credit: NASA / UA / E. Tesheiner

Spirit began the month of April in the midst of a science campaign on Kit Carson, light-toned material unearthed during the final week of March. It turns out that Kit is rich with iron sulfate and opaline silica, according to Arvidson.

Given than a Mars sol is 39 minutes longer than an Earth day and the fact that the MER team uses a “hybrid solar time” for most of the planning, as Nelson explained, there was no sol that corresponded to April 2, 2009 on Earth. That Earth day was essentially lost to the time drift between the local times on the two planets, because they rotate at different rates. So, on Sol 1866 (April 3, 2009), the rover resumed driving and spent a good part of the first week of the month on the road.

While Spirit only progressed about 3 meters (10 feet) on that sol because of excessive slip, it managed to avoid troublesome terrain during the next drive sol, Sol 1868 (April 5, 2009), and put another 17.5 meters (57 feet) behind it. The rover wrapped up the first week and moved into the second week of April with a pair of drives on Sols 1870 and 1871 (April 7 and 8, 2009) during which it got another 20 meters (66 feet) closer to von Braun and Goddard.

During the second week of April, Spirit, beginning Sol 1872 (April 9, 2009), experienced a setback with a series of anomalous events. It all started, or so it seems, when the rover seriously "overslept," even though its alarm clock was sounding. “The alarm clock is a hardware countdown timer, loaded with the number of ‘ticks’ to wait until the next event – a ‘tick’ is 2 seconds,” explained Nelson. “When it counts down to zero, it triggers an interrupt to the battery control board to wake up the computer. The computer then writes new wakeup times. If, as we believed happened during the Sol 1872 anomaly, the hardware doesn't respond, the count-down timer rolls over from zero to its largest value (equivalent to 27 hours) and proceeds to count down again.” Finally, 27 hours later, the expiring alarm finally roused the little rover. But Spirit had missed three planned events and it promptly went back to sleep.

On Sol 1874 (April 11, 2009), Spirit woke up only to experience a fault of some kind, reset its computer without any command to do so, then suffered a bout of robot amnesia. That means, it did not record any data in flash memory, a type of computer memory where information is preserved even when power is off, such as when Spirit naps to conserve power. Therefore, the team really had no data, no real clues as to what exactly happened, other than what they know about the rover systems.

“A fault usually affects some subsystem,” Nelson said. “For example, we believe the fault on Sol 1874 was in the high gain antenna (HGA) subsystem, something that would mark that subsystem as effectively out of service." The HGA is what the team uses to uplink commands to the rover. "We don't necessarily think there is anything wrong with the HGA," he said. "The most typical reason we get an HGA fault is because the actuator positions are 'unknown.' The software that keeps track of where the antenna is pointed somehow loses track, then marks the actuator positions 'unknown.’ In that situation, the rover is instructed to not move the antenna at all and declare a fault.”

The fault doesn't appear to have affected the HGA pass. "We were uplinking,” Nelson continued. “But one of the effects of this fault is that it kills all the sequences. That prevented us from stowing the actuator and doing a regular shutdown. Because of that, we did not ‘right out’ the Pancam mast assembly (PMA) actuator positions, so those were not saved and the result was they too became 'unknown.' If we attempt to use the PMA after that point, we'll get a PMA fault.”

Spirit's forward view, sol 1,869 Spirit's view in early April
As of Sol 1869, Spirit had rounded the northwest corner of Home Plate and had a good view of the path forward to Von Braun, the peak to the left of this mosaic of three Pancam images.
Credit: NASA / JPL / Cornell / mosaic by Emily Lakdawalla

Even before the ground crew could get a hypothetical handle on that reset event, Spirit on the very next sol suffered a “fatal” event, something like the hard disc on your computer crashing, though the rovers don’t actually have hard disks. “On Sol 1875 (April 12, 2009), we had a 'fatal' during the HGA pass," said Nelson. "The symptoms of that indicated a corrupted file, which we have not been able to verify. We've looked at the file and it looks fine. So we're not sure what's going on there either. But the affect of the 'fatal' was that it took everything down, rebooted the computer, and because it happened during the HGA pass, the HGA actuator positions were not saved either." That left them with both PMA and HGA actuators unknown. So, between the Sol 1875 pass and the 1876 pass, there was no uplink and nothing the team could do about.

On Sol 1876 (April 13, 2009), Spirit apparently woke up to get its dsily commands and promptly declared an HGA fault because the actuators were 'unknown,' then it had another bout of robot amnesia. "It woke up in this strange mode, which we think was crippled mode, so not only do we get the HGA fault on Sol 1876, the rover didn't save any data products to flash memory," said Nelson.

That's because in crippled mode, the rover is programmed to presume that its flash memory is not working; consequently, it puts its data into the random access memory (RAM). Spirit has 128 megabytes of available RAM that can store data, as long as the rover is awake. "We believe the 1876 session took place just as it should and all the data products were created and they were saved in RAM and everything went pretty much as you would expect," Nelson said. "The only problem was it was that the rover then went to sleep, so everything disappeared, because the RAM is volatile memory."

Kit Carson Kit Carson
Spirit took this image of Kit Carson, another trough of possible near-pure silica, with its front Front Hazcam. The rover finished its close-up examination of this light-toned soil that it drug up with its broken right front wheel. It turned out to be mostly iron sulfate and oplaine silica. Cornell grad student Melissa Rice, used the panoramic camera (Pancam) to detect the hydrated minerals in the patch. Credit: NASA / JPL-Caltech

Since Spirit has been so power challenged for the last several years, the engineers instituted a nap in its daily schedule to conserve energy, which is usually taken between its main activities of the sol and the main downlink transmission of data, usually through Odyssey, to Earth. So it was on its regular schedule, but when it woke up for the Odyssey pass, it had no data to downlink. So the team had no idea what had happened. "None of our log files were saved and we've got no record," said Nelson.

The only good news really was that Spirit continued communicating with its crew throughout that weekend and into the new week, although some of the communication sessions were irregular. The rover team, fortunately, has the advantage of multiple communication options. So, even with the HGA out of commission, the rovers can communicate at a slower data rate through a low-gain antenna that does not move. Additionally, communications can be relayed by Mars orbiters via the UHF (ultra-high frequency) transceiver, a separate radio system on the rover. To avoid potential problems using the HGA, they decided to communicate by UHF relay via Odyssey or by using the low-gain antenna.

When Spirit woke up on Sol 1877 (April 14, 2009), it put itself into auto mode and was taking care of itself, just fine. “If it doesn't have any instructions from us, it's got its own little sequence of things it does in auto mode, which is, basically, wake up and execute communications, then go back to sleep," explained Nelson.

While they had no explanation for what had happened yet, they did know that Spirit was power positive, its batteries were fully charged, its solar arrays were producing energy and all temperatures were well within allowable ranges, according to John Callas, of JPL, MER project manager.

Driving in West Valley Driving in West Valley
Spirit drove 6.98 meters (22.9 feet) southeastward on its Sol 1871 (April 8, 2009). It drove backward dragging the locked wheel, as it has since its right front wheel broke years ago, then captured this view looking at the ground covered with its front hazard-avoidance camera. For scale, the distance between the parallel wheel tracks is about 1 meter (40 inches). Spirit is driving through West Valley and this drive helped move it a little closer to its next attractions. Although Home Plate is not within this image, the hill on the horizon in the upper right is Husband Hill, the summit of which is about 750 meters (nearly half a mile) to the north of Spirit's position. It was following this drive that Spirit began unexpectedly rebooting having bouts of 'amnesia.'Credit: NASA / JPL-Caltech

Spirit stayed in auto mode through Sol 1879 (April 16, 2009) as the team began systematically resetting subsystems to bring the rover back to normal operations, while continuing to investigate the strange, anomalous behavior.

Both rovers' onboard software has been updated several times to add new capabilities for the mission, and Spirit got the new R9.3 software last month. The team is looking into whether the unexpected behavior in recent days could be related to the new software, but it's not likely. "There's no evidence to suggest it has anything to do with the new flight software," said Squyres. "The same software is on both vehicles and we haven't seen any evidence of any of this on Opportunity. But one can imagine ways in which there are complex hardware-software interactions that are unique to the Spirit vehicle.” Or, it could just be "age-related effects," as Callas has suggested.

Just as Spirit seemed to be returning to normal, however, the rover woke up and suffered another bout of amnesia on Sol 1880 (April 17, 2009), then once again rebooted its computer on the following sol, behavior that was now getting pretty eerie. Once again, Spirit put itself back into auto mode, just chilling and keeping itself healthy. They could find no causal link between the amnesia events and the reboots. "This is a true anomaly," said Nelson.

The team revised plans in order to regain Earth control of the rover's operations and resumed what diagnostic and recovery activities they could. "We are proceeding cautiously, but we are encouraged by knowing that Spirit is stable in terms of power and thermal conditions and has been responding to all communication sessions for more than a week now," Sharon Laubach, chief of the rover sequencing team, which develops and checks each day's set of commands, announced early last week.

Past those undesirable incidents, Spirit, being the good-girl rover it's always been, was "well-behaved," according to the mission mangers. Even Mars took sympathy and offered up a small dust cleaning on Sol 1881 (April 18, 2009), something that boosted not only the rover's power by more than 10%, but every team member's spirits by an incalculable percentage. Still, three amnesia events in 10 days, plus one on January 25 of this year were not events that brought smiles to the faces of the MER team.

The JPL engineers considered all possibilities. During diagnostic activities, Spirit successfully moved its HGA and its Pancam mast, indicating there weren't any mechanical issues with those components that could be related to the problems. They further worked to determine if some capabilities were ultimately shown to be diminished what the best way to keep using the rover would be. "For example, if we do determine that we can no longer use the flash memory reliably, we could design operations around using the random-access memory," Laubach said.

“Like everything else with these vehicles, as they age, the best thing to do is to come up with the best way to work around it,” said Squyres.

Home Plate in 3-D Home Plate in 3-D
Spirit acquired this full resolution mosaic of Home Plate on Sol 1886 (April 23, 2009). The image was taken before a drive, as the rover made its way around the western edge of Home Plate. The field of view spans about 90°. This image was created from Pancam's 753-nanometer, 535-nm, and 432-nm filters. This stereo anaglyph simulats the scene in 3-D, so get out your red-blue glasses and take in the view. You're on Mars.Credit: NASA / JPL-Caltech / Cornell University

But the incidents were so bizarre and so confounding, they decided to convene their "Council of Experts," as Nelson put it, "the people who were involved in actually building and programming the rovers." That council includes Matijevic, former chief of MER's spacecraft and rover engineering team, Jim Donaldson, the lead engineer for the Avionics Subsystems, whose team designed and built the electronic boxes that contained the rovers' computers, as well as interfaces for communicating with other systems; Glenn Reeves, chief flight software engineer and software architect; Tracy Neilson, fault-protection software writer; John Waters, electronics engineer; Todd Litwin, software developer; Dave Hecox, software developer, and Rob Manning, development manager for the rover's descent through Mars' atmosphere and landing on the surface.

"These are the big stars from the development days of the vehicles," said Squyres "and they were as baffled as we were. So we have an interesting puzzle on our hands. But given the nature of this team and the smart engineers we have, it's actually been something that's been quite a stimulating thing to try to deal with and we're working it."

The council made a number of suggestions, however, that are being instituted. “One is to have Spirit stay awake through the Odyssey pass, so if it's stores data in RAM, we keep the power on until after we've had the chance to at least downlink the highest priority stuff before we shut down," Nelson explained. And, because there is a need to get back on the road and keep trying to get to Goddard and von Braun, the council, after determining that it wouldn't hurt for the rover, also recommended that Spirit carry on with the understanding that these events may occur again.

Slippery soil Slippery soil
Spirit slipped in soft ground during short backward drives on its Sols 1886 and 1889 (April 23 and 26, 2009). The rover used its front hazard-avoidance camera after driving on Sol 1889 to get this wide-angle view, which shows the soil disturbed by the drives. It drove 1.11 meters (3.6 feet) on Sol 1889 and 1.68 meters (5.5 feet) on Sol 1886. The rover drags its right front wheel, which no longer rotates. For scale, the distance between the wheel tracks is about 1 meter (40 inches). This view is looking northward, with Husband Hill on the horizon.Credit: NASA / JPL-Caltech

"We decided not to wait until finishing the investigations before trying to drive again," Laubach said. "Given Spirit's limited power and the desire to make progress toward destinations to the south, there would be risks associated with not driving."

In addition, the team is considering reformatting the flash drive, which would be a real pain in that it means taking the memory off the rover or clearing the slate, reformatting, then putting everything back on, much like users of pcs have to do when their systems go wonky. And, they're thinking about putting in some software triggers that will alert them if the rover goes into certain modes and save that data in some non-volatile memory. “Since we can't guarantee it would save into flash, and of course we can't guarantee Sprit will be awake if it's saved into RAM, we're trying to work out a way where we could save it into EEPROM – electrically erasable programmable read only memory," said Nelson. "That's a little tricky, because that's an area that's normally an area only used for sequences, so it's not easy to have the sequence change the sequence memory, although it's possible. We're looking into it."

For now, the plan ahead is to treat Spirit as if it's healthy and to resume near-normal operations. "Other than these anomalies, the rover is healthy," underscored Nelson. “It's power-positive. It's thermally stable. It's commandable. So, even though we know the rover isn't fully 100% and we don't understand why it's doing these things, as long as it's functioning, we're just picking up and starting over again,” he said. And, they have changed the rover's siesta time to before the Odyssey pass instead of afterward, so now any data the rover saves to RAM will not be lost.

“Clearly we have some kind of poorly understood intermittent problem, but this team is a team of problem solvers and when something like this comes along, it keeps the job interesting,” said Squyres. "We're just going to work around it as best we can and keep pushing southward. Maybe we'll get lucky and get some information along the way that will help us better understand what is it on the vehicle that's making these things happen."

With the next Martian winter on the not-so-distant horizon, time is now really of the essence for Spirit to get to von Braun and Goddard and they scheduled a short drive for Sol 1886 (April 23, 2009). The rover responded to all commands last Thursday and, for the first time since Sol 1871 (April 8, 2009), Spirit roved on, albeit for a very short distance of about 1.7 meters (5.6 feet), because of the slippery, powdery soils.

A view of Von Braun A view of Von Braun
After driving on Sol 1871 in West Valley (April 8, 2009, Spirit used its navigation camera to capture this view of the terrain to the southeast. The mound on the horizon in the upper left is Von Braun, the rover's next attraction, which was, at this juncture, about 160 meters (525 feet) away.Credit: NASA / JPL-Caltech

Spirit drove again on Monday, its Sol 1890, and again only progressed a few centimeters. Actually, the drive was automatically stopped because of excessive slipping in the fluffy terrain. “It took one step forward and one back,” said Nelson. “Because we drag this broken right front wheel, the rover tends to yaw counter-clockwise, so we move forward and then we try to correct the yaw by doing a turn that brings us back to the direction we want to go, then move forward again, then turn to correct and so forth.” As a result, it’s been slow going in this area."

“It's been easy until now, but the drive over the weekend was slightly up hill and in this sandy material," said Arvidson. “We exposed some more light-toned material, but didn't have time to stop and smell the roses. We’re really putting all the watt-hours into driving and post driving imaging.".

“We're getting into some challenging terrain,” added Squyres. “We'll deal with it as we always do, one step at a time. No one expected this drive was going to be easy the whole way and there are some tricky patches along the way. We seem to have gotten into one and we'll see how it goes."

Meanswhile, Mars continued to be supportive and sent winds to clear more dust from Spirit's filthy solar arrays just a couple of sols ago, Banerdt informed, allowing about 42% of the sunlight hitting the solar panels to get through and be converted to energy. That boosted the rover's power levels to 371 watt-hours, giving Spirit energy it hasn't had in more than an Earth year.

During the last day or so, the team has decided to have Spirit to change tact in the West Valley. “We're looking at perhaps backing out of the loose soil that we seem to be kind of bogged down in and moving around a little bit to the right and downhill a bit,” said Banerdt. “There's sort of a high road and low road, as we're referring to it. The high road goes uphill a little bit and opens up into a wider sort of range in which we can drive south, but we've been unsuccessful of getting up that lip. Alternatively, we can go just a little to the west and down the low road, which is a fairly narrow pathway that is kind of rocky and has some dips and depressions filled with loose soil, little areas with which Spirit has had trouble before. There's not much room to maneuver in there, so if we run into problems going down this corridor, it could be dicey. Both ways are a little hairy. But we're going to take a shot at going on the low road now.”

Home Plate in living false color Home Plate in living false color
Spirit acquired this mosaic of Home Plate on Sol 1886 (April 23, 2009). The image was taken before a drive, as the rover made its way around the western edge of Home Plate. The field of view spans about 90°. This mosaic was created from Pancam's 753-nanometer, 535-nm, and 432-nm filters. This full resolution false color full resolution: an approximately true color rendering, a false color version was stretched to bring out subtle color differences in the scene.
Credit: NASA / JPL-Caltech / Cornell University

This weekend, as May blooms at Gusev Crater, Spirit will continue to moving toward the low road to head south through the more narrow pathway. Von Braun and Goddard are about 150 meters (about 500 feet) away, but are now in clear site. The rover is about two-thirds of the way there and it's only a matter of negotiating the difficult terrain.

The agenda for Spirit in coming sols is a lot like that of its twin: “Drive, drive, drive,” as Arvidson put it. “Our intention is to continue to the south,” confirmed Squyres. “Right now I would say there are no indications that suggest we cannot continue to the south. We don't view the path to be blocked in any substantive way right now. We'll just press ahead. If we learn more and the situation changes, we'll adapt to it. We always do."

Meanwhile, the engineers are ready for more of same reboot and amnesia anomalies. "As strange as it may seem, I am actually hoping that it does, because I want us to have a chance to identify what's happening so we can fix it," said Nelson. "We need data and we're going to run this rover as if it's problem free until trouble shows up again.”

opportunity from meridiani planum

Opportunity route map Opportunity route map
This image taken by the HiRise camera onboard the MRO, labeled by Eduardo Tesheiner, an active participant on UnmannedSpaceflight.com, shows Opportunity's route since its arrival at Victoria Crater to Sol 1870 (April 28, 2009).
Credit: NASA / JPL-Caltech / UA / E. Tesheiner

As April dawned at Meridiani Planum, Opportunity was in the midst of finishing up its close-up examination on the stretch of exposed rock outcrop called Cook Islands, homing in on Penrhyn, a target named for the most remote and largest atoll of the 15 Cook Islands on Earth.

It's part of a systematic study of bedrock and soils that the rover is conducting on its long journey to Endeavor, the 22-kilometer (14-mile) diameter that the team hopes to get to sometime in 2010. The rover continued that work through the first week of the month.

On Sol 1845 (April 2, 2009), Opportunity positioned its robotic arm or instrument deployment device (IDD) and gently placed its Mössbauer spectrometer on the outcrop target Penrhyn for a multi-sol integration, to determine its iron content.

The stop for science came an opportune time, since Opportunity was forced to take a break to rest its right front wheel. About two months ago, that wheel was getting a little too hot from all the driving the rover had been doing. The actuator or motor was drawing more than normal current, a symptom the team had seen before fortunately. Basically, it indicated that the lubrication inside the wheel assembly was unevenly distributed, something that happens after a period of intense driving and they knew just what to prescribe: driving backward, a mitigation technique to redistribute the lubricant, and periods of rest.

By the end of the first week of the month, Sol 1850 (April 7, 2009), the team provisionally lifted the temporary stand-down on driving.  Opportunity retracted its IDD and moved into the driving-stow position, then roved backward about 62.5 meters (205 feet). The drive went well, but the good news was that the right-front actuator exhibited currents near normal levels. More good news came in the form of another cleaning event, this one boosting the rover’s power levels by 40%, to 515 watt-hours. The dust factor improved to 0.642, meaning that 64.2 percent of sunlight hitting the solar array was penetrating the layer of accumulated dust and being converted to energy.

“We never really managed to get Opportunity very dirty," said Squyres. "And so the cleaning event is certainly welcome, but not really a surprise given its history."

Back on the road to Endeavour, Opportunity followed the course charted by its planners, veering to the west, away from a sea of purgatoids and by a series of small, shallow craters that were too inviting to just fly by. These small craters -- just a few meters or yards in diameter – are, the scientists believe, from fairly recent impacts, occurring in the last 100,000 years, according to Matt Golombek, MER science team member and JPL senior research scientist. 

Marquis and Adventure Marquis and Adventure
Opportunity took this image – which was recently mosaicked by Cornell's Pancam team – on its Sol 1854 (April 11, 2009). It shows Marquis Crater in the foreground and Adventure Crater in the background. "When these big ripples formed, they required lots and lots of blowing fine sand to drive the coarser particles in creep mode," explained Robert Sullivan of Cornell. "The coarser particles, such as the blueberries, probably weren't blown directly by the wind in saltation, because they were probably too big and too heavy for that. But smaller grains that are much more easily driven by the winds could have bashed them along. We don't know if fine basaltic sand was the main driving material for these large ripples, but it's a handy candidate to point to now, because small quantities are present currently.  This type of sand may have been much more abundant at the time the larger ripples were on the move. The hematite concretions that are all around this area are very hard and difficult for the wind to move, so once you form these ripples, it's kind of hard to keep them moving unless you maintain a large supply of basaltic sand handy, ready to be driven by the wind." Credit: NASA / JPL-Caltech / Cornell

The team named them Discovery, Resolution, Adventure, Rayleigh, and Marquis after Capt. James Cook’s ships (other than the Endeavour). “Because of their proximity to one another, there’s a good chance that these craters formed at the same time," noted Squyres. "That means they're all the same age and by looking at how craters of different sizes have aged, and if you know they've aged at the same time, you can actually learn something about the crater degradation process.”

They could be primary craters created directly by impacts or secondary craters, formed by fallout emanating from bigger impacts. Some of them are, intriguingly, right smack on top of the ripples.

Opportunity slowed down and from Sols 1854 to 1858 (April 11 to 15, 2009) conducted some drive-by shootings with its Pancam. “We took a lot of images of those craters as we passed through that field,” Squyres continued. “It will take some time to create the stereo models and do all the geometry to really work out the science associated with it, but we've collected what I think is going to be a really good data set on those.”

“This cluster is similar," said Golombek, "to many craters observed to form in the past 10 years. They are younger than the large ripples and scattered across the craters are small dark pebbles. Because the pebbles are superposed on the craters, they must be younger or have come with the crater and thus impact derived."

Opportunity has been seeing these pebbles since Eagle crater, Golombek pointed out. "The question is: are the pebbles meteoritic or if the craters are secondaries pieces of Mars from elsewhere?”

The MER scientists are now on the lookout for pebbles big enough pebble to analyze in order to determine which of these two possibilities is correct.  “We're keeping our eyes open for either concentrations of these that we can look at – like the blueberries back at Eagle – or perhaps we can find larger samples,” said Banerdt.

Once it finished with its photography, Opportunity hit the road again, using a combination of forward and backward driving. The longer drives – such as those on Sols 1859 (April 16, 2009) for 62 meters (203 feet) and on Sol 1860 (April 17, 2009) for 88 meters (289 feet) – included slip checks. Happily, the wheel currents all showed normal current levels.

Thrills on blueberry hill Thrills on blueberry hill
Opportunity captured this blueberry-rich scene with its Pancam on Sol 1861 (April 18, 2009) while en route to Endeavour Crater.
Credit: NASA / JPL-Caltech / Cornell

During the last couple of weeks, Opportunity's agenda has shifted almost completely to again driving. Last week, it drove for three consecutive sols, logging 96 meters (315 feet) on Sol 1863 (April 21, 2009), 137 meters (449 feet) on Sol 1864 (April 22, 2009), and 95 meters (312 feet) on Sol 1865 (April 23, 2009). It seemed like nothing could stop this rover, then on the last of those drive it experienced a close encounter small purgatoid or "purgatina," as Banerdt put it.

Although Opportunity wound up having to spend a little time extricating itself, it was "only a slight embedding event," said Squyres, and "not a big issue. The backup drive on Sol 1866 (April 24, 2009) had only limited success, with Opportunity moving only about 28 centimeters (11 inches) before limits stopped the drive, but on the next sol the rover completed the backup, roving away for about 3.7 meters (12 feet). The rover drivers now have experience getting out of these sneaky sand drifts, and they also have experience “not getting into them too,” Banerdt noted. “What stopped the rover this last time was the slip check. There's a certain amount of slip it will tolerate, then it stops the drive and waits for further instructions.”

Spirit and Opportunity have always had the capability of doing slip checks. In fact, when Opportunity got really stuck way back in 2005 in Purgatory Dune, it was because the team had disabled the slip check feature. “Everything had been working so well, it was like we don't have to watch where we're going, Banerdt said. Safety takes time. “The slip checks require the rover to stop and do some imaging and analysis. That tends to slow the rover down. The frequency with which you do slip checks and the limits you put on them have a big affect on the maximum drive you can accomplish,” he explained. “There's a trade-off between safety and boldness in terms of how much terrain you can cover.”

Actually, Squyres in his talks and presentations on the rovers reflects back sometimes on those 5 excruciatingly long weeks in Purgatory Dune. “What we were doing was equivalent to driving down the freeway at 100 miles per hour with our eyes closed," he's said. It turned out to be a lesson learned in prudent driving for the whole team. And Opportunity has never gotten stuck like that since.

Onward Onward
Opportunity put the pedal to the metal this month and left Victoria Crater, its home for the past 2 years, for good. It's been making regular 100+-meter drives and last weekend nearly broke its own record when it logged a 216-meter drive. This image, which the rover took with its Pancam was enhanced with humor by Stuart Atkinson, an active participant on UnmannedSpaceflight.com.Credit: NASA / JPL / Cornell / Stuart Atkinson

Last week, Opportunity took off again, like a bat out of -- well you know, and made some more really serious tracks. “We managed to get back onto better terrain," said Nelson. "In the last week and a half or so, we've driven another half-kilometer of distance, so it's really been moving along,” Banerdt added.

This week, Opportunity logged another 64 meters (210 feet) during drives on Sols 1870 and 1871 (April 28 and 29, 2009), which took it around the centerline of one of "these little noses that stick out" from the sea of purgatoids, as Nelson described it. But, on Sol 1871,they noticed a return of the increase in the amount of current drawn by drive actuator in the right front wheel. Now the engineers are again considering mitigation practices of resting the actuator and/or driving backwards for a while.

MER Update Archives

Support our core enterprises

Your support powers our mission to explore worlds, find life, and defend Earth. You make all the difference when you make a gift. Give today!

Donate