Mars Exploration Rovers Update: Opportunity Logs Sol 4000, Digs Spirit of St. Louis Crater

After investigating some flat, light and dark toned rocks around Spirit of St. Louis Crater in April, Opportunity chalked up another milestone achievement for the Mars Exploration Rovers (MER) mission – the 4000^th sol or Martian day of surface operations – then did what she does best. By month's end, the robot field geologist was closing in on Lindbergh Mound, the strange rockpile near the small crater's rim.

“There was something about seeing that number with the 4 and all those zeros and realizing, 'Okay, we are today planning Sol 4000,'You kind of go, 'Really?  Wow!'" said Steve Squyres, MER principal investigator, of Cornell University, in a moment of reflection. "We just cracked four thousand. I don't know what else to say other than it’s astonishing and gratifying at the same time.”

John Callas, MER project manager at the Jet Propulsion Laboratory (JPL), home to all of NASA's Mars rovers, was on the same page. "Sometimes,” he said, “you have to pinch yourself.”

It has been written many times in these pages, and it begs repeating: this rover was sent on a 90-day expedition, with the mission success mobility objective of driving 600 meters. In March, Opportunity completed 42.195 kilometers or 26.2 miles. It’s the first marathon “run” on another planet. And in April – the 4000^th sol. “This rover just keeps giving and giving,” said Planetary Society President Jim Bell, professor of astronomy and planetary scientist at Arizona State University and lead scientist on the MERs' panoramic cameras (Pancams).

So does the MER team. Down to a skeleton crew on a skeleton budget compared to Curiosity, the veteran rover and her ops team are still roving to new places and seeing new things every month, sometimes every week. Most importantly, they are uncovering history, producing solar-system-class science, and, chapter by chapter, writing the story of a time when Mars was more like Earth, a time no other mission is investigating.

Opportunity is blazing some of the first human trails on Mars and opening the windows on an alien planet that now looks familiar. The robot is still returning data almost every day and sending home rich, big picture panoramic postcards of breathtaking landscapes, like the one released this month of Spirit of St. Louis Crater. In these panoramic scenes, the planet humans want so desperately to explore is revealed in all its glory, as if you were standing right there looking out. “It’s been spectacular and continues to be spectacular,” said Bell. And so far, the radiation hasn't impacted the rover's brain.

The veteran rover spent much of April in workaday mode, finishing up the science campaign on the light and dark toned outcrops in the apron area surrounding Spirit of St. Louis Crater, the unusual, oblong-shaped depression that lies on the outer portion of the western rim of Endeavour Crater, and right at the entrance of Marathon Valley. From there, Opportunity drove to the rim of the small crater and 'landed' on a mound of rocks dubbed Lambert Field, as in the St. Louis International Airport. After checking out a rock there, the rover took a toe-dip into the small crater, and then backed out, and drove around to its southern end, to face Lindbergh Mound.

As productive a month as it was in terms of accomplishing science assignments, April wasn’t without issues. The robot field geologist suffered five more amnesia events. “Each of those were benign, and no science was lost," said Bill Nelson, chief of MER engineering at JPL. But the, the rover also suffered an unexplained and sudden reboot – and that is never good.

Opportunity and her software team had successfully reformatted the flash drive in March, masking off a corrupted sector, known as bank 7, and the rover returned to using six banks of storage in the drive. Generally, things have greatly improved, but this sudden reboot is unlike any the team has seen either with Opportunity or Spirit. “We know there was an error, and we’re still investigating exactly what the cause of that was,” reported Bekah Sosland, a MER flight director and the flash tiger team lead at JPL.

While Opportunity was electronically unscathed, the warm reset is a concern. "It’s an eleven-and-a-half-year-old rover – I mean, every day is a concern," said Squyres. "Considering how old the vehicle is and everything that we put it through, a little funkiness from time to time is not unexpected. We deal with it as it comes."

To the mission’s good fortune, Mars was “cooperating” as human, vicarious Martian explorers like to say. Summer dust storms swirled around the northern hemisphere of Mars causing the skies over Meridiani Planum and Endeavour Crater to be hazier in April. But the weather where Opportunity is, south of the equator, remained storm free.

Despite the really hazy skies – the Tau or atmospheric opacity hovered at or slightly above 1.010 throughout the month – the solar-powered rover’s energy was fairly robust. Opportunity was producing in the range of 560 to 600 watt-hours, more than half her full capability. Notably, the dust factor remained strong, fluctuating between 0.701 and 0.744, likely because the summer breezes were gently removing more dust from the rover’s solar arrays than was “raining” down from the haze overhead.

In coming sols, Opportunity will drive, at long last, into a mother lode of clay minerals in Marathon Valley. These clay minerals, known as smectites, form in near neutral water, as in water we humans might be able to drink. They were discovered back in 2008 in data from the Compact Reconnaissance Imaging Spectrometer for Mars (CRISM), an instrument onboard the Mars Reconnaissance Orbiter (MRO) that looks for signs of present and past water.

Equipped with that data and additional CRISM data, Ray Arvidson, MER deputy principal investigator of Washington University St. Louis, created a map to zero in on the best patches of real estate in Marathon Valley for Opportunity to check out. The clay minerals are there, just waiting to be found.

“We drove long and hard to get to this place,” said Squyres. “And now the fun is ready to begin."

When April dawned at Endeavour Crater, Opportunity was in the midst of the science campaign of the flat, light and dark toned outcrops in the southwestern part of the apron area of Spirit of St. Louis Crater. The robot field geologist essentially spent the first week of the month gleaning the chemical compositions of different chosen targets on the outcrops called in the Thessaloniki and Thermopylae with her Alpha Particle X-ray Spectrometer (APXS).

Following two sols of looking at the chemical composition of different spots on Thessaloniki, Opportunity bumped to Thermopylae, driving just 6 meters (almost 20 feet) on Sol 3979 (April 3, 2015). The rover reached out to check out a new outcrop surface target, taking close-up pictures with her Microscopic Imager and sleuthing the chemistry with her APXS. The robot also took atmospheric argon measurements with the APXS for the subsequent two evenings, for the mission long study on the Martian atmosphere.

However, during the evenings of Sols 3979 and 3980 (April 3 and April 5, 2015), Opportunity suffered bouts of amnesia. Both events were benign with no loss of science data, and put the software engineers back to work.

The plan was for Opportunity to end the week, Sol 3982 (April 7, 2015), continuing work on Thermopylae, but a DSN problem prevented the sequence plan from being transmitted to the rover. So, Opportunity resorted to conducting other tasks, carrying out another sequence that is already onboard and waiting for the times when these things happen.

Since February, Opportunity has been finding a diversity of rocks, each a little different than any rock found before on the expedition. From Jean Baptiste Charbonneau in February to Sgt. Pryor and Sgt. Charles Floyd, and then the really bright-toned outcrop Athens in March, and in April the flat outcrops near Athens, dubbed Thessaloniki, and Thermopylae, the robot has uncovered a diverse bounty of new rocks the likes of which the rover has never before seen.

Although the light-toned rocks looked at first a little like Whitewater Lake Formation, the oldest strata the rover uncovered on Matijevic Hill in the Cape York segment of Endeavour's western rim, “they’re not,” said Arvidson. “But they’re still not like anything we’ve seen before.”

As the second week of April took hold, Opportunity kept her “nose” to the light toned stone Thermopylae. After taking close-up pictures with her Microscopic Imager (MI) for a mosaic on Sol 3984 (April 9, 2015), the robot later placed her APXS on the target for a multi-sol integration to determine its chemistry.

Meanwhile on Earth, members of the MER ops team at JPL and others who work at the Lab celebrated Opportunity's marathon on April 9^th with a relay marathon. More than 90 runners on eight teams hit the course of the first Mars Marathon on Earth. It took them around the NASA center’s perimeters.

"It was a nice way for individuals to have a personal involvement in the marathon on Mars," said Callas. " People are recognizing the history of it and there were a lot of people who had no direct connection with MER who participated and were excited to do so. A number of them came up to me afterwards and thanked me for the opportunity to be part of this historic event." For more coverage see: MER Special Update: MERathon Celebrates Opportunity's Marathon.

"We had a sense early on, during the initial development of the rovers, that mobility would be critical, would be the absolute key to this mission," said Bell. “But to be able to exploit it in such a way, really treat it like another instrument, a completely other part of the overall investigation is pretty incredible and very rewarding. Remember, we would have been happy to go 600 meters in 90 days, right?”

Back on Mars, Opportunity turned her attention to a different target within the work volume on Sol 3986 (April 11, 2015), and repeated the same MI-APXS set of science observations. It was all going swell until the evening of Sol 3987 (April 12, 2015) brought on another amnesia event, and then another the following evening of Sol 3988 (April 13, 2015).

Both were benign and resulted in no loss of science data. Opportunity was otherwise in excellent health. But the flash tiger team and the software engineers on Earth were feeling some frustration, according to several reports, understandably.

On Sol 3989 (April 14, 2015), the rover bumped about 60 centimeters (24 inches) into a position so to reach other surface targets on Thermopylae. Opportunity also carved out time to take some additional opacity measurements and battery data to support the InSight mission.

As the third week of April began, Opportunity was still at work conducting off-set investigations of different surface targets on Thermopylae. On Sol 3991 (April 16, 2015), she took the pictures needed for an MI mosaic and then placed her APXS on the target for a multi-hour integration. The rover also took images with her Navigation Camera (Navcam) for a small mosaic.

The next sol, the MER team put Opportunity on Navcam Dust Devil Watch. “We have rarely but occasionally seen dust devils with Opportunity, though not like Spirit,” said Arvidson. No reports and no images would be forthcoming in April.

On Sol 3993 (April 18, 2015), the robot took more MI mosaics and conducted more APXS integrations. “We were looking at the chemistry of both the light areas and the dark spots in these outcrops, to see if there are differences,” explained Arvidson.

The next sol, 3994 (April 19, 2015), Opportunity moved her APXS to an adjacent target for another integration. Then, the rover began to prepare for the afternoon UHF relay pass, where she uplinks her findings to the Mars Odyssey orbiter, which in turn downlinks it to Earth via the Deep Space Network (DSN). But suddenly, in the midst of that preparation, the rover suffered a warm reset that stopped her in her tracks. To make matters more frustrating, she had another an amnesia event later that night during the wakeup to start Deep Sleep.

Opportunity bounced back. By Sol 3996 (April 21, 2015), the rover was online so to speak, under master sequence control, and ready to rock and roll. The technological issues are pesky and annoying and puzzling, but they’re not enough to stop this rover. Still, the new amnesia events are puzzling and the warm reset or sudden reboot is a definite concern.

The reformat in March and masked off a corrupted sector or bank, specifically bank 7. was deemed successful. Opportunity almost immediately went back to storing her daily data in the flash drive and has been using this long-term memory ever since. “We are still not seeing any errors writing to the flash memory, since masking off Bank 7. That in itself is a huge step forward,” pointed out Sosland.

As familiar as the amnesia events seem, given the successful reformat, they don’t know the exact cause. “The thinking is that there might be some relationship to the flash memory, but we don’t know that,” said Nelson. “We believe the flash bank 7 problems are behind us, but we know very little.”

The reasons for that are pretty straightforward. When Opportunity has an amnesia event, there aren’t a lot of event records or EVRs in the computer’s log. The engineers did see a record this time that basically says ‘failure to mount flash' / 'creating fallback in RAM.' That means, in essence, that the rover's computer system is going into what was long ago dubbed 'crippled mode,' which is RAM mode.

The problem is that when the rover’s system falls back to RAM, whatever data the team may have been expecting to save is lost in that wake-up, because RAM is volatile memory that disappears when the rover shuts down for the night.

While amnesia events are crippled mode events, not all crippled mode events produce amnesia. "If, as we occasionally have, keep the rover awake through the afternoon UHF pass, whatever is collected in RAM can be returned to Earth, and from this we know about the 'failure to mount flash' and 'creating fallback in RAM' EVRs," explained Nelson. "In a true amnesia event, we go into crippled mode, but don’t have a chance to return any telemetry before shutting down. This creates a 'hole' in the expected telemetry when nothing is returned."

Most of Opportunity’s amnesia events occur during the late evening wake-up and an APXS integration and start of mini-sleep and in the end it’s kind of a no-harm/no-foul situation. Since the APXS instrument also has a flash drive, it records the data, which is then read-out the next morning and downlinked, and so no science is lost. 

“But we have had now three events that have happened on the wake-ups to starting regular DeepSleep,” said Nelson. The first one happened back on Sol 3817 (October 20, 2014) one sol after shooting Shooting Comet Siding Spring, the second on Sol 3944 (February 26, 2015), and this month on Sol 3988 (April 13, 2015).

“Each of those happened at our regular DeepSleep time,” said Nelson. “The nominal start time for DeepSleep is around 18:30 local Mars time. And these three events have all been auto-mode wake-ups. In other words, we haven’t been doing anything other than our standard wake-up to shut down.” 

The flash tiger team and all the MER software engineers, past and present, are considering all the possibilities. "We’re looking at possible failure modes that might possibly explain this,” said Nelson. “We’re looking at the history of the parts in the sense of how much accumulated radiation might they have been exposed to?  What other parts may be affected by it? We’ve got a lot of parts actually to investigate.”

They are also looking at the code, of course, to try to figure out what is going on. And of course, it’s complex. Three layers of software and code form the artificial life that is this rover, and they each interact and they each came from different sources. “There’s our code; the true flash flight software FFS, which we got from SanDisk; and the real-time operating system (RTOS) from Wind River,” said Nelson.

One layer of software talks to the next, which talks to the next. The objective is to tease out what’s happening. “Except that the software don’t reveal what’s happening as it’s happening and they’re each doing a lot of different things,” said Nelson. “There are a lot of different possibilities, which is why doing the software investigation is taking such a long time. There are just a lot of pathways that ultimately get us to the cripple mode."

The sudden reboot or warm reset, though, is even more confounding, if only because the “symptoms” are different from any other event the MER team has experienced with either Opportunity or Spirit. “This is a new issue," said Nelson.

“We don't know the origin and one of the symptoms, one of the observables doesn't make sense, so it needs some investigation," summed up Callas.

The observable, as Callas called, it, came in the form of an EVR that informed: ‘Fatal EVR / Bad address.’ “Apparently somewhere – we still don’t know where – but somewhere in the code, an address, a reference was created that points off into never-never land, to non-existent hardware,” said Nelson. “We suspect that it is somehow related to the flash memory. But we don’t know that yet.”

The software engineers are determined to find out. Sosland her flash tiger team and the MER ops team have secured the help of a group of the original MER software engineers – Jim Donaldson, Glenn Reeves and Todd Litwin, Alan Lee, and Tracy Nielson. “They are reviewing their notes from 12 years ago and are currently conducting a fishbone analysis,” said Sosland.

Opportunity seems to be taking it all in stride as she rovers on. Actually, the software glitches haven’t seemed to phase the rover at all. Other than the amnesia events and the sudden reboot, the rover is doing “very well,” by all accounts.

"Since we reformatted six weeks ago, the vehicle has been very stable and very productive,” added Callas. “If this is as bad as it gets, then we can live with this for a long, long time.”

Added Nelson: “We’ve had good power. We’ve been doing our science investigations. We’ve been taking our imaging. We done a couple of bumps, moved around a little bit. We are in good shape.”

“Every day on Mars is a gift,” said Squyres. “We take it a day at a time.”

These are the mantras the MER team, rover included, lives and breathes. It seems to be working for them.

“Just when you think something is coming along, it's like, ‘Oh, this is going to be really problematic,’ the response is like, ‘Oh no.  We can figure that out.  We got a way to deal with that,’" reflected Bell. "There's never been a point in this mission where I was just suddenly amazed at what this team's able to do. It's been just continuous from the beginning, and to see new people come in and make contributions, it’s very, very exciting.”

On Sol 3998 (April 23, 2015), Opportunity left Athens, Thessaloniki, and Themopylae, and drove 8 meters (a little more than 26 feet) to reach the rim of the Spirit of St. Louis Crater, and a small mound of rocks just inside, dubbed Lambert Field.

The robot field geologist had spent about a month examining the light and dark toned outcrops around Spirit of St. Louis Carter and the analyses on Earth are still underway. "It's too early to say but they're definitely rocks that are different than anything else we've seen,” said Arvidson. "It looks like they have been altered and we're trying to figure out the nature of the alteration, but there's definitely a compositional difference between the really bright parts of the outcrop and the darker parts," he said.

"We don’t have a good story for you on that yet, nothing that we can really take to the bank yet,” agreed Squyres. “Much of the APXS data is down [on Earth], but we really need to have a good discussion about it to kind of put the story together. That hasn’t happened yet. There wasn’t anything to kind of leap up in the air, startling about it. But we’re working our way through it.”

As Opportunity roves on, it may become overwhelming. Just in the last couple of months, she has found a diversity of rocks like never before. “It continues to amaze us – we got the purple rocks that are different than the blue rocks that are different than the soil,” Arvidson recounted. “And now we have these bright rocks on the apron of Spirit of St. Louis Crater that seemed to be altered by fluids."

The rocks are deemed purple and blue by virtue of the color they take on when the Pancam images are processed in false color, as covered in last month's MER Update.

It's a technique that enables scientists to better discern the different geologic components in a given scene.

Rock by rock, the MER scientists are gathering together the clues needed to unravel the mystery of what Endeavour Crater was like three to four billion years ago. One thing seems certain: Mars once really was once more like Earth. "The more we look, the better it gets in terms of warm wet conditions either on the surface or in the interior," summed up Arvidson.

With those outcrops behind her, Opportunity woke up on April 26^th to her 4000^th sol or Martian day of surface exploration. To celebrate, she drove 5 meters (about 15 feet) from the edge of St. Louis Crater to just inside the crater, onto “an outcrop peninsula,” as Mike Seibert, MER rover planner and lead flight director, described it.

"It’s easy to say and it’s true: 4000 is just another number,” said Squyres. “My feeling going in would have been ‘Eh, 4000, okay, it’s 1 more than 3999.’ But actually seeing it up on the screen...these round number milestones, you hit them, and they feel good. They feel real, real good.”

As Opportunity focused on a purple rock at Lambert Field, carrying out the routine MI pictures and APXS integrations, NASA released the mission's latest big picture, a panorama of Spirit of St. Louis Crater.

When Bell processed the image created from images the rover took at the end of March, it came out as crisp and breathtaking as any jumbo “postcard” the robot field geologist has sent home. "The front sapphire windows that cover protect the cameras have gotten dusty on and off throughout the history of the mission,” said Bell. “And when I put together the last few big panoramas over the past two years, they've been challenging in that the dust on the front window makes it very difficult to get seamless mosaics because there are these large gradients of brightness [from one single column of image to the next]."

But this time, when he hunkered down with a cup of tea "to do battle again" with the dusty seams, Bell got a surprise. "It just came out beautiful on the first try, which is pretty rare,” he recounted. “That means the windows have been cleaned relatively recently. So apparently, Opportunity has had some relatively recent dust cleaning events that cleared up those windows."

It is summer in the southern hemisphere of Mars and summer is the season for windy sols and dust storms.  There have been a number of storms, including a regional guster in the northern hemisphere that have increased the opacity in the skies. At the same time, local wind gusts have sent whisking breezes across Opportunity. It’s the give and take of dust on Mars.

"We are watching a couple of storms out there," said Callas at month’s end. “But Opportunity is in good shape right now, so it's continuing moving forward.”

Actually, Opportunity drove out of April and into May en route to Lindbergh Mound. Looking something like a Martian cairn, Lindbergh Mound resembles those stacks of rocks left on beaches around the world that have tumbled from their one majestic heights by the forces of nature.

Whatever Lindbergh Mound is, the science team had already chosen a blue rock for the robot field geologist to examine – which may or may not be like the blue rock Sgt. Charles Floyd examined in March. Perhaps too, the rover might find “a piece of the thing that formed the crater in the first place,” mused Squyres.

Once the rover is done checking out Lindbergh Mound, if all goes according to plan, she will head up rim to a few other rocks of interest. Then, finally, after nearly seven years of dreaming, wondering, and anticipating, the mission will drive into Marathon Valley on another “E ticket” back in time.

"We want to be in Marathon Valley during conjunction [mid-June to late June],” said Arvidson. The idea is to get into Marathon Valley with enough time to scout around. “If we can, we want to be doing long duration APXS measurements on a brush target during conjunction, right in the middle of where CRISM says we have the greatest smectite signature," he said.

The solar conjunction occurs about every 26 months when Mars and Earth are on opposite sides of the Sun. During the days surrounding such an alignment, the Sun can disrupt radio transmissions between the two planets. So the MER ops team will put Opportunity into 'autopilot,' as they always have during conjunctions, to work a ‘science light’ campaign, and keep communication to a minimum.

At the moment, as May peeks over the horizon, Opportunity is facing an embarrassment of scientific riches and mysteries. “There's a lot of exploration to be done right here under our feet and we haven't even gotten to Marathon Valley where we are expecting the mother lode to be,” said Callas. “We’re discovering that there are more rides at Disneyland than we thought.”