Mars Exploration Rovers Update: Opportunity Emerges from Solar Conjunction to Wrap-Up Work on Matijevic Hill

As the Mars Exploration Rover (MER) team waited out solar conjunction, Opportunity spent most of April atop the western rim of Endeavour Crater, conducting a chemical analysis of an ancient waterborne vein on Matijevic Hill. It was by the book until the last week of the month when the robot field geologist suffered an electronic "hiccup" known as a warm re-boot, and went into auto mode, a kind of safe mode when something doesn't go right.

"Given the state of vehicle, being out of touch two and half weeks for solar conjunction, then coming back and finding it's in auto mode is no big surprise - this is nothing that freaked anybody out," assured Steve Squyres, MER principal investigator, of Cornell University.

Solar conjunction is a regular celestial event that occurs about once every 26 months, when Earth's and Mars' orbits put them on opposite sides of the Sun. Since the Sun's positioning between the two planets compromises communications of solar conjunction, planetary science teams suspend all new commands and institute a communications blackout during this period. This time around, the rover was effectively hunkered down for nearly three weeks.

True to her MER nature, Opportunity emerged unscathed, ready to attempt a bump and one more analysis of another part of the vein during the first days of May before heading south on a 2-kilometer (1.24 mile) journey along Endeavour's western rim to Solander Point, where the rover is to spend the coming Martian winter.

"Opportunity is back under master sequence control and in good health," reported John Callas, MER project manager, of the Jet Propulsion Laboratory (JPL), as the MER Update was being edited for posting. "She came through solar conjunction in fine style," added Bill Nelson, chief of rover engineering at JPL, home base for all the American Mars rovers.

For several months prior to solar conjunction, Opportunity had been having issues with its Flash memory system and the MER engineers suspect that another such issue is what caused the rover's computer to re-boot. Knowing there was a possibility of a glitch and re-boot during solar conjunction, they were prepared.

"When we designed our solar conjunction plans this time, we gave consideration to the fact that we may have a Flash memory problem that might result in a re-set and we made sure we were resilient against that kind of thing and that the rover was resilient," informed Callas. "Although we're always concerned whenever there's an anomalous behavior in the vehicle, we do think we know what probably caused this and right now, it's an annoyance rather than something we would lose sleep over."

Other than that little wrinkle, April 2013 turned out to be a rare, quiet month for the MER mission, a month where sleep for the human crew was actually gained rather than lost.

When April dawned at Endeavour Crater, Opportunity was already at the Big Nickel outcrop taking close-up pictures with her Microscopic Imager (MI) for a large mosaic image of the waterborne vein, named Esperance. By April 2nd, the rover had her chemical detecting Alpha Particle X-ray Spectrometer (APXS) down on a spot dubbed Esperance-3 and was settled in for solar conjunction.

The MER tactical operations team ceased all commanding uplinks on April 9th and the team scientists and engineers began turning their attentions to much-earned vacations or catching up on writing, research, or other work that has been stacking up. Just before the Sun effectively blocked the planets, Opportunity checked in one more time, downlinking her vitals on April 13th (Sol 3278) to let her teammates know she was healthy and working as commanded.

It wasn't until April 26th, when the JPL engineers uplinked the first post solar conjunction commands, that they discovered that Opportunity had gone into auto mode. From the looks of the data they received back, the re-boot probably happened on April 22nd, the rover's Sol 3286. Although the robot field geologist had been in auto mode for several Martian days, there was no reason to believe, no other indication that Opportunity was in any kind of trouble.

A less experienced team might have panicked, but there was no fear at the MER mission control. Beyond years of experience, the cause was immediately suspected, and going into auto mode was exactly what the rover is supposed to do after an unexpected re-boot like this. "We have various levels of fault protection and auto mode is the most benign," explained Nelson. "In auto mode, the rover typically wakes up, stays up for 90 minutes, honors communication windows, and then shuts down and goes into DeepSleep overnight."

The MER engineers are pretty confident that the Flash issue, which they think is basically the same issue they had years ago with Spirit, caused the re-boot. "We saw a lot of these Flash errors right before in the data we got back from
Opportunity, and then the next thing is this warm re-boot, which is like a restart on your computer," said Nelson. [A cold re-boot is like powering down you system and then starting it back up.] "When we had the Flash problem with Spirit, we also had a number of these warm re-boots," he noted.

What the engineers don't yet fully understand is exactly why Opportunity is experiencing these Flash events. "We think it may be corrupted sectors in the memory system, as it was with Spirit, but we don't really have an easy way to find out," said Nelson. "Making matters worse is the fact that this is an intermittent problem."

The team's software engineers managed to fix Spirit's Flash issue by reformatting the memory system, and if the issues are being caused by corrupted sectors in the memory system as they suspect, then in all likelihood can fix Opportunity's Flash issue the same way, just not right now. The glitches must become more frequent - or in other words the bad sectors have to degrade enough - before programmers can flag sectors as bad and successfully reformat the memory, as reported in previous MER Updates.

"We're going to continue investigating, but it looks like this happens when the rover tries to write to Flash," said Nelson. "Somehow we get a corrupted address that causes the rover to try and write to an invalid memory location, and that triggers the warm re-boot. But we don't know why we're getting this invalid address. All this stuff happens at the same time, so we presume it's causal."

As April came to an end, the MER mission ops team uplinked what they call a "real-time activate" to bring Opportunity back under master sequence control. In the same packet, they sent commands for the rover to make a tiny move at Esperance - what Squyres called "a bump with an intent to grind" - in order to get her Instrument Deployment Device (IDD) or robotic arm within reach of another, hopefully more accessible part of the boxwork vein target running through the Big Nickel outcrop on Matijevic Hill.

It didn't work. The commands didn't recover the vehicle as planned and Opportunity didn't move.

"Doing the post-mortem, it looks to me like we made a bonehead mistake," said Nelson. "We sent a command that killed the communication window that we were commanding into. Any time you reconfigure telecommunications, it terminates any window that's currently in process even if you configure it to the same configuration it's in. In this case, we changed background rates to its existing background rate, but it still killed our window."

That meant the commands sent were not received. "We followed what we thought was our standard recovery, but there were some subtle differences between this recovery and past recoveries that we didn't think through," Nelson said. "We're taking it very seriously, and there will be changes in procedures we will follow as a result of this. We have put together a Tiger team, an ad hoc investigation team that will include some of the outside developers who are no longer working fulltime on the mission, to get a little more horsepower on this situation." This team is going to start meeting in the next few days, he said.

While the Flash re-boots can result in lost science, the only science task Opportunity missed with the glitch in April was one Pancam measurement of the dust in the atmosphere, a Tau as the MER team calls it, "and we've gotten more since then," said Callas. "This time it had a very minimal impact. But we've lost days because of Deep Space Network (DSN) outages and lost relays and other things, so in that scheme of things, this was not much of a loss."

Other than the Flash problem and the few breaks and aches that have occurred in the last nine-plus years, Opportunity is "surprisingly healthy," summed up Nelson. "We don't have any other problems to speak of."

If all goes as expected, Opportunity will be bumping into May in hopes of getting a good grind into Esperance and one last analysis of its chemical composition. Veins that run through rocks are evidence of past water and so the team and their rover are putting in the extra effort to try and get deeper into part of the tiny, narrow, uneven surface of this target. The plan is for the rover to use her Rock Abrasion Tool (RAT) to grind off the top layers of accumulated dust to get a better look at the vein itself.

"There is one location on this boxwork structure in this fractured area at Big Nickel that is Esperance," said Ray Arvidson, MER deputy principal investigator, of Washington University St. Louis. "From where we are now, we can't do a RAT. But if we rotate the vehicle a little bit, we might be able to grind a part of the vein close to the Esperance-3 target we're on now. The idea is to bump to this new Esperance target, to offset a little bit, RAT it if we can, take some MI pictures and look at it with the APXS, and then get moving towards Solander Point as quickly as possible."

Esperance, however, is a difficult target to say the least. "This vein is just about as nasty a RAT target as you can imagine," said Squyres. "It's irregular, tiny, and has newberries surrounding it which complicate the task."

Beyond that, the truly excellent road trip that MER followers, as well as team members, have been anticipating for weeks now is looming large on the MER agenda. Opportunity has to get on the road soon if she is going to make it to Solander Point by August when the next Martian winter will begin to descend on Meridiani Planum.

"We're always trading the science we have right in front of us - which we know about, have sense of its value, and a sense of how hard it is to get - against completely unknown science somewhere down the road," said Squyres. "But we do want to get going pretty soon."

Once Opportunity does leave Matijevic Hill, the plan is for the rover to put her pedal to the metal and drive straight for Solander Point. "We calculated a rapid transit route through the winter strategy planning group, a straight-forward engineering route that would get us to Solander Point in time," Arvidson informed. "The issue is, if we leave in say mid-May roughly, there's just not a whole lot of time to do diversions to scientifically interesting locations. This route allows for a short stop at Nobby's Head so that the rover can take some stereo images of the northern side of hill for engineering characterization of slopes, just in case we need a possible bail-out location for winter, but other than that it's a path that will take Opportunity to Solander as quickly as possible. It's equivalent to what we're doing on Curiosity to get that rover to Mt. Sharp."

The command for Opportunity to "bump with an intent to grind" was being re-sent as the MER Update went to press. Whether the rover succeeds in grinding into on Esperance or not, chances are better than good that Opportunity will be departing Matijevic Hill and embarking on the journey south before the end of the second week of the month.

As dusty as the robot field geologist is these days, it needs to get to a good north-facing slope for the coming Martian winter if it's going to be able to survive and work through another harsh season. Solander Point has choice north-facing slopes where the rover can position her solar arrays toward the winter Sun. But if the robot field geologist is going to make it there in time, with at least a little margin to spare, it needs to get roving soon.

"Based on energy predictions we have in hand right now, we need to get the vehicle positioned at Solander Point sometime in early August," said Callas. "That date will change as a function of the energy predictions. If it gets worse, we will have to get there more quickly. If it gets better, then we can take more time. But we have only so many planning days between now and early August and the more time we spend here, the fewer days we have to drive, meaning the pace either would have to be increased or we work weekends to get it there. That's the trade space we're working in and those are the challenges we have."

The decision on the day of Opportunity's departure will be made soon, said Squyres. "Having some sols in our back pocket for discoveries along the road is a good thing, so we're not going to over-do it here. But what's down the road - we're not going to know until we get there. Right now, we've got a little time and we want to spend it wisely. We don't know how this is going to play out. We'll find out in next week or so."

It's been more than nine years since Spirit and Opportunity defied all odds and bounced onto Mars - still amazing after all these years - and a lot of life has happened since then. MER team members have pushed the limits of engineering and expanded the horizons of planetary science. They have won all kinds of awards, given birth to children who are growing up with rovers on Mars, suffered and survived car accidents, surgeries, the loss of pets - and the loss of a number of gifted teammates.

For many in the MER family, there will be a certain kind of unspoken finality in leaving Matijevic Hill.

The passing of Mars rover pioneer Jake Matijevic last August, just a couple of weeks after Curiosity landed, came so unexpectedly that there are many who have not yet quite processed the loss, and with Opportunity roving around the hill named in his honor, he has, in a way, been ever-present. In coming days, "Little Miss Perfect," as Jake called Opportunity, will be roving on. It's what she was born to do. Jake, as much as anyone, would want that. And yet - although no one who knew the shy, brilliant JPL engineer will ever forget him - the rover's departure from Matijevic Hill will close a chapter in the book of MER.

In Memoriam: Tom Wdowiak

As the MER Update was being written, more sad news arrived. Tom Wdowiak, an astrophysicist and Emeritus Associate Professor of physics at the University of Alabama Birmingham (UAB), who operated the Mössbauer spectrometers passed away on April 27th at St Vincent's Hospital in Birmingham. He suffered from renal failure and a heart attack, and did not recover.

One of the original MER Athena Science Team co-investigators, Wdowiak, the payload uplink lead and payload downlink lead of the German supplied Mössbauer spectrometers for the MER Science Operations Working Group, brought decades of experience to the table. He'd studied meteorites, thermal springs on land and thermal vents in the deep sea, even global mass extinction events caused by asteroids or comets striking the Earth with Mössbauer spectroscopy.

"I've been preparing for this mission since I was seven or eight years old," he said of MER in the days after the twin robot field geologists took the world by storm in their bounce-down landings.

As Wdowiak saw it, he "became a scientist" at the age of 7. He was so 'blinded by science' that by the time he was 8 years old, his friends started calling him Tommy Test Tube. Around the same age, he found Mars, stumbling across a stack of science books and magazines. For probably obvious reasons, he zeroed in on Air Trails, featuring paintings by Chesley Bonestell, the artist who created visions of other planets on canvas and inspired an entire generation of astronomers, artists, writers, and engineers.

One Bonestell painting in particular, of the Martian landscape, set Wdowiak on his life's path. "My journey began that moment in 1947, in the basement library of Woodrow Wilson School in Binghamton, New York!" he said in his MER bio. "It was the first time I was exposed to the concept of Mars as a world. Not as a planet. There's a difference. A planet, you look at. But a world is a real place that you can stand on, can walk around in," he explained later to Dale Short for an article in the University of Alabama Magazine.

Wdowiak received an AAS in Chemistry from Broome (NY) Technical Community College in 1959, and his BA in Astronomy from the University of South Florida in 1966. Five years later, he earned his PhD in Astronomy and Astrophysics from Case Western Reserve University. He went on to work at Teledyne Brown Engineering, NASA's Marshall Space Flight Center, University of Alabama Huntsville and Miles College, before joining the faculty at UAB.

Along the way, Wdowiak he became an expert in Mössbauer spectroscopy, named for the German science prodigy Rudolph Mössbauer whose discovery of the technique earned him the 1961 Nobel Prize for Physics. For Wdowiak, it was a research technique that allowed him to combine his interests in chemistry, geology, astronomy, biology, physics, and space exploration.
A Mössbauer spectrometer can measure with tremendous precision just how gamma rays - the invisible radiation that rocks and materials absorb and reflect - are changed by the iron atoms of the objects they strike. These measurements, in turn, allow scientists to determine the atomic and molecular structure of the substances.

The technique is highly valuable in determining the composition or origin of mysterious rocks, such as possible meteorites or long-buried fossils or rocks from another time on Mars. Wdowiak's talent for studying meteorites using the Mössbauer technique was one of the credentials that landed him a spot on the MER team.

During his career, Wdowiak published more than 100 papers on astrophysics, astrochemistry, astrobiology, planetary science, and instrumentation development, served on NASA review panels, and was a reviewer for numerous scientific journals. He resurrected Tommy Test Tube later in life to write a "Just for Kids" column for the Birmingham News/Post Herald. The popular column featured a full color page of science experiments, "all based on, and tested by, my childhood experiences," as he put it. "but now they're a bit more 'foolproof,'" he says, "thanks to a lifetime of accumulated experience."

"Tom was a really passionate, really enthusiastic, and a much-loved member of our team," said Squyres. "When you lose someone like him, it hits pretty hard ... life goes on, but it's tough."

An expanded summary of the MER Science results presented at the recent Lunar and Planetary Science Conference will be posted soon in a special edition of the MER Update.