The Opportunity selfie: 5000 Sols in the making

In late February, NASA's Opportunity rover captured its first-ever selfie to commemorate 5,000 sols on Mars. It was a yet another incredible achievement for the 14-year-old spacecraft, which was designed to last just 90 sols. This is the story of how that selfie came to be.

The story begins back in September 2007 (1288 sols into the 90-sol mission), when the Opportunity rover used its microscopic imager to inspect the mirror inside the Mini-TES instrument. Data had suggested there might be something wrong with it, but by reaching the rover's robotic arm up in front and turning the microscope toward the top of the mast, 12 images proved that the mirror was behaving fine and the anomalous data was the result of Martian dust sticking to the mirror.

The MER Microscopic Imager was of course designed to look closely at rocks and soils from a 60mm standoff, and is fixed focus. Staring into the rover's own mast the images were very much out of focus, but they were enough to see the outline of the Mini-TES mirror inside its housing.

When I saw that image, one thing was very much in focus for me. Opportunity needed to take a proper self-portrait. The world 'selfie' didn't even exist yet, but I knew Opportunity needed to take one.

There was only one small problem: I lived 5,500 miles away in England. That didn't stop me from wondering what a full self-portrait from Opportunity would look like, or how it might it be taken. It would take a lot of images, and they would have to be stitched together. Using some animation software, and a model of the rover that I had made for Michael Howard's Midnight Mars Browser, I tried to simulate how it might work.

Fast-forward a few thousand sols, just over ten years, and now I live in Los Angeles, work at JPL and am an engineering camera payload uplink lead on both the Curiosity and Opportunity rovers. Taking images with that very same microscopic imager is now part of my day job. And with Sol 5000 fast approaching, the opportunity (I swear, pun not intended) to celebrate that incredible milestone with a self-portrait seemed a worthwhile thing to try and argue for.

First, I needed a rover planner to actually sequence the robotic arm to point the microscope the right way. Enter long-time MER rover planner Ashley Stroupe, who fell in love with the idea just as much as I did. Unbeknownst to me, she emailed the SOWG Chair (the scientist in charge for the day) for Sol 5000, Aileen Yingst, the night before Sol 5000 planning began. Aileen was already the strongest science team advocate of taking selfies with Curiosity, using its higher resolution, color, and most importantly, focusable, MAHLI microscope. 

"So, the engineering team want to try and take a self-portrait," Aileen told the virtually assembled science team at the regular 9 a.m. SOWG meeting. We couldn't see their faces, but we could hear the collective "What?" from the science team. Our tactical uplink lead for the day, Keri Bean, had managed to convince the project to take a sunrise image in the previous day's plan, Sol 4999, but this was more than just a picture. This was a whole complex sequence of arm activities. The Mission Manager for the day here at JPL immediately went and spoke to the Project Manager. The Project Manager came into our sequencing room and wanted to know what on Earth we were trying to do. So, we showed him that simulation and he understood. However, to be sure everybody was on board, he reached out to the science team management, and thankfully, deputy principle investigator Ray Arvidson understood what we were trying to do. "It's only Sol 5000 once," he said. "Let's do it."

At first the rest of the science team were reticent to forego science observations for something that some might perceive as a gimmick, but slowly they came around. In fact, by the end of that SOWG meeting, rather than prioritize the selfie data behind the science data for the day, they advocated prioritizing ahead of it, so that we would be pretty much guaranteed to see all the data even if our UHF relay pass thru Mars Odyssey were to underperform for some reason. "What if we took a picture of the rover's shadow…that would be a sort of selfie?" one science team member suggested. "Where's the Earth going to be in the sky, could we take a picture of home?" added another. (Fun fact – the Earth is just to the right of the rover's mast in the final selfie, but is obviously not visible.)

Aileen stepped in to stop things getting out of control.

"Right – everybody out of the pool."

Now all we had to do was figure out how to take it.

Ashley had already been hard at work figuring out how and where to point the MI to get the images we needed. It wasn't easy – the software tools the rover planners use are really, REALLY not designed for planning this kind of activity. We were holding bits of paper up to the screen and guesstimating our field of view to further guesstimate the overlap between each image. We planned 17 images in total, rows of 2, 2, 4, 5, and finally, 4 images.

I wrote a sequence that Ashley's robotic arm activities would fire off each time we wanted to take an image. I needed to use almost no data volume to avoid treading on our science priorities for the day. So I commanded the rover to shrink the images down from their native 1024 x 1024 to just 256 x 256 pixels; one-sixteenth as much data. We could get all 17 images for the 'price' of about 1 normal MI image. I knew they would be badly out of focus, so shrinking them shouldn't really cost any 'detail' in the pictures. We regularly shrink images from the hazard cameras and navigation cameras, but a search of both Spirit and Opportunity's past activities showed that we had never asked either  MI to shrink its images 'in flight'. There was no reason to suspect it wouldn't work, but it was just one more thing to worry about. The compression we use for the MI isn't deterministic – we don't know exactly how much data each image will use; we simply give it a quality level and it compresses them as much as necessary. I estimated something like 2 or 3 megabits of data in total. 

After the rest of the planning day was completed, Ashley's robotic arm activities had been handed over to John Wright, another veteran of MER, for review and delivery (rover planners always work in pairs on Opportunity). The approval to radiate commands to the rover for Sols 5000 and 5001 was given, and we all went home. More than a few of us spent the night tossing and turning. What if the arm had a problem while doing it? Would it forever be stuck waving hopelessly in midair? What if we miscalculated the pointing and the images wouldn't line up? What if it generated so much data we would bump all the science data out of the downlink queue and the science team wouldn't get their data?  

The images were due to make it back the following morning. It was one of JPLs regular days off – an 'RDO'. But we all came in to work anyway. I had barely slept, partially because my wife and I had welcomed our daughter into the world just 6 weeks before on Christmas day. But if I'm honest, mainly because I was just so nervous about our selfie.

Crowding round the workstation of former Cassini ACE, now MER systems engineer Michael Staab, we watched for the packets of data to come down in real time. The data started flowing. The ID for thumbnails, tiny preview images, started flowing. Then the actual MI images. Before we had time to grab our laptops and start looking for pictures, another rover planner and image pipeline engineer, Hallie Gengl, shouted across the cube-farm walls, "What are you all doing? I've got pictures here." They must have been small to downlink so fast from Mars Odyssey.

I had MAESTRO, a mission planning tool, already open on my laptop. I hit refresh and all 17 images were there. The thumbnail of the very first image told me everything I needed to know. It worked. There was the rover's camera mast – its face – looking back at us.

Before I had a chance to even look at the rest of them, Hallie had done a quick stitch of all 17 images. We've spent 14 years and 5,000 sols looking out of the rover's own eyes, but for the first time she was looking right back at us. From 227 million kilometers away.

It was all I could do to not burst into tears. Yes, it was fuzzy – but not as fuzzy as we thought it would be (and not as fuzzy as my simulation from 10 years before!). Yes, we didn't QUITE get the pointing right, we cut the top of the mast off just a little bit, but it actually worked.

Almost forgotten in the euphoria – we had also asked the left eye of the Navigation Cameras to take a picture of the microscope at the same time. The rover looking at itself, looking at us. Very meta, but beautifully poignant.

The following week, the science team didn't just like the selfie – they wanted to add more frames to it! My fellow ECAM PUL Nathan Williams was on shift. He used my same compression from the Sol 5000 efforts. Another two frames on top to complete the mast, another 5 images across the bottom to see the front wheels, and one more frame just bolted onto the side. Heather Justice and Paolo Bellutta worked some magic as rover planners that day to get the arm into some pretty funky orientations to get just about every bit of the rover it's possible to see with the MI captured.

Those last 8 frames made a total of 25 images. All told, it took the rover just under 30 minutes spread across Sols 5000 and 5006 to take the images. After all my concerns about taking up too much data, it ended up being much smaller than we thought – fuzzy images compress really well – less than one quarter of a megabit in total.

As luck would have it, the MER science team meeting was happening during the downlink of Sol 5006's extra few frames. I asked the principle investigator if he would indulge me a little, and let me present it to the team assembled at Caltech as scientific discussions wrapped up on the second day. I furiously stitched together my own best version of the selfie, threw it onto a PowerPoint slide and somewhat nervously showed the assembled team.

She's old, a little forgetful, somewhat arthritic and 4010 sols past warranty. But there's our robotic envoy, on the rim of Endurance crater. Still exploring. Still making discoveries. 

Here's to the next 5,000 sols.

Editor's note: Here's an additional version of the selfie, colorized with a realistic Mars palette: