Who is the photographer behind Mars rover photos? Answer from Mark Lemmon

Editor's note: I received an email from Miguel Santesmases, a Spanish fine arts student, asking: "Who is operating [the rover] cameras, who shoots the trigger, who directs them to a precise object or mountain, how is all this done? I have been searching the web but it is as if the cameras shoot by themselves, the dream of the old days of photography, the picture made by itself, but I know it has to be someone there, or a team of photographers." I forwarded his question to two Martian camera operators, Mark Lemmon and Jim Bell. Here is Mark's response. --ESL

It would be nice if the pictures took themselves. But it takes a village, it seems, to get a picture taken on Mars.

Technically, you could say that the Payload Uplink Lead who writes the command sequence is the photographer. In that sense, Elaina McCartney at Malin Space Science Systems (formerly at Cornell) has taken more pictures on Mars than anyone else. This person (PUL, or on Curiosity, PUL-2--I'll explain the number below) is as close as you get to someone who "pulls the trigger". The PUL writes the commands, validates them, makes sure they are safe and will achieve the objective, and approves them for uplink and execution. The command sequence is what the rover sees and executes, after format conversion (essentially, compiling).

For some pictures, that adequately captures things. Drive direction mosaics [sets of images taken to plan the rover's next drive] and other routine sequences are executed the same way every time. The PUL takes the desire to have a product and turns it into the commands. There is no aesthetic consideration involved, just a repeatable operational consideration.

Other pictures require a bigger perspective. Someone on the rover science team looks at the routine Navcam survey and says, "Hey, that's cool, we need a color picture." They provide a specific intent for what is in the picture, whether it should be color or use other filters, what resolution and level of compression is acceptable. All of this is mostly done with internet-friendly planning tools. I can designate a target and have everyone see it, and create an activity plan that says what kind of picture I want. Usually, this is simple: there is a target, and the camera will be aimed right at it.

Some images require even more consideration. Aiming straight at a single target works, but some pictures are actually composed in detail. Mostly, this is to help capture multiple interesting things, or an extended target. Mosaics, especially, get this kind of attention. A science team member may draw a box on a Navcam image, or specify azimuth and elevation coordinates, or just specify a center and size.

Lighting is commonly treated very intentionally. Some images are taken whenever they can be. More often, illumination and shadowing is a concern, even to the point of requiring the rover to shut down during the normal operations day in order to be awake in the late evening or early morning.

Finally, some images do get a very detailed attention to composition, with aesthetic considerations and scientific or operational considerations all playing a role. These usually have illumination and shadowing specified, as well as very specific composition. A recent color mosaic across Endeavor crater comes to mind, as does a section of the Phoenix site panorama that showed the Robotic Arm poised above the lander ready for sample delivery. Any sunset or astronomical imaging gets at least this level of detail. The Phobos and Deimos transits were planned over weeks, with dozens of people involved (in that case there was a balance between getting the desired images and a requirement to avoid damaging ChemCam in the process, as it is very sensitive to direct Sun light).

At the other extreme, the Phoenix SSI is as close as I've been to pictures that take themselves. With the Phoenix operating system, subroutines were stored on board. A single command line from the ground could then take a large multi-filter mosaic; usually it is one command per picture (or stereo pair).

The process of getting Mastcam images is fascinating and fun. As an example, say we've just driven with MSL, and gotten an end of drive Navcam mosaic (courtesy of yestersol's Navcam PUL). Scientists come into the planning process with a dozen ideas of images to get, among other things. The day starts as chaos. Over hours, that chaos becomes a plan, with a down-selected set of images and other activities in a scripted plan that captures all of the intent of the picture-taking.

Over a few more hours, that plan becomes lines of code that will execute on the rover the next day. Several people may participate in designing a mosaic (over telecon, internet chat, email, etc.). At this time, for Curiosity's Mastcam (and MAHLI and MARDI), a first-shift PUL-1 from the science team will be deeply involved in the discussions, and have a good understanding of the product that is desired and the resources (bits, time) it will take to get there. A second-shift PUL-2 from the camera engineering team will come in and eventually write the sequence. The two PULs work together in the middle, and sometimes for a while later, to make sure that the intent is followed to the best of the rover's ability.

A late afternoon M-100 mosaic of Mt. Sharp on sol 45 really tested this--we had driven since the last targeting Navcam to include all of Mt. Sharp. So I worked closely with the PUL-2 throughout the sol (back on Mars time) to design the mosaic to actually get all of this big thing that refused to show up in our planning tools. Every frame was tweaked a few times until we were satisfied. Then, we all go home, and the next day it is chaos-to-code in hours all over again.

By the way, for a single snap shot you might have the Geology Science Theme Group conceive and design it en masse; the PUL-1 plan it; the entire (on staff) Science Operations Working Group discuss it and include it in the daily plan, the PUL-2 actually write it, and the engineering uplink team review and approve it before the Ace hits the button to radiate it, with the sol's command, bundle to the rover. That's a group the size of a small village.

For my part, when a person puts in effort at composition, especially when it makes the result aesthetic as well as functional, that person is the photographer. But of course, that is what I mostly do, so it is a bit self-serving. I prefer to write command sequences relatively infrequently.