Mastcam-Z team blog: Preparing for five-hour operations

It takes hundreds of scientists and engineers many years to design and build just one instrument for a Mars mission. Have you ever wondered what is going on behind the scenes during all that time? Your authors are members of the science team for the Mars 2020 rover mast-mounted Zoom cameras (known as Mastcam-Z), and this is the first in a series of Mastcam-Z team blog posts about the work that goes into a mission before it ever launches. In this post, we'll talk about the special challenges we expect for Mars 2020 operations, and how we're planning to overcome them.

Team meeting

The Mastcam-Z science team consists of a Principal Investigator (PI), a Deputy PI (DPI), 21 science Co-Investigators (Co-Is) from universities, companies, and government labs in the U.S. and six other countries, and several dozen undergraduate and graduate students and management, administrative, and technical support staffers who work closely with the PI, DPI, and/or individual Co-Is.   Our team includes scientists who have done extensive geological field studies on Earth. Others' careers have focused on virtual field work on Mars. In addition to their scientific experience, they know what works and what doesn't when it comes to operating rovers and cameras on Mars. Some science team members know how to write software, create stereo maps, operate the cameras, and organize and manage teams to get work done right and on time. Everyone brings their enthusiasm for Mars exploration and science as well as their curiosity, dedication, and creative ideas in science and problem solving. That's what makes this a great team.

The full Mastcam-Z team gathered most recently on a sunny couple of days in Pasadena, California last February to catch up with each other. The discussions were lively and encompassed a wide range of topics, from the practical to the philosophical. We discussed minute details of camera building and calibration targets, exciting software tools we will use in 2021, and how to plan a day of science observations on Mars with a rover that hasn't been built yet.

A science team meeting isn't all science, however. After work finished for the day, we enjoyed socializing and catching up with each other over dinner. Some of us have worked together for decades on various projects, so lasting friendships have naturally formed. Other team members are new to Mars rover missions, and it's great to share perspectives, hear new ideas to address old problems, and get to know each other.

Planning for Mastcam-Z Team Operations

The Mastcam-Z cameras will take single ("monoscopic") and dual-eye ("stereoscopic") pictures of both the Mars landscape and the rover itself in many color bands. They are based on the Mastcams on Curiosity, but this time we are adding zoom lenses. Zoom capability means that for every image we take, we can balance how much ground to cover in one image with how much we zoom in on the details. And we can do it for both the left and the right eyes, literally adding a whole new dimension (depth!) to our rover's color vision. Read more about the specifications of Mastcam-Z here.

Capability is one thing, but before we even approach landing on Mars we have to figure out what we are actually going to do once we land, how we are actually going to take our images, and how we will conduct the overall science mission. This whole area is called "mission operations," but when we speak we abbreviate that to "ops".

Some of the questions we ask ourselves as we begin to plan mission operations are:

• How are we going to control a rover that is millions of miles away, with a team of hundreds of scientists and technical support folks literally spread all over the world, across at least eight time zones?
• How do that many scientists and engineers look at the same data from Mars, at the same time?
• How do we develop tools to enable the engineers and scientists to communicate efficiently and make decisions together in real time?
• How do we give them the necessary training to use those tools?
• How will they have live, real-time discussions to come up with one single science plan for each sol (a day on Mars is called a "sol")?
• How do we make sure that the plan that satisfies all the tight constraints on the rover and doesn't pose a risk to our precious vehicle or instruments?

Once we answer these questions, we need to be ready with adequate computer power and disk space, functioning software tools, and step-by-step procedures long before we get to Mars. We need to get everyone trained and perform testing, testing, and more testing to ensure that all the tools are working right for each job. Just as important, we need to learn how to work together in spite of, and because of, our varying areas of expertise and what we each uniquely bring to the job.

Five-hour operations

Tactical operations planning for Mars 2020 will be a complicated and incredibly detailed process that ends with hundreds of commands being sent to the rover almost every single Earth day. This isn't different from previous rover missions. But the Mars 2020 team also faces a new challenge: we must accomplish all this in only five hours of work on each typical planning day. No rover mission has been run this way before. When Curiosity first landed, it took 16 hours to plan one sol. Planning has gotten more efficient over time, but it still typically takes 8 or 9 hours. Why the focus on five-hour planning days?

The main reason is that the Mars 2020 mission is trying to avoid the inefficiency imposed by "restricted sols." To explain: the Mars day is longer than the Earth day by 40 minutes. When the rover finishes work in its afternoon and downlinks data, it's at roughly the same time every Mars sol, but downlink creeps later and later every Earth day. If we were willing to work around the clock on Earth, we would simply shift our working hours by 40 minutes every day, and sometimes end up working at night and sleeping in the day. We call that working on Mars time. All Mars missions have worked on Mars time for at least three months right after landing, but it is very difficult on humans who have lives and families away from Mars; it's not sustainable for long periods of time.

When landed missions transition to "Earth time," we run into a situation several days a month when we have not yet received yestersol's downlink when we are planning the next sol of operations. We only have two-day-old data available to make decisions. That can mean the difference between being able to drive or not drive to a new target location, or can leave us without enough information to make other important scientific decisions. We refer to this as being in "restricted sols". In some cases, being in restricted sols can delay work on Mars by a sol. As those restricted sols accumulate, it delays our mission from reaching its goals by a significant amount.

If we are able to make plans quickly enough and are willing to accept a slightly wider range of working hours in the Earth day schedule, we can avoid restricted sols. For the Mars 2020 mission, the management team has decided that it's acceptable to start work as early as 6:00 am and end as late as 11:00 pm in the Pacific time zone. Within those limitations, it turns out that if we can get our work done in five hours, we can still catch the most up-to-date downlink every day when we start planning, and never experience a restricted sol.

Experiment Operations Plan

The February 2017 Mastcam-Z science team meeting took place almost exactly four years before landing on Mars. To get science planning and commanding done in 5 hours a day, we are working now, four years before landing, to develop several new technologies and software tools for the Mars 2020 mission in order to avoid restricted sols altogether and conduct our scientific mission with maximum efficiency.

At this stage of the mission operations preparations a main focus has been preparing our "Experiment Operations Plan" (EOP). The EOP is a document in which the Mastcam-Z team describes to the Mars 2020 project (our JPL colleagues and NASA managers) how we plan to operate the cameras in a way that will fit into Mars 2020's overall rover operations plan. We have to describe our team's plans for developing software and procedures to command the cameras and check (and triple check) that everything is done exactly right. We also have to list what computer and other IT support equipment we need to do our work as we build our operations facilities, and how that will change when we are operating the 2-Earth-year long (or more) mission on Mars.

We have to lay out a plan for all the jobs we need to do. We have to describe when and how we are going to staff and train scientists and engineers to fill all those jobs. Naturally, we also have to include our schedule and budget for carrying out all of the tasks that need to be completed in the six years or so it will take to design, test, and fly our cameras to Mars and complete the nominal mission.

Last but not least for our Mastcam-Z investigation, we have to describe in the EOP how we intend to communicate with and involve and engage the public (what we call Education and Public Outreach, or EPO). This is one of our favorite perks. Reaching out to and engaging with you all puts smiles on our faces and reminds us why we do all this hard work. If you'd like to know more about our plans and how you can get involved, check out our Mastcam-Z EPO page.

Mars 2020 Software Tools

We have some pretty cool operations software tools in the works as well. For example, the Jet Propulsion Laboratory is currently planning a tool named ASTTRO (Advanced Strategic and Tactical Targeting Tool for Remote Operations—yes, NASA loves acronyms!). ASTTRO is not just for visualization: users like science team members and rover planners (also known as rover drivers) will use it to create science target suggestions for discussion with others. Then "Payload Uplink Lead" (PUL) engineers can read the suggested targets from ASTTRO when they write detailed instrument command sequences for uplink to the rover.

The five-hour operational timeline creates stresses on instrument teams to work fast without sacrificing accuracy. In those five hours, we have to define, write, check, test, and uplink our commands. You don't usually get a second chance to take any of the pictures we take of Mars, so we want it to be right the first time. Every time. To that end, for Mastcam-Z, we wanted our own very specialized and highly capable camera visualization and commanding tool to interface with JPL's ASTTRO program. We are already beyond planning and deep into the development and testing phase of this tool, which is called Viewpoint.

Viewpoint will be able to read in and display all the images we have taken on Mars so far from previous rovers, as well as HiRiSE images of the landing site and traverse path taken from Mars Reconnaissance Orbiter. Combining surface and orbital images, creating mosaics on the fly, and being able to virtually move the rover and oneself around in the environment to gain different perspectives, Viewpoint is akin to using Google Mars on your home computer. There's an "overhead view" that uses Mars data from the MRO orbiter that enables you to see the Mars rover in the landscape from above, providing excellent context to where we are right now, and where we are going next. There's also the "street view" where you can look out on the Mars landscape from the perspective of the rover. Several of us have already started to use Viewpoint on the Curiosity and Opportunity rovers. This software enables us to see the sand and rocks the Curiosity rover is standing on, for example, and pan all the way to the horizon where you can see the Gale Crater walls, and Mt. Sharp in the distance.

In the Viewpoint tool we can quickly and accurately add new frames (image locations) and build future mosaics of the landscape. Once we've laid out mosaics, we can move them around and adjust command parameters such as image compression. We can also visualize the rover hardware as seen from Mastcam-Z to check whether hardware will obstruct our desired view from our chosen camera pointing. For instance, we can check whether the rover arm is out front and could be partially or wholly obstructing scientific targets that we are trying to image near the rover. Then we can re-point the images to avoid the arm. We will also be able to visualize shadows cast by the rover at any time of day. The scientists usually prefer that images be taken either in full Sun or full shadow, because consistent lighting aids scientific analysis of the image. We will adjust as necessary to get the best science images we can.

Viewpoint for Mars 2020 has heritage from (that is, it's based on) a version that was developed for the Panoramic Camera (Pancam) on the Spirit and Opportunity rover missions, and we are currently testing a beta version on the Curiosity rover to prepare for using it on Mars 2020. It's always great when you can use heritage software, because it means you have the basics down already, have worked out some of the bugs, and you can refine it with every new mission and version of the tool. That usually makes for a better tool and also saves time and money. The other big advantage for us as we prepare for operating the cameras on the Mars 2020 rover is that we can use it during early testing and operations simulation activities that we'll be participating in over the next four years.

Hardware and more

Operations and software were just two of many issues we covered in the last team meeting. There were also updates on hardware development and landing site selection, which will be the subject of future Mastcam-Z team posts. Stay tuned!