Curiosity's balky drill: The problem and solutions

Curiosity is heavier, more capable, and more expensive than its rover predecessors in order that it can acquire samples of Martian materials and deliver them to two sophisticated laboratory instruments inside the rover belly. Since December 1, 2016, Curiosity has been unable to perform that function because of a serious problem with one of the drill's motors. Engineers have been hard at work on the problem ever since, and there is now realistic hope that the drill can be returned to function, but they'll have to use it in a way it wasn't designed for. As I did before for Curiosity's wheels, I'll explain how the drill works, the nature of the problem, the work being done on Earth to understand it, and the path forward for Curiosity.

First things first: How drilling is designed to work

Let's orient ourselves to the drill. In order to perform a drilling operation, the engineers select a target and then place the arm and turret against it. The turret contacts the drill target through two stabilizers that project in front of the drill. These aren't motorized, but the arm does have sensors that can detect how much force is pressing against the stabilizers. The rover "preloads" the arm against the target by pressing down on the stabilizers with the motors in the arm. Once preloaded, the arm holds absolutely still until drilling is complete. Drilling happens with motion of mechanisms within the drill. The video below shows a successful drilling operation on Mars.

And here's a diagram of the major mechanisms within the drill, which I'll explain in a bit more detail below.

Drilling depends upon three mechanisms in the drill. One, the drill spindle mechanism, rotates the drill. Another, the drill percussion mechanism, pounds on an anvil rod behind the drill 30 times a second in order to pulverize the rock below the drill bit. Finally, the drill feed mechanism moves the whole shebang (drill bit, spindle, and percussion components) along a set of linear rails to advance the drill into the ground. There is one more mechanism in the drill, the drill chuck mechanism, which would allow Curiosity to release its drill bit assembly and exchange it for a new one in one of the bit boxes mounted to the front of the rover. That would only be necessary if the existing drill bit wore out (it's currently in like-new condition) or if the drill bit were to get totally stuck in the ground (more on that in a bit).

The first drill problem: Battle short and sol 911 percussion anomaly

The problem that cropped up on December 1, 2016 is not the first issue with Curiosity's drill. Here's a section from my forthcoming book describing an issue that had cropped up even before launch:

A potentially serious problem was discovered during Earth testing of a testbed version of the drill mechanism in 2011. A broken bushing caused a short circuit in the test drill that could have fried the rover's motor controller if engineers had not acted swiftly. The consequences of such an event happening on Mars would be dire. It was too late to make any changes to the flight drill. Engineers in Florida opened the belly pan of the rover to install a "battle short" that would route half of the excess current to ground if such a short circuit developed in flight.

On sol 911, sensors detected current flowing through the battle short as Curiosity was using drill percussion to transfer sample from the drill to CHIMRA, halting the operation. There is no way to know if the cause is the same as the problem discovered on Earth, but the effect is similar. The shorts have recurred since sol 911, but are intermittent and extremely brief.

...[The mission has] changed the way they operate the drill: originally, they began drilling with a medium percussion level and adjusted based on the penetration rate, but they now begin with very light percussion and only increase the rate as needed. Engineers have also developed a new rotary-only drilling technique, made possible by the softness of the rocks within Gale crater, but rotary-only drilling has not yet been used on Mars because of a different drill anomaly.

The worse drill problem: Sol 1536 drill feed anomaly

On sol 1536, the mission attempted to drill at a site named Precipice. It would be the very first use of rotary-only drilling. They had selected the target, braced against the rock, and commanded the drill to feed forward -- and it didn't work. The arm remained braced against the rock, but the drill bit didn't move toward it. The drill feed mechanism had stalled. Curiosity has not drilled since.

The rest of this post is based upon a September 1 interview with Curiosity deputy project manager Steven Lee. Steve has worn many other engineering-related hats on Curiosity in the past, including Guidance, Navigation and Control Systems Manager and Strategic Uplink Lead. I'm very grateful for his time.

The question was: why had it stalled? It's not easy to figure out. The mechanisms on Curiosity's arm are "a miracle of integration and miniaturization," Lee told me. But the dense packaging allows for only a small number of sensors to give engineers information about what had happened. On sol 1536, all they knew is that current was flowing through the motor windings to generate torque, but that the torque was producing no motion; the rover correctly detected this, stopped operation, and declared a fault, ending operations with the drill and awaiting further instruction from Earth. (The rover is smart about arm-related faults -- it allows other non-arm activities like camera imaging and weather data collection to proceed as commanded even if the arm has faulted.)

The team stood down from drilling activity and tried to figure out what could produce the observed behavior. Among lots of ideas, they fairly quickly zeroed in on a likely cause: a problem with the brake in the drill feed mechanism. When the drill feed is not being used, an internal brake holds the feed firmly in position; this prevents other energetic activities (such as percussing the drill) from forcing the feed backwards. The brake consists of two plates that are pressed firmly against each other with a set of springs. To advance the drill feed, the rover energizes a solenoid within the drill feed mechanism. Energizing the solenoid pulls one plate (the "movable brake") away from the other (the "fixed brake"), freeing the feed to move as the motor winds it out with a worm drive. Unfortunately, there is no sensor within the brake to tell the rover whether the brake is actually engaged or disengaged.

The drill feed and the proper motion of its brake is such an important component that there is quite a lot of redundancy built into the motor. Engineer Louise Jandura explains in a 2010 article:

The windings of the motor are not fully redundant but again these elements are created by multiple physical wires terminated at multiple pins so a degraded torque capability would be available in the event of a problem. Each Arm actuator has a power-off brake that is mechanically engaged when non-powered to lock the motor rotor, preventing rotation. A brake solenoid is energized to release the motor. The brakes have redundant coils, each capable of releasing the brake, energized by separate brake drivers and separately cabled. All actuators with brakes on the Rover are configured with redundant solenoids.

The obvious thing to do, then, is to try the other solenoid and see if that released the brake. They tried; the behavior was identical. That strongly suggested a single cause for the drill feed stall: something -- such as a displaced component or a piece of foreign debris -- is interfering with the motion of the movable brake, preventing it from fully disengaging when commanded. There doesn’t appear to be anything wrong with the solenoids, but energizing either solenoid with the usual voltage didn’t result in brake release, and the motor stalled, and the feed didn't advance. This was a very serious problem. If the drill feed does not advance the drill, there is no drilling and no rock sampling. All Curiosity can do to deliver samples without that is scoop. And the rover was, after nearly five years of driving, just approaching a completely new and different type of rock.

December 2016-March 2017: Feeling out the problem

A number of obvious tests presented themselves, and the team began to sort through which ones would be best to try on Mars and in what order. Could they energize the solenoids with more voltage? Use both at the same time? Alternate? Command them multiple times? Try it all with the drill held in different orientations?

Work proceeded very slowly. Recognizing that returning the drill to operation would be a project taking months at least, the mission left Precipice behind and continued driving southward across the Bagnold dunefield toward Vera Rubin Ridge, interspersing drill diagnostic tests with routine mission science -- routine, that is, but for the lack of drilling and sampling.

Why was it so slow, given how important drilling is to the Curiosity mission? Changing the way the drill feed brake is operated requires the use of low-level commands that had never been used on Mars, because they hadn't been needed yet. All first-time activities with the rover have to be tested on Earth first, with the testbed rover. The team also has to take extreme care to protect the irreplaceable asset on Mars. They had to think each command through carefully -- would commanding the drill in a new way cause some other problem? Might it impart forces on other parts of the turret? Might it cause heating that's not being modeled properly, and overheat something? Throughout the four months after the anomaly, the team carefully and steadily built up a toolbox of low-level commands usable on Mars for testing.

The work resulted in slow progress on improving feed reliability. As testing proceeded, they gained confidence. They were able to extend the drill feed all the way out and wind it all the way back in. They were preparing to try feeding forward and backward again at the slower rate needed to actually drill into rock when they rolled up to a site named Ogunquit Beach.

Ogunquit Beach was a sand dune site where the rover scooped up sand and prepared to deliver samples to the laboratory instruments. To prepare a sample, the rover spins the turret so that the scooped sample rests on top of a fine-holed sieve, and vibrates the turret for about 20 minutes, rocking it back and forth, to encourage the finer-grained sample material to pass through the sieve into a chamber where it will be portioned for delivery to the instruments. The rover uses a few more rocking motions and a bit more vibration to portion out a sample and drop it into the SAM instrument. It’s all routine activity for the rover, and prior to the work at Ogunquit Beach, vibration activities had been conducted several times since the drill feed anomaly first appeared.

This time, however, after preparing and delivering the first Ogunquit Beach sample to SAM on sol 1651, the engineers found on March 29 that the drill feed was behaving differently -- and much worse -- than before. Once again, they could no longer advance the drill feed all the way out. That was obviously not good. They halted all sampling activity, not even dropping the Ogunquit Beach sample into the other lab instrument, CheMin, and considered what to do next.

March-September 2017: Exploring new ways of drilling

"For us, Ogunquit was a pretty big event," Lee says. "It started shifting our thinking from returning to drilling normally, to 'Gee, we need to have another option.'" Lee asked the team to continue working on returning to normal operations while also starting to think about how they might drill, acquire, portion, and deliver samples without a functioning drill feed. Initially, the idea seemed preposterous. But JPL engineers are nothing if not creative. As April gave way to May, "it didn't seem as ridiculous as we thought at first," Lee says.

Feed-Extended Drilling (FED)

In order for drilling to work at all, the feed has to be extended fully, past the stabilizer prongs. Otherwise, the drill cannot penetrate into the ground. So the first item of business was to get the drill feed to work just well enough, just long enough to feed the drill out past the prongs. "The problem was really thorny," Lee said. After Ogunquit, the brake was particularly stuck; it would stall after just a couple of motor revolutions. May turned to June and then July. In July, Mars passed behind the Sun as seen from Earth -- a geometry called conjunction -- which required a stand-down of Curiosity operations.

Shortly before conjunction, the engineering team had a particularly successful day, getting the feed to move about 60 millimeters, roughly half the full extension distance. Two actions helped the feed extend: cycling the brake many times, and pointing the drill in the air during the activity. By aiming the drill skyward, Mars' gravity may have imparted just the tiny extra bit of force needed to pull the moveable brake away from the fixed brake. Curiosity then sat with the drill pointed up for the whole three weeks of conjunction. After conjunction, they commanded the same activity, and the feed quickly spooled out, without sticking, all the way to its full extension of 110 millimeters.

A clearer look at the fully extended drill:

Extending the feed enables drilling again, but that doesn't mean drilling will actually work. While the team was working on getting the feed extended, they were also working with the testbed rover on understanding how drilling would work without a functioning feed. It's straightforward to use arm motion instead of feed motion to move the drill bit downward. But the arm wouldn't be braced against the rock with the stabilizers -- only the tip of the drill bit would be in contact with the target. Would arm motion on top of the drill actually result in drilling a hole, or would the drill bit skitter and bounce uselessly across the surface of the rock? If drilling worked, would the hole be straight enough that the bit wouldn't get stuck?

The new drilling regime is called feed-extended drilling (or FED, among initialism-obsessed engineers). Happily, Earth testing of feed-extended drilling into three representative rock types, limestone, siltstone and sandstone, suggests that skittering doesn’t preclude drilling a hole; the drill can penetrate pretty straight into the rock without the benefit of stabilizer prongs.

The shape of the hole is a slightly bigger concern. The arm is capable of moving in a smooth way, compensating intelligently for the rotation of components as it presses into a rock target, keeping the drill moving in a nearly straight line. But as anybody who has ever used hand tools knows, holes or cuts that look perfectly straight can have just slight curves that cause blades or bits to bind. With feed-extended drilling, Curiosity can drill a hole that looks perfectly straight, but the bit still binds a little when Curiosity pulls on the arm to extract the bit.

I want to be careful here to emphasize that we're not talking about the bit getting trapped in the rock; it's just a little harder to extract the bit with feed-extended drilling than with ordinary drilling. Harder to extract means that there are tensional forces being exerted on the arm that are higher than the rover on Mars has dealt with before. The forces observed in the testbed so far are well within Curiosity's design tolerances, but any time there is a major change to forces acting on the rover, the engineers take very special care to make sure those aren't causing any unintended consequences to other systems on the rover. They can test on Earth, but of course the rover on Mars is in a different gravity regime and is a different machine, so aside from testing, the engineers just need to sit and have a really good think about what could go wrong. The worst thing would be to cause a new problem while attempting to fix an existing problem.

It does look like feed-extended drilling is going to work. Curiosity will be able to drill again, using arm motion instead of feed motion. But the drill feed has a second function that's going to be more difficult to replace.

Feed-Extended Sample Transfer (FEST)

The way Curiosity drilling worked until sol 1536, the drill feed would push the drill bit into a rock. Most of the drill bit is enclosed in a sleeve. Channels curling up the drill bit auger powdered target material upward, inside the sleeve, into a sample chamber above the bit. Once drilling is complete, the feed retracts. Retracting the feed aligns the outlet of the sample chamber with an inlet funnel for Curiosity's sample handling and portioning apparatus, named CHIMRA. (To learn more about CHIMRA, read this guest post by engineer Dan Limonadi.) A few motions of the turret move the powder out of the sample chamber, into the funnel and into CHIMRA for sieving, portioning, and delivery. If the feed can't retract, it can't deliver the powder to CHIMRA, the powder can't be sieved and portioned. Even with feed-extended drilling, it's not obvious how it can be delivered to the instruments.

The drill feed is not completely dead. It does respond to commands, sometimes, balkily, and unreliably. It currently performs well but could worsen as a result of dynamic activities (like CHIMRA vibration). So if worse comes to worst, the team could use the feed only to align the sample chamber outlet port with the CHIMRA inlet port and deliver sample that way. Nobody wants to do this, because the feed is unreliable enough that any motion could potentially be the last. Now that the feed is out, there is a strong desire not to move it again (or at least, to move it very little just for testing, always keeping the drill bit fed beyond the stabilizer prongs).

So how else could the rover deliver samples to SAM and CheMin? In feed-extended sample transfer (or FEST), engineers are now exploring delivering sample directly from the bit. The rover would position the drill bit directly over a sample inlet, and rotate the drill backwards. The drill would auger material out of the sample chamber, out through the sleeve, and the stuff would fall directly in to the sample inlet. It's actually pretty straightforward; the drill can be positioned very precisely, and the drill auger allows a reasonably controlled delivery method.

There are two main problems with reverse-auger sample delivery, both of them having to do with bypassing CHIMRA. One: the sample material will not have been sieved beforehand. The material will not have a uniform, very fine grain size. The team is actually not terribly concerned about this issue, because drill tests on Earth using a variety of rock types show that the drill reliably produces powder with very small grain sizes. As long as Curiosity continues to drill into similar rocks, the powder should be okay for delivery to both instruments. If Curiosity drills into a very different-looking rock, it can drop some sample onto its observation tray for scientists to check out the grain size and other properties before approving delivery to the laboratory instruments. If the drilled sample turns out to have some coarse-grained material in it, both lab instrument inlets have 1-millimeter screens over them to prevent large grains from falling into the instruments, so they will be safe from the largest grains.

A bigger problem with reverse-auger sample delivery is that it's difficult to know how much material is being delivered. The team can experiment with drops onto the observation tray to characterize how much sample gets dropped with how much augering, but it's going to be imprecise and there will be no way to know if an especially small or large sample gets dropped. This is particularly difficult for SAM; some of SAM's scientific experiments are quite sensitive to the amount of material that is delivered. But other experiments aren't so sensitive and can proceed even if the team doesn't quite know how much sample is being dropped.

The engineering team has explored other methods of sample delivery, including dropping all the drilled sample on the ground and then attempting to scoop it up. If they can get the sample into the scoop, that puts it into CHIMRA for ordinary delivery. Unfortunately, the speed of the scoop works against this. Anyone who has ever swept a floor knows that if you move a dustpan slowly, you’ll just push the swept pile forward. It takes a quick motion to scoot the lip of the pan underneath the pile. Trying to scoop dropped sample mostly just pushes the sample around without lifting any of it into the scoop. They've also looked at dropping it on sand, scooping a mixture of drilled material and sand, running an experiment, and then scooping unsullied sand and running it as a blank. Unfortunately, it would likely be too hard to pull the signal of the drilled sample out of the noise of the contaminating sand for this to be worth the effort. For now, reverse-auger sample delivery seems to be the best path forward.

The Path Ahead

Curiosity is now just about to climb up onto Vera Rubin Ridge and a juicy new type of rock. Hopefully, Earth testing will continue to produce results that can eventually lead to the first attempts at feed-extended drilling on Mars. I'm not sure when that's going to happen. According to Lee, “It’s very difficult to predict exactly when we can get back to drilling with FED/FEST. We first need to complete more testbed testing beyond proof-of-concepts to flesh out the approach and characterize performance.” In the meantime, Curiosity does have a suite of remote sensing and contact science instruments that enable it to do survey science.

It's easy to feel frustrated about the pause in drilling and the loss of opportunities to sample so many rocks that have passed under Curiosity's wheels. But it's all part of the adventure of doing something no one has done before, with a machine that's already survived far longer than warranted. It always impresses me to talk with spacecraft engineers. I won't say they're never frustrated -- I'm sure they more than anyone else wish everything were working perfectly -- but they never give up. Every problem they encounter stimulates new creativity, and they'll keep pushing their machine to do its old tasks in new ways as long as the machine's still talking to Earth. The Curiosity team is a long, long way from being out of new tricks yet.