Dawn Journal: Ion thrusting (or not)

Dear Dawnomalies,

Farther from Earth and from the sun than it has ever been, Dawn is on course and on schedule for its March 2015 arrival at Ceres, an enigmatic world of rock and ice. To slip gracefully into orbit around the dwarf planet, the spacecraft has been using its uniquely capable ion propulsion system to reshape its heliocentric orbit so that it matches Ceres’ orbit. Since departing the giant protoplanet Vesta in Sep. 2012, the stalwart ship has accomplished 99.46 percent of the planned ion thrusting.

What matters most for this daring mission is its ambitious exploration of two uncharted worlds (previews  of the Ceres plan were presented from December 2013 to August 2014), but this month and next, we will consider that 0.54 percent of the thrusting Dawn did not accomplish. We begin by seeing what happened on the spacecraft and in mission control. In November we will describe the implications for the approach phase of the mission. (To skip now to some highlights of the new approach schedule, click on the word “click.”)

The story begins with radiation, which fills space. Earth’s magnetic field deflects much of it, and the atmosphere absorbs much of the rest, but there is no such protection for interplanetary spacecraft. Some particles were energized as recently as a few days earlier on the sun or uncounted millennia ago at a supernova far away in the Milky Way galaxy. Regardless of when and where it started, one particle’s cosmic journey ended on Sep. 11 at 2:27 a.m. PDT inside Earth’s robotic ambassador to the main asteroid belt. The particle penetrated one of the spacecraft panels and struck an electrical component in a unit that controls the ion propulsion system.

At the time the burst of radiation arrived, Dawn was thrusting as usual, emitting a blue-green beam of high velocity xenon ions from engine #1. Ten times as efficient as conventional chemical propulsion, ion propulsion truly enables this unique mission to orbit two extraterrestrial destinations. With its remarkably gentle thrust, it uses xenon propellant so frugally that it takes more than three and a half days to expend just one pound (0.45 kilograms), providing acceleration with patience.

Dawn’s electronics were designed to be resistant to radiation. On this occasion, however, the particle managed to deposit its energy in such a way that it disrupted the behavior of a circuit. The control unit used that circuit to move valves in the elaborate system that transports xenon from the main tank at a pressure of 500 psi (34 times atmospheric pressure) to the ion engine, where it is regulated to around two millionths of a psi (ten million times lower than atmospheric pressure), yielding the parsimonious expenditure of propellant. The controller continued monitoring the xenon flow (along with myriad other parameters needed for the operation of the ion engine), but the valves were unable to move in response to its instructions. Thrusting continued normally for more than an hour as the xenon pressure in the engine decreased very gradually. (Everything with ion propulsion is gradual!) When it reached the minimum acceptable value, the controller executed an orderly termination of thrust and reported its status to the main spacecraft computer.

When the computer was informed that thrust had stopped, it invoked one of Dawn’s safe modes. It halted other activities, reconfigured some of the subsystems and rotated to point the main antenna to Earth.

The events to that point were virtually identical to a radiation strike that occurred more than three years earlier. Subsequent events, however, unfolded differently.

In normal circumstances, the mission control team would be able to guide the spacecraft back to normal operations in a matter of hours, as they did in 2011. Indeed, the longest part of the entire process then was simply the time between when Dawn turned to Earth and when the next scheduled tracking session with NASA’s worldwide Deep Space Network (DSN) began. Most of the time, Dawn operates on its own using instructions stored in its computer by mission controllers. The DSN is scheduled to communicate with it only at certain times.

Dawn performs a carefully choreographed 2.5-year pas de trois from Vesta to Ceres. Celestial navigators had long known that the trajectory was particularly sensitive to glitches that interfere with ion thrusting during part of 2014. To ensure a prompt response to any interruptions in thrust, therefore, the Dawn project collaborated with the DSN to devise a new method of checking in on the spacecraft more frequently (but for short periods) to verify its health. This strategy helped them detect the condition soon after it occurred.

When an antenna at the DSN complex near Madrid, Spain, received the explorer’s radio signal that morning, it was apparent that Dawn was neither in exactly the configuration to be expected if it were thrusting nor if it had entered one of its safe modes. Although they did not establish until later in the day what was happening, it turns out that not one but two anomalies occurred on the distant spacecraft, likely both triggered by particles in the radiation burst. Dawn encountered difficulty controlling its attitude with its usual exquisite precision. (Engineers use “attitude” to refer to the orientation of the craft in the zero-gravity conditions of spaceflight. In this case, the spacecraft’s orientation was not controlled with its usual precision, but the spacecraft’s outlook was as positive and its demeanor as pleasant as ever.) Instead of maintaining a tight lock of its main antenna on faraway Earth, it was drifting very slightly. The rate was 10 times slower than the hour hand on a clock, but that was enough to affect the interplanetary communication. Ultimately one of the onboard systems designed to monitor the overall health and performance of all subsystems detected the attitude discrepancy and called for another, deeper safe mode.

In this safe mode, Dawn further reconfigured some of the subsystems and used a different part of the attitude control system to aim at the solar system’s most salient landmark: the sun. It switched to one of its auxiliary antennas and transmitted a wide radio beam.

Meanwhile, the operations team began working with the DSN and other missions to arrange for more time to communicate with Dawn than had previously been scheduled. Projects often collaborate this way, making adjustments for each other in the spirit of shared interest in exploring the solar system with the limited number of DSN stations. Later in the day on Thursday, when an antenna near Goldstone, Calif., was made available to point at Dawn, it was stable in safe mode.

The team decided to aim for resuming thrusting on Monday, Sep. 15. They had already formulated a detailed four-week sequence of commands to transmit to the spacecraft then, so this would avoid the significant complexity of changing the timing, a process that in itself can be time-consuming. This plan would limit the duration of the missed thrust during this sensitive portion of the long flight from Vesta to Ceres. Time was precious.

While it was in safe mode, there were several major challenges in investigating why the spacecraft had not been able to point accurately. The weak radio signal from the auxiliary antenna allowed it to send only a trickle of data. Readers who have heard tales of life late in the 20th century can only imagine what it must have been like for our ancestors with their primitive connections to the Internet. Now imagine the Dawn team trying to diagnose a very subtle drift in attitude that had occurred on a spacecraft 3.2 AU (almost 300 million miles, or 480 million kilometers) from Earth with a connection about one thousand times slower than a dial-up modem from 20 years ago. In addition, radio signals (which all regular readers know travel at the universal limit of the speed of light) took 53 minutes to make the round trip. Therefore, every instruction transmitted from JPL required a long wait for a response. Combined with the intermittent DSN schedule, these conditions greatly limited the pace at which operations could proceed.

To improve the efficiency of the recovery, the DSN agreed to use its newest antenna, known as Deep Space Station 35 (DSS-35), near Canberra, Australia. DSS-35 was not quite ready yet for full-time operational use, and the DSN postponed some of the planned work on it to give Dawn some very valuable extra communications opportunities. It’s impressive how all elements of NASA work together to make each project successful.

Engineers hypothesized that the reconfigurations upon entering safe mode might have rectified the anomaly that prevented the spacecraft from maintaining its characteristic stability. While some people continued the previously planned work of finalizing preparations for Ceres, most of the rest of the operations team split into two shifts. That way, they could progress more quickly through the many steps necessary to command the spacecraft out of safe mode to point the main antenna to Earth again so they could download the large volume of detailed data it had stored on what had occurred. By the time they were ready late on Friday night, however, there was a clear indication that the spacecraft was not ready. Telemetry revealed that the part of the attitude control software that was not used when pointing at the sun in safe mode — but that would be engaged when pointing elsewhere — was still not operating correctly.

Experts at JPL, along with a colleague at Orbital Sciences Corporation in Dulles, VA, scrutinized what telemetry they could receive, performed tests with the spacecraft simulator, and conducted other investigations. The team devised possible explanations, and one by one they tested and eliminated them. Their intensive efforts were powered not only by their skill and their collective experience on Dawn and other missions but also by plenty of pizza and fancy cupcakes. (The cupcakes were delivered in a box lovingly decorated with a big heart, ostensibly by the young daughter of the team member who provided them, but this writer suspects it might have been the team member himself. Regardless, embedded in the action, your correspondent established that the cupcakes were not only a yummy dessert after a pizza lunch but also that they made a terrific dinner. What a versatile and delectable comestible!)

Despite having all the expertise and creativity that could be brought to bear, by Saturday afternoon nothing they had tried had proven effective, including restarting the part of the software that seemed to be implicated in the pointing misbehavior. Confronting such an unyielding situation was not typical for such an experienced flight team. Whenever Dawn had entered one of its safe modes in the preceding seven years of flight, they had usually established the cause within a very few hours and knew precisely how to return to normal operations quickly. This time was different.

The team had still more ideas for systematically trying to fix the uncooperative pointing, but with the clock ticking, the mission director/chief engineer, with a conviction that can only come from cupcakes, decided to pursue a more dramatic course. It would put the spacecraft into an even deeper safe mode, and hence would guarantee a longer time to restore it to its normal operational configuration, but it also seemed a more likely solution. It thus appeared to offer the best possibility of being ready to start thrusting on schedule on Monday, avoiding the difficulty of modifying the four-week sequence of commands and minimizing the period of lost thrust. The idea sounds simple: reboot the main computer.

Rebooting the computer on a ship in deep space is a little bigger deal than rebooting your laptop. Indeed, the last time controllers commanded Dawn to restart its computer was in April 2011, when they installed a new version of software. Such a procedure is very delicate and is not undertaken lightly, given that the computer controls all of the robot’s functions in the unforgiving depths of space. Nevertheless, the team made all the preparations that afternoon and evening, and the computer rebooted as commanded two minutes after midnight.

Engineers immediately set about the intricate tasks of verifying that the probe correctly reloaded all of its complex software and was still healthy. It took another 12 hours of reconfiguring the spacecraft and watching the driblet of data before they could confirm around noon on Sunday that the attitude control software was back to its usual excellent performance. Whatever had afflicted it since the radiation burst was now cured. After a brief pause for the tired team members on shift in Dawn mission control to shout things like “Yes!” “Hurray!” and “Time for more cupcakes!” they continued with the complex commanding to point the main antenna to Earth, read out the diagnostic logs, and return each subsystem to its intended state. By Monday afternoon, they had confirmed that hundreds upon hundreds of measurements from the spacecraft were exactly what they needed to be. Dawn was ready to resume ion thrusting, heading for an exciting, extended exploration of the first dwarf planet discovered.

Throughout the contingency operations, even as some people on the team delved into diagnosing and recovering the spacecraft and others continued preparing for Ceres, still others investigated how the few days of unplanned coasting would affect the trajectory. For a mission using ion propulsion, thrusting at any time is affected by thrusting at all other times, in both the past and the future. The new thrust profiles (specifically, both the throttle level and the direction to point the ion engine every second) for the remainder of the cruise phase and the approach phase (concluding with entering the first observation orbit, known as RC3) would have to compensate for the coasting that occurred when thrusting had been scheduled. The flight plans are very complicated, and developing them requires experts who apply very sophisticated software and a touch of artistry. As soon as the interruption in thrust was detected on Thursday, the team began formulating new designs. Initially most of the work assumed thrusting would start on Monday. After the first few attempts to correct the attitude anomaly were unsuccessful, however, they began looking more carefully into later dates. Thanks to the tremendous flexibility of ion propulsion, there was never doubt about ultimately getting into orbit around Ceres, but the thrust profiles and the nature and timeline of the approach phase could change quite a bit.

Once controllers observed that the reboot had resolved the problem, they put the finishing touches on the Monday plan. The team combined the new thrust profile with the pre-existing four-week set of commands already scheduled to be radioed to the spacecraft during a DSN session on Monday. They had already made another change as well. When the radiation burst struck the probe, it had been using ion engine #1, ion engine controller #1, and power unit #1. Although they were confident that simply turning the controller off and then on again would clear the glitch, just as it had in 2011 (and as detailed analysis of the electrical circuitry had indicated), they had decided a few days earlier that there likely would not be time to verify it, so prudence dictated that near-term thrusting not rely on it. Therefore, following the same strategy used three years earlier, the new thrust profile was based on controller #2, which meant it needed to use ion engine #2 and power unit #2. (For those of you keeping score, engine #3 can work with either controller and either power unit, but the standard combination so far has been to use the #1 devices with engine #3.) Each engine, controller, and power unit has been used extensively in the mission, and the expedition now could be completed with only one of each component if need be.

By the time Dawn was once again perched atop its blue-green pillar of xenon ions on Monday, it had missed about 95 hours of thrusting. That has surprising and interesting consequences for the approach to Ceres early next year, and it provides a fascinating illustration of the creativity of trajectory designers and the powerful capability of ion propulsion. Given how long this log is already, however, we will present the details of the new approach phase next month and explain then how it differs from what we described last December. For those readers whose 2015 social calendars are already filling up, however, we summarize here some of the highlights.

Throughout this year, the flight team has made incremental improvements in the thrust plan, and gradually the Ceres arrival date has shifted earlier by several weeks from what had been anticipated a year ago. Today Dawn is on course for easing into Ceres’ gravitational embrace on March 6. The principal effect of the missed thrust is to make the initial orbit larger, so the spaceship will need more time to gently adjust its orbit to RC3 at 8,400 miles (13,500 kilometers). It will reach that altitude on about April 22 which, as it turns out, differs by less than a week from the schedule last year.

During the approach phase, the spacecraft will interrupt thrusting occasionally to take pictures of Ceres against the background stars, principally to aid in navigating the ship to the uncharted shore ahead. Because arrival has advanced from what we presented 10 months ago, the schedule for imaging has advanced as well. The first “optical navigation” photos will be taken on about Jan. 13. (As we will see next month, Dawn will glimpse Ceres once even sooner than that, but not for navigation purposes.) The onboard camera, designed for mapping Vesta and Ceres from orbit, will show a fuzzy orb about 25 pixels across. Although the pictures will not yet display details quite as fine as those already discerned by Hubble Space Telescope, the different perspective will be intriguing and may contain surprises. The pictures from the second approach imaging session on Jan. 26 will be slightly better than Hubble’s, and when the third set is acquired on Feb. 4, they should be about twice as good as what we have today. By the time of the second “rotation characterization” on about Feb. 20 (nearly a month earlier than was planned last year), the pictures will be seven times better than Hubble’s.

While the primary purpose of the approach photos is to help guide Dawn to its orbital destination, the images (and visible and infrared spectra collected simultaneously) will serve other purposes. They will provide some early characterizations of the alien world so engineers and scientists can finalize sensor parameters to be used for the many RC3 observations. They will also be used to search for moons. And the pictures surely will thrill everyone along for the ride (including you, loyal reader), as a mysterious fuzzy patch of light, observed from afar for more than two centuries and once called a planet, then an asteroid and now a dwarf planet, finally comes into sharper focus. Wonderfully exciting though they will be, the views will tantalize us, whetting our appetites for more. They will draw us onward with their promises of still more discoveries ahead, as this bold adventure into the unknown begins to reveal the treasures we have so long sought.

Dawn is 1.2 million miles (1.9 million kilometers) from Ceres. It is also 3.65 AU (339 million miles, or 546 million kilometers) from Earth, or 1,475 times as far as the moon and 3.67 times as far as the sun today. Radio signals, traveling at the universal limit of the speed of light, take one hour and one minute to make the round trip.

P.S. While Dawn thrusts tirelessly, your correspondent is taking the evening off for Halloween. No longer able to fit in his costume from last year (and that has nothing to do with how many cupcakes he has consumed), this year he is expanding his disguise. Expressing the playful spirit of the holiday, he will be made up as a combination of one part baryonic matter and four parts nonbaryonic cold dark matter. It’s time for fun!