Future High-Resolution Imaging of Mars: Super-Res to the Rescue?

MEPAG recently posted a report from the Next Orbiter Science Analysis Group that describes the need for a new orbiter to launch in 2022 or 2024 to support future Mars robotic and human exploration. Some of the top priorities are to understand active processes and to locate and characterize candidate landing sites and surface exploration regions with HiRISE-class (30-centimeter-per-pixel) or better (10- to 15-centimeter-per-pixel) imaging. They recommend using solar-electric propulsion as well as advanced telecommunications (Ka-band and high-power RF amplifiers, and a laser communication experiment), which together enable a much higher data rate to Earth than past orbiters. Solar-electric propulsion also increases the potential mass of the science instrument payload that can be launched to Mars by a particular rocket. The desire to achieve a spatial resolution better than HiRISE is driven by science objectives such as understanding recurring slope lineae (RSL) and also to support landing site characterization, surface activity planning, and even diagnosis of engineering anomalies (such as how the Beagle-2 lander failed).

A factor of two or three improvement over HiRISE image resolution doesn’t sound very difficult, but it is. Increasing a small camera’s resolution by a factor of two or three is not difficult, but HiRISE -- the largest telescope ever sent to another planet -- is already pushing against several limits. Large telescopes are difficult to accommodate, including the mass, volume, and pointing stability. Resolution is inversely proportional to the diameter of the telescope aperture. HiRISE has a half-meter aperture, so doubling the optical resolution requires a primary mirror about a meter in diameter, which, in turn, enlarges the other mirrors and the structure. So the mass and volume must increase substantially compared to HiRISE. Alternatively, new technologies can reduce the mass of the telescope (like James Webb Space Telescope), but that increases cost and introduces risk, and NASA is very risk-averse.

The challenges don’t stop with simply flying a larger telescope. If the pixel scale is smaller or if more time-delay integration lines are required, then the pointing stability requirement for the entire spacecraft becomes more challenging. This stability challenge may be worsened by flying the large solar arrays needed for solar-electric propulsion.

High-resolution imaging of Mars from orbit is challenging. The atmosphere of Mars causes two problems. First, the orbit cannot be lower than about 250 kilometers (or even 300 kilometers when the air is especially dusty and warm) because of drag on the spacecraft. We have substantial coverage of the Moon at half a meter per pixel from the Lunar Reconnaissance Orbiter Camera (LROC), which is much smaller than the HiRISE camera. The intrinsic resolving power of LROC is about 10 times coarser than HiRISE, but in its prime mission Lunar Reconnaissance Orbiter traveled at an altitude of just 50 kilometers, compared to about 300 for HiRISE. The thin atmosphere of Mars is also surprisingly bright due to suspended dust. As a result, more than half of the brightness of Mars orbital images is from atmospheric scattering; less than half of the light reaching the camera is from direct reflection from the surface that reveals small-scale features. This atmospheric haze can be subtracted from the images to increase contrast, but it still contributes photon noise proportional to the square root of the signal. The solution is to acquire images with a very high signal-to-noise ratio (SNR). The top-of-atmosphere SNR needs to be about 150:1 (a typical value for HiRISE) to insure that the SNR of surface features is at least 50:1 in the majority of images. (Sometimes the dust is so thick that the surface SNR is much worse, approaching zero.)

Another challenge is that the mass of Mars dictates that low circular orbiters must travel at a high velocity of about 3.2 kilometers per second. To image at a scale of 0.3 meters per pixel, this means the line time is only (0.3m)/(3200m/s) = 0.000094 second or 94 microseconds. With HiRISE optics, using that line time as the integration time produces an image with SNR of only 10, rather than 150. The HiRISE solution is use of time-delay integration, imaging each surface patch 128 times and summing the signal on the CCD detector. What if the imaging scale is reduced to 15 centimeters per pixel? Then the integration time is just 47 microseconds and ~256 time-delay integration lines are needed (assuming a 1-meter primary mirror) for adequate SNR, and the columns are half as wide, so the stability must be four times better. The spacecraft pointing stability needs to be very precise to make sure a surface feature is re-imaged in the same TDI column, not jittering or wandering over multiple columns, smearing the image. Mars Reconnaissance Orbiter went to considerable effort (and cost) to achieve high stability for HiRISE. The cost of a factor of two improvement in image resolution is not confined to the cost of the camera; it drives the cost of the entire spacecraft.

Super-Resolution Restoration to the Rescue?

A recent press release from the University College of London presented their Super-Resolution Restoration (SRR) technique, including comparison of regions near Spirit rover (Figure A).

This UCL press release made some startling proclamations:

"Exciting pictures of the Beagle-2 lander, the ancient lakebeds discovered by NASA’s Curiosity rover, NASA’s MER-A rover tracks and Home Plate’s rocks have been released by the UCL researchers who stacked and matched images taken from orbit, to reveal objects at a resolution up to five times greater than previously achieved."

"Co-author Professor Jan-Peter Muller from the UCL Mullard Space Science Laboratory, said: "We now have the equivalent of drone-eye vision anywhere on the surface of Mars where there are enough clear repeat pictures. It allows us to see objects in much sharper focus from orbit than ever before and the picture quality is comparable to that obtained from landers."

"By stacking and matching pictures of the same area taken from different angles, Super-Resolution Restoration (SRR) allows objects as small as 5cm (about 2 inches) to be seen from … stacks of between four and eight 25cm images of the Martian surface taken using the NASA HiRISE camera…"

"In the future, we will be able to recreate rover-scale images anywhere on the surface of Mars and other planets from repeat image stacks" said Mr Yu Tao, Research Associate at UCL and lead author of the paper."

These statements sound too good to be true. A factor of 5 resolution enhancement implies 25 times the information content, yet they only processed 4 to 8 images. In addition, the HiRISE dataset is far from ideal for super-resolution restoration. Here’s what is needed for ideal super-resolution restoration:

HiRISE sometimes satisfies #1 above, but the typical SNR of the surface is ~50:1. As for #2, most planetary mission images are aliased (few pixels across the point-spread function) because data rate is limiting. In other words, it is too expensive in data volume to "oversample" the point-spread function as is done by Earth-based telescopic imaging (although they call it "adequately sampling" the point-spread function). The HiRISE image point-spread function is about 2 pixels wide at half max, which is useful but not ideal for super-resolution restoration. For #3, Mars Reconnaissance Orbiter's orbit enables crudely identical resolution and illumination conditions via its sun-synchronous orbit, although sun angles vary considerably with season. The viewing angles vary from about -30° to +30° cross-track, which is the only way to image a surface location multiple times per Mars year. The resolution varies by up to 15% with viewing angle. To co-register the images, the geometry difference with viewing angle can be overcome with sufficient processing, as done at UCL. Given the changing viewing angles plus pointing jitter, the sub-pixel offsets from image to image (#4) are random, which is far from ideal. This means that the resolution improvement varies from place to place in the super-resolution image. For #5, the surface of Mars does change over time, especially in places we favor for multiple images, to monitor the changes.

Clearly, HiRISE images are not ideal for super-resolution restoration. However, the Pancam on Opportunity rover is able to acquire near-ideal super-resolution datasets because they can sit still on the surface and move the camera pointing by fine increments, so conditions #3, 4, and 5 are well satisfied, and the SNR is high. This effort is described in a paper by Planetary Society president Jim Bell et al.; their Figure 25 is copied below. They did not provide a quantitative estimate of resolution improvement, but based on artificially degrading images by various factors (see Figure C), it appears that the improvement in resolution is about 2x. The pixel scale is 8x smaller, but that is not the same as resolution (capability to measure the size of image objects).

Figure A, from the UCL paper, shows obvious sharpening in comparison of the super-resolution image to a single HiRISE image (labeled "original"). However, the "before" image does not fully represent the resolution potential of single HiRISE images. The "original" image (PSP_001513_1655) was acquired during moderately high dust opacity, so the SNR is poorer than other images, and the image has been map-projected at the original scale, and this resampling of the pixels reduces the resolution a bit. A better choice of "before" images is shown below, in Figure D. There is still clearly improvement in the super-resolution product, both in resolution and SNR, but comparison to Figure C suggests that the resolution improvement is closer to a factor of 2 than 5. Tau and Muller (2016) estimate super-resolution enhancement of "2–5x" based on rock counts on the "original" and super-resolution images, but the smaller rocks "not detected" in the single image are in fact detectable as dark spots, so the issue with that approach is how the automated rock-counting algorithm decides whether or not to measure a feature. A better test would be to apply super-resolution processing to a set of images over features of known size, then measure the sizes of those features in the original images and super-resolution product. The size measurement error is the resolution in each case.

How useful is super-resolution restoration for future Mars imaging?

HiRISE has captured enough (more than 4) repeat images useful for super-resolution restoration of only about a dozen areas on Mars. Areas with larger numbers of images are targeted repeatedly because of surface changes over time, so a super-resolution product would defeat the purpose of understanding these changes. For large landing sites such as for the ExoMars rover or InSight lander, it takes years to acquire even single HiRISE coverage over the whole "3-sigma" landing error ellipses, and twice as long to acquire full stereo coverage. It would not be wise to sacrifice full coverage of landing ellipses for the sake of super-resolution processing on a just a small part of the ellipse. However, the next Mars orbiter may have a much larger data rate and the landing site ellipses are shrinking due to advances in entry, descent, and landing techniques, which makes super-resolution restoration more attractive. In conclusion, for the highest-priority sites, super-resolution restoration may provide an economical alternative to flying a more capable camera and spacecraft, in order to achieve a modest improvement in resolution (probably no better than a factor of two).

Drones (tiny helicopters) are not made obsolete by super-resolution restoration—they can acquire images with much higher native resolution, and can themselves acquire better sets of images for super-resolution restoration than can a fast-moving orbiter. In fact, a drone might accompany the 2020 rover. However, this drone is very limited in range, and can only travel about half a kilometer per day before stopping to recharge, so we need orbital imaging to first find the best places to land.