Explaining the new black-and-white Mastcam and MARDI raw images

If you've been obsessively checking the Curiosity raw images websites for new pictures from Mars, you might have noticed something weird: a bunch of Mastcam images and a few from MARDI that are black-and-white instead of color, and which have a peculiar checkerboard pattern. Here's an example full-resolution image. Let's examine it up close:

Weird, isn't it? Why are they checkerboardy, and why are they black-and-white when Mastcam is a color camera?

Before I talk about the checkerboard, I just want to note that there's a little block of especially dark pixels -- that's likely a blemish on the detector.

Okay. There are several things going on here. The first has to do with how Mastcam achieves its color vision, which is different from how most space cameras do it. Most space cameras are black-and-white cameras, and you get color pictures by placing three color filters in front of the camera, taking one picture at a time, and combining the three into a single color photo.

Mastcam is more like the camera you might have in your pocket.  Its detector has color filters painted on it. But you need three different color filters to reconstruct color as the human eye would see it, so each pixel in Mastcam's detector is painted over with a red, or green, or blue filter, in a design originally described by Bryce Bayer. There are two green pixels for each red and each blue pixel, because the human eye is most responsive to color in that wavelength. The green pixels are arranged just as in a checkerboard; one row goes red, green, red, green, red, green... and the next row goes green, blue, green, blue, green, blue...

(That picture's from Wikipedia.) Now, the pixels sitting behind these filters don't know anything about color; they only know how many photons hit them. So the resulting data file is still technically a grayscale image, but it's one where every fourth pixel shows the number of photons received through a blue filter. The grayscale Mastcam photo at the top of this post is a photo that has not been processed to decode the color information that it contains. It's sort of neat because you can easily see the Bayer pattern. Mars is generally orange, so it's brightest in red wavelengths, darkest in blue wavelengths. It's easy, then, to pick out the blue pixels -- they're the dark ones arranged in a neat pattern.

To turn this image into a color picture, it must be "de-mosaiced." You take only the green pixels, and interpolate values into the places where the red and blue pixels were; you take the blue pixels, and interpolate values into the red and green areas; and same for red. There are lots of different, often computationally intense ways of demosaicing Bayer-filtered images to turn them into color.

Some of the guys at unmannedspaceflight.com have had a go at demosaicing these raw images. Here's two example results, one of them with the color left alone, the other processed to bring out details. There's something I particularly like about the second one -- it looks like a vintage postcard! But they're also a little bit weird looking, because there's still a checkerboard pattern of weird green and purple squares on top of the image. Why is that? (No, they haven't been run through Instagram filters.)

It has to do with how the images were posted to the Web. They came down from the rover in a sort of camera raw format, with actual pixel values, with lossless compression. But before they were posted to the Web, they were saved as JPEGs. The JPEG compression algorithm is a "lossy" one -- it changes the values of pixels in ways that help it to reduce the file size. It works on 16-by-16 blocks of pixels. When there are sharp boundaries between colors in JPEG-compressed images, you often see artifacts that have a square or blocky shape. And since this entire photo has every fourth pixel considerably darker than the adjacent ones, the JPEG compression resulted in edge artifacts on those dark pixels, which means they leaked into adjacent red and green pixels, and caused wonky effects during demosaicing.

Unless JPL decides to change the way they share these particular kinds of images (by, say, saving them in PNG or GIF format, which would be lossless), any color images produced from processing these JPEGged versions of the uncompressed Mastcam pictures are going to have these weird blocky color artifacts.

Even with those artifacts, though, they are inexpressibly cool. We have never, ever seen a landscape like this on Mars, at least not from the ground. And we're going to be driving closer to it, much closer, and even in to some of this canyon country. Let's get rolling!