The SkyBox satellites are the first to use area array rather than pushbroom sensors for survey work, but they certainly aren't the first to use area array sensors. I think the first satellites to do that were the KH-11 surveillance satellites, versions of which are still the principle US optical spysats in use today. The first KH-11s sported area array sensors of about the same resolution as a standard definition TV. The most recent KH-11s probably have a focal plane similar to this tiled 18,000 x 18,000, 10 micron focal plane (shown below, that circle is a foot in diameter).
Optical spysats have two missions; call them surveillance and survey. When you already know where the thing is, that's surveillance. Response time matters, but throughput is usually not a big deal. When you don't know where your thing is, or you don't even know what it is yet, you are doing survey. Throughput is king in survey work, and if response time matters, you have a problem. Coast Guard aerial search and rescue, for example, has this problem. You can read about the difficulties of search at sea in this NY Times article on rescue of a fisherman last July.
General Schwarzkopf said after the first Gulf War that spysats (he must have been referring to the earlier KH-11s) could not provide useful, timely imagery. He was comparing single pictures of targets after a hit to the target camera footage of his planes, which gave him continuous video snippets of the target before, during, and after a hit. These videos were very popular at press conferences and with his upper management.
Satellites are excellent for getting access to denied airspace -- there is no other way to take pictures inside China and Russia. But in Iraq, Afghanistan, and Pakistan they are completely outclassed by airplanes and now drones with long-range optics (like the MB-110 reconnaissance pod which I still haven't written up). In a 20 year battle against irrelevancy, I suspect that getting near-real-time imagery, especially video, from orbit has been a major NRO focus. I'm sure the Block IV KH-11 launches in 2005, 2011, and recently in August 2013 can all do real-time downlinks of their imagery through the SDS satellite constellation. However, the second part of real-time is getting a satellite into position to take the picture quickly. The three KH-11s in orbit often cannot get to a surprise target in less than 30 minutes, and cannot provide continuous video coverage. Guaranteeing coverage within 30 minutes would require dozens of satellites. Continuous coverage, if done with satellites 300 km up, would require around 1000. The KH-11 series is expensive (they refer to them as "battleships") and the US will not be launching a big constellation of these.
The Next Generation Electro-Optical program, which started in 2008 or so, is probably looking at getting the cost of the satellites down into the sub-$500m range, while still using 2+ meter telescopes, so that a dozen or two can be launched over a decade within a budget that NRO can actually sell to Congress. My guess is they won't launch one of these until 2018. In the meantime, SkyBox Imaging and ExactEarth, who are both launching constellations of small imaging sats, will be trying to sell much lower-resolution images that can be had more quickly. These civilian operators have 50-60 cm apertures and higher orbits, and so can't deliver the resolution that NRO and NGA customers are used to, and they can't or don't use the SDS or TDRS constellations to relay data in real time. (SkyBox can do video, but then downlinks it 10 to 90 minutes later when they overfly one of their ground stations.)
The second spysat mission is survey: looking for a needle in a haystack. From 1972 to 1986 we had this in the form of the KH-9 Hexagon, which shot the entire Soviet Union every 2 to 4 months at 1 to 2 foot resolution. The intel community at the time could not search or inspect all that imagery, but the survey imagery was great once they'd found something surprising. Surprise, a new site for making nuclear weapons! Survey answers the question: What did it look like during construction? Or, How many other things like this are there? Nowadays, Big Data and computer vision have got some handle on finding needles in haystacks, but we no longer have the KH-9 or anything like it to supply the survey imagery to search. We still use the U-2 for aerial survey imagery, but we haven't flown that into denied airspace (e.g. Russia and China) for many decades.
From 1999 to 2005 Boeing ran the Future Imagery Architecture program,which was intended to make a spy satellite that could do radar, survey, and surveillance. The program took too long and ran way over budget, and was eventually salvaged by cancelling the optical portion and having the team design a synthetic aperture radar satellite, which did get launched. (Apparently this was the successor to the Lacrosse radar satellite.)
As I wrote, SkyBox does survey with a low-resolution area array. They would need about 16,000 orbits to cover the entire surface of the earth, which is 2.7 years with one satellite. I'm sure they can optimize this down a bit by steering left/right when over the ocean. But this is 70 cm GSD imagery.
Two of the telescopes designed for FIA were donated to NASA in 2012, and the few details that have emerged tell us about late 1990s spy satellites. From 300 km up, they could deliver 7 cm imagery, and had a (circular) field of view of about 50,000 pixels. This could have been used with a 48,000 x 16,000 pixel tiled focal plane array. Using the simple expedient of shooting frames along the line of the ground track, the ground swath would have been 3.2 km wide, and could have surveyed the entire Earth in about 2.7 years (the same number is a coincidence -- spysats fly at half the altitude and this one had twice my presumed field of view for SkyBox).
However, to keep up with the ground track pixel velocity, the sensors would have to read out at over 6 frames per second. That's almost 5 gigapixels per second. I don't believe area array sensors that big can yet read out that fast with low enough noise. (The recent Aptina AR1411 reads out at 1.4 gigapixels per second, but it's much smaller, so the column lines have far less capacitance.)
The large number is not a result of the specifics of the telescope or sensor design -- it's fundamental to high resolution orbital survey. It's just the rate at which the satellite flies over ground pixels. Getting 5 billion tiny analog charge packets to A/D converters every second is hard. Once there, getting 20 gigabits/second of digital data to the ground is even harder (I don't think it's been done yet either). I'll defer that discussion to a later post.
Pushbroom sensors are more practical to arrange.
The satellite simply stares straight down at the ground. Attitude corrections are very slow.
It's easy to get lots of A/D converters per sensor, you simply add multiple taps to the readout line.
It's easy to tile lots of sensors across the focal plane. You stagger two rows of sensors, so that ground points that fall between the active areas of the first row are imaged by the second row, like this focal plane from ESA Sentinel-2. Once stitched, the resulting imagery has no seams.
Tiled area array sensors are more difficult, but have the advantage of being able to shoot video, as well as a few long exposures on the night side of the Earth.
The image must be held steady while the field of view slides along the ground. Although this can be done by rotating the whole satellite, survey work is going to require rapidly stepping the stabilized field forward along the optical path, several times a second. Fast cycling requires a lightweight optical element, usually the secondary mirror, to have a fast and super precise tip/tilt mechanism to track out the motion. Cycling this element back into position between shots can add vibration to the satellite.
While the secondary mirror is moving the image back into position, the pixel photodiodes must not accumulate charge that affects the values read out. This typically means that either the cycling time can't be used for readout, or (as in the VisionMap A3) the sensor is an interline CCD with two capacitors per pixel, one of which is shielded. With this choice comes a bunch of minor but annoying problems.
In one line time, charge is transferred from the pixels all the way across the array to the readout. The bit lines can be long and capacitive and add noise.
Take another look at the first pic in this blog post, and note the seams between the active arrays. These are annoying. It's possible to take them out with clever combinations of sparse arrays and stepping patterns.
Lenses generally resolve a circular field of view, and pushbroom sensors take a rectangular stripe down the middle. It's possible to put an area array sensor in the leftover upper or lower crescent around a pushbroom sensor. This gives a smaller area sensor, but in the context of a 50,000 pixel diameter focal plane, a "smaller" area sensor might be 10,000 pixels on a side, with 50 times the pixel count of an HD video sensor. This allows for a 10:1 "digital zoom" for context with no loss of display resolution.
If I were building a government spysat today, I'd want it to do survey work, and I'd make surveillance the secondary mission. Airplanes and drones are better for most surveillance work. I'd want to shoot the whole Earth each year, which can be done with three satellites at 300 km altitude. I'd use a staggered pushbroom array as the primary sensor and a smaller area array for surveillance.
The step-stare approach that SkyBox is using makes sense when a big, fast area array sensor covering the whole field of view can be had at low risk. Sensors are developing quickly, so this envelope is growing over time, but it's still an order of magnitude away from what large-aperture spysats can do.
Maybe I'm wrong about that order of magnitude. In 2010 Canon announced a 205 mm square CMOS sensor that supposedly reads out 60 frames per second. Here it is pictured next to a full-frame 35mm DSLR sensor -- it's slightly bigger than the tiled array at the top of this post. Canon did not announce the resolution, but they did say the sensor had 100 times the sensitivity of a DSLR, which suggests a pixel size around 35 microns. That's too big for a spysat focal plane, unless it's specifically for use at night.
No subsequent announcement was made suggesting a purpose for this sensor. Canon claims it was a technology demonstration, and I believe that (they would not have been allowed to show a production part for a spysat to the press). Who were they demonstrating that technology to? Is this the focal plane for a Japanese spysat?
[Update: I'd like to thank Phil Pressel, John Gregor, and Gordon Petrie for their corrections to this post. The changes required have been so extensive I have not marked them.]
[Update 2: Phil Pressel just released his book Meeting the Challenge: The Hexagon KH-9 Reconnaissance Satellite (get it at AIAA or Amazon). I've ordered it and will post a review after I get it.]
From 1957 to 1965, a high tech startup called Itek made the world's most sophisticated satellite reconnaissance cameras for a single customer -- the CIA. The company has a fascinating subsequent history, as they ended up building cameras for Apollo and Viking. Eventually they ended up building the DB-110 reconnaissance pod, which I'll do a blog post on some day.
Merton Davies at RAND apparently originated the idea of using a spinning camera mounted on a spin-stabilized satellite to take panoramic shots with a narrow-angle lens. Amrom Katz passed the concept to Walter Levinson at Boston University Physical Research Laboratory (BUPRL). Levinson refined the idea to that of using an oscillating lens, for use in the HYAC-1 panoramic camera in Air Force high altitude reconnaissance balloons. Itek, just a few weeks after incorporation in late 1957, bought BUPRL, which was developing the HYAC-1 panoramic camera to take pictures from high altitude balloons.
Soon after, the CIA contacted Itek to discuss the camera requirements for the first spy satellites. All of these initial satellites came to use the rotating panoramic camera. I think this is the KH-4A or KH-4B Corona.
Itek also built versions of these cameras for use in the U-2 and SR-71 aircraft, in which they were called the OBC (Optical Bar Camera, named for the appearance of the field of regard on the ground). These were first used in the 1960s and are still in use today. Here is an Itek Optical Bar Camera that goes in the nose of an U-2:
Take a look at the big, wide blue bar under the airplane. That's what a single frame of the camera shot. It's huge. I've heard that this "bar" looking frame was why they called it the optical bar camera. However, the NRO's Hexagon documentation refers to the rotating optical assembly (what in many cameras is called the "Optical Tube Assembly") as the optical bar.
After a string of successful programs, requirements racheted up and tensions grew between NRO and CIA over the next program, Fulcrum. Early on, Itek was contracting with both the NRO and the CIA on competing projects. Itek pulled out of the CIA's project, and some combination of the NRO and the CIA took all their (government-owned) work and gave the job to Perkin-Elmer. When the dust settled the project was renamed Hexagon. Perkin-Elmer built the KH-9 Optical Bar Camera with their own design rather than Itek's, as they didn't think the Itek design would work. Here is a look into the aperture of the KH-9 Optical Bar Camera.
The Itek OBCs in the U-2, SR-71, and Apollo spacecraft all had a rotating structure called the roller cage, which I suspect was fixed to the rotating camera. The Perkin-Elmer design in the KH-9 deleted the roller cage and the rotating fold mirror inside it, and instead had a servo controlled twisting platen.
Here is a comparison of various optical bar cameras built up to 1971 (the launch of the first KH-9).
Take a look at the area collected by the KH-9. The Soviet Union was a big place: 20m km2. Each of the four film re-entry capsules could return the entire USSR, in stereo, with plenty of side overlap and margin for images fouled by clouds. Typically they chose to take more frames with smaller swaths (90 degrees or 60 degrees) to get higher average resolution, which brought down the total take somewhat to around 60 million km2.
My resolution numbers up there are slightly inflated. The film used could only eke out 130 lp/mm when given the maximum possible amount of contrast, such as would be seen at a shadow line. For finding something like paint on a road they were about half that. Pixellated sensors have a much more drastic cliff, of course. So the e.g. KH-9 resolution above might be compared to anything like a 20 to 30 gigapixel camera today. I'll note that I don't have any of those to suggest.
There are two big concepts here that I think are important. The first is the mechanical and logistical difficulties of using film. Below I've spelled out some of the details. The second is that despite these headaches, until very recently, film has been superior in many ways to electronic sensors for massive survey work.
The basic problems with using film stem from the fact that the sensor surface is a thin, pliable, moving, relatively inexpensive object that has to be held within the camera with very high precision. There must have been several problems associated with getting the film aligned within +/- 11 microns of the lens's focal plane. Among other things, the machines resemble Van de Graf generators, and so the film is subject to static electricity buildup and discharges, along with heating that tends to make it even more pliable and sticky. To reduce the static buildup, many of these cameras slid the film over surfaces with hundreds of pores lifting the film with pressurized air.
I think the Itek designs spun the entire optical assembly at a constant rate. The entire spinning optical assembly is inside a gimbal which rocks back and forth 1.6 degrees in each cycle. The rocking motion accomplishes forward motion compensation, so that the sweep of the slit across the ground is orthogonal to the direction of flight. This compensation ensures that the motion of the image on the film moves almost exactly with the film, and there is no blurring in the direction of flight during longer exposures. This rocking motion must have required fairly large torques, and I'm sure this is one of the things that the Perkin-Elmer folks balked at when considering the Itek design in space. Note that a constantly rotating camera sweeps at the outer edges faster than at the center, so the forward motion compensation probably had to vary it's rate of rocking as it went to compensate.
Here is a diagram which shows how the film was wrapped around the roller cage in the Itek designs. As the optical assembly (including the roller cage) twists counterclockwise, the film is transported clockwise.
However, even with a constant spin rate, the film does not transport at a constant rate. For instance, in the SR-71 OBC, a frame is exposed for 640 ms, during which time the film rips around the roller cage at 2.9 meters per second (that's what the rollers see). For the next second, the film advances at just 1 meter per second, so that the frame boundary going across the top of the roller cage can meet up with the slit going around the bottom. Because of the speed change, many of the freewheeling rollers on the roller cage will contact unexposed film coming from the framing roller with a tangential speed difference of 1.9 meters per second. As each freewheeling roller changes speed to match the film, it seems to me it would tend to scuff the film. I'm sure they made sure to make those rollers as lightweight as possible to reduce their rotational momentum.
Note the unlabeled slit after the second mirror right next to the film wrapped around the roller cage. Only the portion of the film after this in the optical chain has light on it, so this is the only spot that must be held accurately. I don't really know how it was done, since every moving belt of material that I've even seen has vibrated. They may have had a glass reseau plate that the film slid across. Sliding film across glass at 2.9 meters per second seems like an excellent way to scratch one or both. I have no evidence for it yet, but this seems like a good application for the compressed air film handling scheme.
The Itek forward motion compensation gimbal also took out airplane pitch motions. Airplane roll motions were taken out by varying the optical tube roll rate. That's easy enough (it doesn't require torque), but the film rate supplied to the roller cage assembly in the back also had to change to match.
That last diagram gives a pretty good clue to another challenge in this design -- film curvature. Although I've not found any labelled dimensions, it looks like the roller cage in the Itek designs was about 300 mm in diameter. The design of the roller cage really had me puzzled for a while, because the film transport path is cylindrical, but the film being exposed by the slit has to be flat. I finally figured it out when I took a good look at this photo of the Apollo OBC (also made by Itek):
The roller cage is a cylindrical cage of ROLLERS! Duh! As the film passes between the rollers, it's locally flat, and that's how they keep the film surface matched to the lens focal plane. Here's a 5x blowup of the roller cage in the picture above. It looks like the rollers are packed together as close as they can get in order to minimize the polygonal variation in film distance from the center of roller cage rotation. I think this variation leads to a (small) variation in film speed at the exposure site.
There should be a spot in the roller cage where there aren't any rollers, and the film becomes planar for a while. This spot would be over the exposure slit. In the Apollo OBC pictured here, the gap between the rollers must be at least 7.6mm, and given the orientation of the lens at the top it should be on the back side of the roller cage that we don't see here.
The film is pulled taut around the stuttering, bucking and twisting roller cage with an assembly that looks like the following. During the exposure film is pulled around the roller cage at 5.7 meters/second. Here's the film path:
If the tension on the film is too small, small irregularities in it's "set" curvature will make it lift off as it goes around the roller cage, and small patches of the film will lift away from the cage. With a +/- 11 micron depth of focus, it doesn't take much lift off to make a problem. If the tension is too high, the film will wrinkle longitudinally.
The Perkin-Elmer design did not have the roller cage or the gimbal. Instead, they had a twisting platen assembly at the focal plane. This would twist back and forth through 130 degrees as the optical bar rotated through 360 degrees. The two were nearly locked together through the 120 degrees of rotation that were used for exposure.
Because the Perkin-Elmer design has no rocker gimbal doing forward motion compensation, and the optical assemblies rotate at constant speed, the sweep is faster at the edges than in the center, and the area swept in each frame is slightly S shaped. They may have splayed the roll axes of the optical bars to get first order forward motion compensation, but this doesn't change the S shape. To keep the image from smearing across the film, the KH-11 has to keep the slit perpendicular to the motion of the slit across the ground, accounting for the changing sweep rate versus spacecraft velocity, as well as the rotation of the Earth, which is 6% of the spacecraft velocity at the equator.
This is why the twisting platen in the KH-11 is servo locked to the twisting optical assembly. They have to vary the relative twist of the two a little bit to keep the projected slit perpendicular to it's projected motion.
After the picture was shot the film sat around for a month in a temperature and humidity controlled environment, and then was dropped, recovered, and developed. There was a lot of opportunity for the film to shrink differentially. All the mechanical twisting, as well as film shrinkage, must have made photogrammetry a nightmare.
The Sunny 16 rule says that you can properly expose ISO 100 film in bright daylight conditions with a f/16 lens with a 10 ms exposure. The KH-9 used mostly monochrome Kodak 1414, which has an Aerial Film Speed of 15, which I think is equivalent to ISO 38. In full sun a 1 ms exposure at f/3 would have worked fine. On the Apollo OBC that corresponds to a 1.9mm wide slit. They did have exposure control hardware, and it looks like they could exposure for two stops dimmer than full sun. They might have also stopped down from that, in order to avoid blowouts over ice.
At this point, I'm sure those of you who have worked with digital sensors are feeling pretty smug. But consider how wonderful film was for capturing and returning large amounts of imagery.
Each of the four re-entry vehicles on the KH-9 would bring back 5,000 36 gigapixel images. If somehow compressed to 1 bit per pixel, that would be about 20 terabytes. These days that's about 10 hard disks, and would take about three months to downlink at a constant 25 megabits per second. Since they were returning these re-entry vehicles every few months, a modern downlink is only barely adequate. It has only been in the last 10 years or so that disk drive capacities have become high enough to fit the data into the payload of one of those re-entry vehicles -- 30 years after the KH-9 was originally deployed.
In 1971, the area of film actively integrating photons in the KH-9 was 155 mm x 15 mm. The largest, fastest TDI CCD sensors commercially available in 2013 are 64 mm x 1.3 mm. The pixels on these are 5.2 microns rather than 3.7 as on film. The smaller integrating length (1.3 mm versus 22 mm) gives a maximum exposure time of 240 microseconds, which is smaller than the 2 milliseconds we would prefer. 155 mm x 10 mm CCDs with 3.7 micron pixels are not commercially available, but could probably be custom made.
Another issue would be the readout rate. A fast TDI in 2013 reads out lines at 90 KHz. The KH-9 was exposing film in the roller cage at 5.5 meters/second, which corresponds to a line readout rate of 1.5 MHz. This could be achieved with a custom built TDI sensor maybe 5 years ago. It would require 1300 ADCs running at 45 MHz, which would burn a lot of power. This might be possible with interesting packaging.
The capture rate of the KH-9 was so enormous it completely overwhelmed the ability of the photointerpreters at the CIA to examine the photographs. It's only recently that computers have gotten large enough to store and process imagery at this volume, and it's not at all clear that anyone has yet developed algorithms to find the "unknown unknowns" in a bunch of raw imagery. I think they call this "uncued image search".
To my knowledge the U.S. never again fielded a spysat with the ability to survey major portions of the earth at high resolution. Later Keyhole satellites appear to have concentrated more on taking valuable single shots at higher resolution (7 cm GSD), and on having the orbital maneuverability to get those shots. I think the intelligence folks lost interest in survey satellites when it became clear that they couldn't take advantage of the comprehensive coverage which was their primary feature. It's kind of ironic that the very problem that Itek was founded to solve (managing the huge volume of survey photography) ended up being a major reason why satellites with survey capacity made possible by Itek's cameras faded away. It's fascinating for me to see what has become of this problem.
Brian McClendon is on record as saying that Google has 5 million miles and 20 petabytes of Street View imagery. That's the processed imagery, not the raw take. The KH-9 raw take that overwhelmed the CIA photointerpreter capacity 30 years ago was less than 1% of what Google Street View shot. Unlike the KH-9 imagery, most of which I suspect has never been looked at, every one of the Street View panoramas has been seen by a real user. And Google is hardly the only organization using Big Data like this. The consumption problem that the CIA and NRO never solved has been utterly crushed by organizations with entirely different goals. (Granted, uncued search remains unsolved.)
Now that Big Data has caught up with the photo throughput of digital satellites, it's fun to think about what could be built with modern technologies. But that's probably best saved for another post.
Here's an exotic high flying camera: ARGUS-IS. I've been trying to figure what these folks have been up to for years, and today I found an SPIE paper they published on the thing. What follows is a summary and my guesses for some of the undocumented details. [Updated 30-Jan-2013, to incorporate new info from a Nova documentary and a closer reading of the SPIE paper.]
Vexcel Ultracams also use four cameras
with interleaved sensors
It looks like BAE is the main contractor. They've subcontracted some of the software, probably the land station stuff, to ObjectVideo. BAE employs Yiannis Antoniades, who appears to be the main system architect. The lenses were subcontracted out to yet another unnamed vendor, and I suspect the electronics were too.
Field of view: 62 degrees. Implied by altitude 6 km and object diameter 7.2 km.
Image sensor: 4 cameras, each has 92 Aptina MT9P031 5 megapixel sensors. The paper has a typo claiming MT9P301, but no such sensor exists. The MT9P031 is a nice sensor, we used it on the R5 and R7 Street View cameras.
It's easy to interface with, has high performance (quantum efficiency over 30%, 4 electrons readout noise, 7000 electrons well capacity), and is (or was) easy to buy from Digikey. (Try getting a Sony or Omnivision sensor in small quantities.)
Focal length: 85mm. Implied by 2.2 micron pixel, altitude of 6 km, GSD of 15 cm. Focal plane diameter is 102mm. The lens must resolve about 1.7 gigapixels. I must say that two separate calculations suggest that the focal length is actually 88mm, but I don't believe it, since they would have negative sensor overlap if they did that.
F/#: 3.5 to 4. There is talk of upgrading this system to 3.7 gigapixels, probably by upgrading the sensor to the Aptina MT9J003. An f/4.0 lens has an Airy disk diameter of 5.3 microns, and it's probably okay for the pixels to be 2.2 microns. But 1.66 micron pixels won't get much more information from an f/4.0 lens. So, either the lens is already faster than f/4.0, or they are going to upgrade the lens as well as the sensors.
The reason to use four cameras is the same as the Vexcel Ultracam XP: the array of sensors on the focal plane cannot cover the entire field of view of the lens. So, instead, they use a rectangular array of sensors, spaced closely enough so that the gaps between their active areas are smaller than the active areas. By the way guys (Vexcel and ObjectVideo), you don't need four cameras to do this problem, it can be solved with three (the patent just expired on 15-Jul-2012). You will still need to mount bare die.
The four cameras are pointed in exactly the same direction. Offsetting the lenses by one sensor's width reduces the required lens field of view by 2.86 degrees, to about 59 degrees. That's not much help. And, you have to deal with the nominal distortion between the lenses. Lining up the optical axes means the nominal distortion has no effect on alignment between sensors, which I'm sure is a relief.
The sensor pattern shown in the paper has 105 sensors per camera, and at one point they mention 398 total sensors. The first may be an earlier configuration and the latter is probably a typo. I think the correct number is 92 sensors per camera, 368 total. So I think the actual pattern is a 12x9 rectangular grid with 11.33mm x 8.50mm centers. 16 corner sensors (but not 4 in each corner) are missing from the 9x12=108 rectangle, to get to 92 sensors per focal plane. The smallest package that those sensors come in is 10mm x 10mm, which won't fit on the 8.5mm center-to-center spacing, so that implies they are mounting bare die to the focal plane structure.
They are carefully timing the rolling shutters of the sensors so that all the rolling shutters in each row are synchronized, and each row starts it's shutter right as the previous row finishes. This is important, because otherwise when the camera rotates around the optical axis they will get coverage gaps on the ground. I think there is a prior version of this camera called Gorgon Stare which didn't get this rolling shutter synchronization right, because there are reports of "floating black triangles" in the imagery, which is consistent with what you would see on the outside of the turn if all the rolling shutters were fired simultaneously while the camera was rotating. Even so, I'm disappointed that the section on electronics doesn't mention how they globally synchronize those rolling shutters, which can be an irritatingly difficult problem.
They are storing some of the data to laptop disk drives with 160 GB of storage. It appears they may have 32 of these drives, in which case they've got enough space to potentially store the entire video stream, but only with very lossy video compression. The design presented has only JPEG2000 (not video) compression, which will be good for stepping through the frames, but the compression ratio will be bulky enough that there is no way they are storing all the video.
They have 184 FPGAs at the focal plane for local sensor control, timestamping, and serialization of the data onto 3.3 Gb/s fiber optics. Supposedly the 3.3 Gb/s SerDes is on the FPGA, which sounds like a Virtex-5 20T. But something is odd here, because having the SerDes on the FPGA forces them to choose a fairly beefy FPGA, but then they hardly do anything with it: the document even suggests that multiplexing the two sensor data streams, as well as serialization of those streams, happens outside the FPGA (another typo?). So what's left for a Virtex-5 to do with a pair of sensors? For comparison, I paired one Spartan-3 3400A with each sensor in R7, and we were able to handle 15 fps compression as well as storage to and simultaneous retrieval from 32 GB of SLC flash, in that little FPGA. Maybe the SerDes is on some other device, and the FPGA is more of a PLD.
The data flows over fiber optics to a pile of 32 6U single board computers, each of which has two mezzanine cards with a Virtex 5 FPGA and two JPEG2000 compressors on it.
Now here's my critique of this system design:
They pushed a lot of complexity into the lens.
It's a wide angle, telecentric lens. Telecentric means the chief rays coming out the back, heading to the focal plane, are going straight back, even at the edges of the focal plane. Said another way, when you look in the lens from the back, the bright exit pupil that you see appears to be at infinity. Bending the light around to do that requires extra elements. This looks a lot like the lenses used on the Leica ADS40/ADS80, which are also wide angle telecentric designs. The Leica design is forced into a wide angle telecentric because they want consistent colors across the focal plane, and they use dichroic filters to make their colors. The ARGUS-IS doesn't need consistent color and doesn't use dichroics... they ended up with a telecentric lens because their focal plane is flat. More on that below.
The focal lengths and distortions between the four lenses must be matched very, very closely. The usual specification for a lens focal length is +/- 1% of nominal. If the ARGUS-IS lens were built like that, the image registration at the edge of field would vary by +/- 500 microns. If my guesses are right, the ARGUS-IS focal plane appears to have 35x50 microns of overlap, so the focal lengths of the four lenses will have to match to within +/- 0.07%. Wow.
"The lenses are athermalized through the choice of glasses and barrel materials to maintain optical resolution and focus over the operational temperature range." Uh, sure. The R7 StreetView rosette has 15 5 megapixel cameras. Those lenses are athermalized over a 40 C temperature range, and it was easy as pie. We just told Zemax a few temperature points, assumed an isothermal aluminum barrel, and a small tweak to the design got us there. But those pixels have a field of view of 430 microradians, compared to the pixels behind the ARGUS-IS lens, which have a 25 microradian PFOV. MIL-STD-810G, test 520.3, specifies -40 C to 54 C as a typical operating temperature range for an aircraft equipment bay. If they had anything like this temperature range specified, I would guess that this athermalization requirement (nearly 100 degrees!) came close to sinking the project. The paper mentions environmental control within the payload, so hopefully things aren't as bad as MIL-STD-810G.
The lenses have to be pressure compensated somehow, because the index of refraction of air changes significantly at lower pressures. This is really hard, since glasses, being less compressible than air, don't change their refractive indices as fast as air. I have no particularly good ideas how to do it, other than to relax the other requirements so that the lens guys have a fighting chance with this one. Maybe the camera can be specified to only focus properly over a restricted range of altitudes, like 4km to 8km. (ARGUS-IR specifies 0 to 10km. It's likely ARGUS-IS is the same, so no luck there.) Or maybe everything behind their big flat window is pressurized.
They made what I think is a classic system design mistake: they used FPGAs to glue together a bunch of specialized components (SerDes, JPEG compressors, single board computers), instead of simply getting the job done inside the FPGAs themselves. This stems from fear of the complexity of implementing things like compression. I've seen other folks do exactly the same thing. Oftentimes writing the interface to a off-the-shelf component, like a compressor or an encryption engine, is just as large as writing the equivalent functionality. They mention that each Virtex-5 on the SBC has two 0.6 watt JPEG2000 chips attached. It probably burns 200 mW just talking to those chips. It seems to me that Virtex could probably do JPEG2000 on 80 Mpix/s in less than 1.4 watts. Our Spartan-3 did DPCM on 90+ Mpix/s, along with a number of other things, all in less than 1 watt.
I think I remember reading that the original RFP for this system had the idea that it would store all the video shot while airborne, and allow the folks on the ground to peruse forward and backward in time. This is totally achievable, but not with limited power using an array of single-board PCs.
Let me explain how they ended up with a telecentric lens. A natural 85mm focal length lens would have an exit pupil 85mm from the center of the focal plane. Combine that with a flat focal plane and sensors that accept an f/1.8 beam cone (and no offset microlenses), and you get something like the following picture. The rectangle on the left is the lens, looking from the side. The left face of the right rectangle is the focal plane. The big triangle is the light cone from the exit pupil to a point at the edge of the focal plane, and the little triangle is the light cone that the sensor accepts. Note that the sensor won't accept the light from the exit pupil -- that's bad.
There are two ways to fix this problem. One way is to make the lens telecentric, which pushes the exit pupil infinitely far away from the focal plane. If you do that, the light cone from the exit pupil arrives everywhere with it's chief ray (center of the light cone) orthogonal to the focal plane. This is what ARGUS-IS and ADS-80 do.
The other way is to curve the focal plane (and rename it a Petzval surface to avoid the oxymoron of a curved focal plane). Your retina is curved behind the lens in your eye, for example. Cellphone camera designers are now looking at curving their focal planes, but it's pretty hard with one piece of silicon. The focal plane array in ARGUS-IS is made of many small sensors, so it can be piecewise curved. The sensors are 7.12 mm diagonally, and the sag of a 85 mm radius sphere across 7.12 mm is 74 microns. The +/- 9 micron focus budget won't allow that, so curving the ARGUS-IS focal plane isn't going to allow a natural exit pupil. The best you can do is curve the focal plane with a radius of 360 mm, getting 3.6 mm of sag, and push the exit pupil out to about 180 mm. It's generally going to be easier to design and build a lens with an exit pupil at 2x focal length rather than telecentric, but I don't know how much easier. Anyway, the result looks like this:
As I said, the ARGUS-IS designers didn't bother with this, but instead left the focal plane flat and pushed the exit pupil to infinity. It's a solution, but it's not the one I would have chosen.
Here's what I would have done to respond to the original RFP at the time. Note that I've given this about two hours' thought, so I might be off a bit:
I'd have the lenses and sensors sitting inside an airtight can with a thermoelectric cooler to a heat sink with a variable speed fan, and I'd use that control to hold the can interior to between 30 and 40 C (toward the top of the temperature range), or maybe even tighter. I might put a heater on the inside of the window with a thermostat to keep the inside surface isothermal to the lens. I know, you're thinking that a thermoelectric cooler is horribly inefficient, but they pump 3 watts for every watt consumed when you are pumping heat across a level. The reason for the thermoelectric heat pump isn't to get the sensor cold, it's to get tight control. The sensors burn about 600 mW each, so I'm pumping 250 watts outs with maybe 100 watts.
I'd use a few more sensors and get the sensor overlap up to 0.25mm, which means +/-0.5% focal length is acceptable. I designed R5 and R7 with too little overlap between sensors and regretted it when we went to volume production. (See Jason, you were right, I was wrong, and I've learned.)
Focal plane is 9 x 13 sensors on 10.9 x 8.1 mm centers. Total diameter: 105mm. This adds 32 sensors, so we're up to an even 400 sensors.
Exiting the back of the fine gimbal would be something like 100 flex circuits carrying the signals from the sensors.
Hook up each sensor to a Spartan-3A 3400. Nowadays I'd use an Aptina AR0330 connected to a Spartan-6, but back then the MT9P001 and Spartan-3A was a good choice.
I'd have each FPGA connected directly to 32GB of SLC flash in 8 TSOPs, and a 32-bit LPDDR DRAM, just like we did in R7. That's 5 bytes per pixel of memory bandwidth, which is plenty for video compression.
I'd connect a bunch of those FPGAs, let's say 8, to another FPGA which connects to gigabit ethernet, all on one board, just like we did in R7. This is a low power way to get connectivity to everything. I'd need 12 of those boards per focal plane. This all goes in the gimbal. The 48 boards, and their power and timing control are mounted to the coarse gimbal, and the lenses and sensors are mounted to the fine gimbal.
Since this is a military project, and goes on a helicopter, I would invoke my fear of connectors and vibration, and I'd have all 9 FPGAs, plus the 8 sensors, mounted on a single rigid/flex circuit. One end goes on the focal plane inside the fine gimbal and the other goes on the coarse gimbal, and in between it's flexible.
I'd connect all 52 boards together with a backplane that included a gigabit ethernet switch. No cables -- all the gigE runs are on 50 ohm differential pairs on the board. I'd run a single shielded CAT-6 to the chopper's avionics bay. No fiber optics. They're really neat, but power hungry. Maybe you are thinking that I'll never get 274 megabits/second for the Common Data Link through that single gigE. My experience is otherwise: FPGAs will happily run a gigE with minimum interpacket gap forever, without a hiccup. Cheap gigE switches can switch fine at full rate but have problems when they fill their buffers. These problems are fixed by having the FPGAs round-robin arbitrate between themselves with signals across that backplane. Voila, no bandwidth problem.
The local FPGA does real time video compression directly into the flash. The transmission compression target isn't all that incredible: 1 bit per pixel for video. That gets 63 channels of 640x400x15 frames/sec into 274 Mb/s. The flash should give 1 hour of storage at that rate. If we want 10 hours of storage, that's 0.1 bits/pixel, which will require more serious video compression. I think it's still doable in that FPGA, but it will be challenging. In a modern Spartan-6 this is duck soup.
The computer tells the local FPGAs how to configure the sensors, and what bits of video to retrieve. The FPGAs send the data to the computer, which gathers it up for the common data link and hands it off.
I'll make a guess of 2 watts per sensor+FPGA+flash, or 736 watts. Add the central computer and switch and we're at 1 kilowatt. Making the FPGAs work hard with 0.1 bit/pixel video compression might add another 400 watts, at most.
No SSDs, no RAID, no JPEG compression chips, no multiplexors, no fiber optic drivers, no high speed SerDes, no arrays of multicore X86 CPUs. That's easily half the electronics complexity, gone.
UPDATE 25-Jan-2013: Nova ran a program on 23-Jan-2013 (Rise of the Drones) which talks about ARGUS-IS. They present Yiannis Antoniades of BAE systems as the inventor, which suggests I have the relationship between BAE and ObjectVideo wrong in my description above. They also say something stupid about a million terabytes of data per mission, which is BS: if the camera runs for 16 hours the 368 sensors generate 2,000 terabytes of raw data.
They also say that the ARGUS-IS stores the entire flight's worth of data. I don't think they're doing that at 12 hertz, certainly not on 160 GB drives. They've got 32 laptop drives in the system (one per single board computer). If those store 300 GB apiece, that's 10 terabytes of total storage. 16 hours of storage would require 0.05 bits/pixel -- no way without actual video compression. The JPEG2000 compressor chips are more likely to deliver at best 0.2 bits/pixel, which means they might be storing one of every four frames.
UPDATE 27-Jan-2013: An alert reader (thanks mgc!) sent in this article from the April/May 2011 edition of Science and Technology Review, which is the Lawrence Livermore National Laboratory's own magazine. It has a bunch of helpful hints, including this non-color-balanced picture from ARGUS-IS which lets you see the 368 sensor array that they ended up with. It is indeed a 24 x 18 array with 16 sensors missing from each corner, just as I had hypothesized.
The article mentions something else as well: the Persistics software appears to do some kind of super-resolution by combining information from multiple video frames of the same nearly static scene. They didn't mention the other two big benefits of such a scheme: dynamic range improvement and noise reduction (hence better compression). With software like this, the system can benefit from increasing the focal plane to 3.8 gigapixels by using the new sensor with 1.66 micron pixels. As I said above, if the lens is f/3.5 to f/4.0 lens they won't get any more spatial frequency information out of it with the smaller pixels, but they will pick up phase information. Combine that with some smart super-resolution software and they ought to be able to find smaller details. Question though: why not just go to the MT9F002, which gives you 14 million 1.4 micron pixels? This is a really nice, fast sensor -- I've used it myself.
The article also mentions 1000:1 video compression. That's very good: for comparison, H.264 level 4 compresses 60 megapixels/second of HDTV into 20 megabits/second, which is 0.33 bits/pixel or 36:1 compression. This isn't a great comparison, though, because Persistics runs on almost completely static content and H.264 has to deal with action movie sequences. In any case, I think the Persistics compression is being used to archive ARGUS-IS flight data. I don't think they are using this compression in the aircraft.
When I read the Pocohontas story to my kids (we have the Disney version), we usually have a little discussion when we get to the page where Pocohontas attempts to dissuade her father (the local Indian chief) from starting a war with the settlers. The kids are interested in the idea that both people are trying to do the right thing, but they have completely different ideas about what the right thing is.
For those of you not familiar with the story, Pocohontas has fallen in love with a mercenary on the voyage (John Smith), and the two of them want to establish peace between the settlers and the natives. The book suggests that peace involves the settlers staying in North America. Powhatan, her father, is assembling a war party to drive the settlers away.
We can look back in history to better understand who was "right".
As the book makes clear, a war between the settlers and the Indians is going to lead to many Indian casualties, since the settlers have guns and the Indians do not. Furthermore, most of the settlers are not intending to do harm to the Indians, as they've been told they are settling land that has no ownership yet. Pocohontas' efforts end up saving many well-intentioned people's lives.
These same settlers would probably understand that, had they landed anywhere in England and built a village where they landed, they would be summarily evicted by whomever owned the land they were on. The racism here is lightly touched on in the book, but it's helpful because it's pretty easy for the kids to see how convenient it is for the settlers to suppose that nobody in North America owns anything yet.
I usually tell the kids what little I know of the Mauri, the indigenous people of New Zealand. As I understand it, they immediately made war with white folks who arrived. I suspect that the Mauri were territorial in a way that worked better with the White conception of property, and because of that Mauri today have a significant representation in the New Zealand constitution and legislature, and own very large amounts of New Zealand's real estate. I expect many Native Americans would prefer the Mauri outcome to their own.
I recently went to see Avatar. It's basically the Pocohontas story, but the ending has changed and the natives switch from the Pocohontas to the Mauri approach. The change comes from two differences:
The Na'vi are territorial. They have a few specific high-value trees. My understanding is that most of the North American natives had a much less specific sense of property.
The movie has the natives resisting under human leadership, which is interesting to think about. It seems a bit condescending (especially the bit where the human, after 3 months of training, is outperforming the best of the natives), but historically North American natives really did not grasp the nature of the European threat fast enough to organize a massive resistance in time, and it seems at least possible that a charismatic European might have communicated the continent-level consequences of the European idea of property to enough of them to organize a resistance.
Although the movie doesn't make it clear enough, guns are a big advantage, but a multi-year supply line is an even bigger disadvantage. Although some of the dialog is a bit trite, I think the story is probably going to be a useful place to start interesting discussions. Hopefully they'll have some story books out at some point, because the PG-13 movie is far too violent for my kids to watch.
I once asked a friend who is a lawyer if all property rights, at least in North America, trace back to peace treaties of some kind, or if some (particular the French claim to the center of the continent that was then sold as the Louisiana Purchase) are based on bald assertions of authority without even a war. I never did get a decent answer.
If, in reading this post, anyone is wondering if I'm willing to cede my house to a Native American, the answer is no.
Here is a set of essays on the calculus of nuclear war, written by someone who used to plan nuclear war. They are short, funny in places, reassuring in places, and generally scary.
Of course, no mention of nuclear weapons is complete without directing readers to the Nuclear Weapons Archive, by Carey Sublette. I remember first reading the FAQ in 1996 or so, and being astounded. It changed the way I thought about The Bomb.
It's the physics bit that got me. I had previously though of fusion bombs as being somewhat like the Sun, only, here. But it turns out that fusion in the Sun proceeds along quite slowly, at comparatively low temperatures and pressures. Fusion bombs operate at much higher pressures and temperatures than stars do, and (obviously) on much shorter timescales. It turns out to be almost completely different physics.
For some reason that really bothers me. The notion that we use physics that can't even be observed anywhere in the natural world seems odd. Perhaps I'm succumbing to nuclear hocus pocus, since I can't think of anywhere in the natural world that we can observe hydrocarbon-oxygen combustion at dozens of atmospheres of pressure, and yet our cars and airplanes do that all the time.
Former Navy General Counsel Alberto Mora: “there are serving U.S. flag-rank officers who maintain that the first and second identifiable causes of U.S. combat deaths in Iraq – as judged by their effectiveness in recruiting insurgent fighters into combat – are, respectively the symbols of Abu Ghraib and Guantanamo."
Jonathan Fredman, chief counsel to the CIA’s CounterTerrorist Center: "If the detainee dies you’re doing it wrong."
In mid-August 2003, an email from staff at Combined Joint Task Force 7 headquarters in Iraq requested that subordinate units provide input for a “wish list” of interrogation techniques, stated that “the gloves are coming off,” and said “we want these detainees broken.”
JPRA Commander Colonel Randy Moulton’s authorization of SERE instructors, who had no experience in detainee interrogations, to actively participate in Task Force interrogations using SERE resistance training techniques was a serious failure in judgment.
Secretary of Defense Donald Rumsfeld’s authorization of aggressive interrogation techniques for use at Guantanamo Bay was a direct cause of detainee abuse there.
I think at this point I'm just sick of all the damage that has been done to my country by Bush and his team. I doubt that throwing many of them in jail will do much to improve the behavior of similarly-minded people, but I'm all for prosecutions so long as they don't shift attention from the job at hand, which is to fix the economy.
I'm not a military guy, I don't know much about how they do things. But I have read Blackhawk Down, and I have some sense that a casualty is a much bigger problem than one guy getting shot. If there is no well-linked rear to which to send a casualty, a fire team has a huge liability. I think the usual rule is that one casualty effectively soaks up four people. It reduces the fire team's mobility and effectiveness, and can rapidly send a mission down a cascade of further problems. So, I got to thinking about how you could improve combat rescue.
Let's assume you control the airspace about the battlefield, or at least the portion of it above small-arms range. Helicopters work pretty well when you want to insert your fire teams, because folks near the target can often be taken by surprise and the choppers can dump their loads and be off before serious resistance is organized. But helicopters are not a good way to get people back out, because they move slowly near the landing zone and are thus pretty easy targets. What you need, getting out, is a lot of acceleration and altitude, right away. You want a rocket.
The wounded guy goes into a stretcher. I'm imagining something like a full-body inflatable splint: it squeezes the hell out of him, totally immobilizing him, and insulating him from cold and wind. You'd design the thing so that it could be popped in a couple of places and still work fine. The stretcher gets attached to a rope attached to a reel at the bottom of the rocket.
The rocket fires a very short exhaust pulse, which sends the thing up 50 feet or so. At this point the rope is entirely unreeled. When the rope goes taut, the main burn starts, accelerating the stretcher at, say, 5G straight up. The exhaust plume is directed out two symmetrical nozzles slightly away from straight down so that the poor guy at the bottom doesn't get burned. Acceleration drops to 1G for ten seconds or so once the guy is at a few hundred miles per hour, and then cuts out. The rocket coasts to a stop at 10,000 feet or so, at which point a parasail pops out.
At this point an autopilot yanking on the control lines can fly the guy ten miles away to get picked up on the ground, or a helicopter or C-130 can grab him out of midair. A midair grab sounds ridiculous but apparently they already use this technique for recovering deorbited film capsules and they haven't dropped any yet. A midair pickup at 2000 feet would have 8 minutes to snatch a guy falling from 10,000 feet at 16 feet/second, which seems plausible with good communication.
[Update: apparently they already use midair grabs for picking up people, too. They use a helium balloon and snag that. The trouble is that when they winch the guy in, he generally spins around in the vortex behind the airplane, and when he gets to the tail of the airplane he can get bashed against the fuselage a fair bit before they get him inside.]
A rocket sufficient to boost 300 lbs of payload to 3200 meters needs about 300 m/s delta-V. With a mass ratio of 80% and an Ve of 2600 m/s, the rocket will weigh 120 pounds. That's not something you want to be carrying around with you, but it is something that one guy can manhandle into an upright position. So you have to deliver this heavy, bulky thing to a fire team in the middle of a combat zone which is already distracted by tending to some casualties. Luckily, you can make the rocket pretty tough.
I suggest dropping the recovery package (ascent rocket, stretcher, medical kit, ammunition) on the fire team as you might drop a smart bomb. Instead of exploding a warhead, this munition pops a parachute or fires a retrorocket right before impact to minimize the damage to whatever it hits and cushion the blow to the medical kit. Someone on the fire team might use a laser designator to pick the landing spot, so that they have good control over the difficulty of recovering the thing. You'd want to be careful: bomb there, recovery kit here.
I posted about this three years ago in this thread: http://groups-beta.google.com/group/sci.space.tech/browse_thread/thread/efb906c8dd19915a/a355a9c6b2ed55f5?hl=en Back then I thought you needed the robot paraglider to deliver the recovery package. Now I suspect something more like the smart bombs we already have would be okay.
