Friday, December 14, 2012

Cloud + Local imagery storage

The Department of Homeland Security has said that it wants imagery delivered in the cloud. Several counties and states have expressed the same desire. Amazon S3 prices are quite reasonable for final imagery data delivered, especially compared to the outrageous prices that imagery vendors have been charging for what amounts to a well-backed-up on-site web server with static content. I've heard of hundreds of thousands of dollars for serving tens of terabytes.

Everyone wants the reliability, size, and performance scalability of cloud storage. No matter how popular your imagery becomes (e.g. you happen to have the only pre-Sandy imagery of Ocean City), and no matter how crappy your local internet link happens to be, folks all around the world can see your imagery. And many customers are more confident in Amazon or Google backups than their local IT backups.

But everyone also wants the reliability of local storage. E911 services have to stay up even when the internet connection goes down. So they need their own local storage. This also helps avoid some big bandwidth bills on the one site you know is going to hammer the server all the time.

So really, imagery customers want their data in both places. But that presents a small problem because you do want to ensure that the data on both stores is the same. This is the cache consistency problem. If you have many writers frequently updating your data and need transaction semantics, this problem forces an expensive solution. But, if like most imagery consumers you have an imagery database which is updated every couple of months by one of a few vendors, with no contention and no need for transaction semantics, then you don't need an expensive solution.

NetApp has a solution for this problem which involves TWO seriously expensive pieces of NetApp hardware, one at the customer site and one in a solo site with a fat pipe to Amazon. The two NetApp machines keep their data stores synchronized, and Amazon's elastic cloud accesses data stored at the colo over the fat pipe. This is... not actually cloud storage. This is really the kind of expensive shoehorned solution that probably pisses off customers more than me because they have to write the checks.

The right answer (for infrequently-updated imagery) is probably a few local servers with separate UPSes running Ceph and something like rsync to keep updates synchronized to S3. Clients in the call center fail over from the local servers to S3, clients outside the call center just use S3 directly.

I feel sure there must be some Linux vendor who would be happy to ship the Nth system they've build to do exactly this, for a reasonable markup to the underlying hardware.

Wednesday, December 05, 2012

Dragonfly Eyes - the Wonders of Low Resolution


Continuing with the recent theme of analyzing both natural and man made cameras, I thought I'd analyze an insect eye in this post.  In previous posts I've analyzed a gigapixel airborne security video camera and the eye of a raptor.  I'm working on analyses of the F-35's Distributed Aperture System as well as a really gigantic film camera for future posts.

The picture above is of a blue-spotted Hawker dragonfly.  This creature apparently catches its prey in midflight, so it's supposed to have big, high resolution eyes.  According to John Patterson in this Discover magazine article, it has 28,000 lenses in each eye.  Judging from the picture, those eyes are probably covering half of the possible 4*pi solid angle, so each tiny lens is covering a solid angle of 112 microsteradians.  That would be Low Resolution to my mind.  Critters like spiders and grasshoppers, with even smaller eyes, must be commensurately worse.

To give some context, a human in bright light has a pixel field of view of 25 nanosteradians.  You can see across your yard (70 feet) what the dragonfly can see from 1 foot.  If it is going after an 8mm long house fly, it needs to close within 80 cm in order to get a full pixel on the target.  At that range the house fly can probably hear the dragonfly, although I haven't analyzed that and insect hearing might also be much worse than human hearing.  This dragonfly is not a long-range hunter like a bird, but rather an opportunist.  I very much doubt it can pull off a high-relative-speed attack like a Peregrine falcon, which requires good angular resolution and gimbal stabilization.

Interestingly, the creature collects about the same amount of light per pixel that the human eye does.  The entire creature is about 70mm long.  Judging from the picture below, the eyes are probably 10 mm across.  That means those little lenses are 53 microns across.  On a bright day outside, your human pupils constrict to 3 mm, 55 times larger.  But since the dragonfly's pixel field of view is 70 times bigger, it actually receives about 50% more light per pixel than you do.
I used to be concerned that the tiny dragonfly lenses would cause diffraction blurring.  Supposing for a moment that they look at green light, their Rayleigh criterion is about 12 milliradians, nicely matched to their pixel angular field of view of 10.5 milliradians.  If the effective pupil of each lens is substantially smaller than the allotted space (low fill factor), then the dragonfly would have to use a shorter wavelength to avoid diffraction problems.  It doesn't look like a big problem.

I don't think the dragonfly has to stabilize the scenery with it's neck.  A dragonfly flying at 1 meter/second sees scenery 50 cm away to the side going by at 2 radians/second.  To get less than 3 pixels of motion smear, it would have to integrate the exposure for less than 15 milliseconds.  That's just a bit quick, so the creature probably sees some motion blur to the side, but not so much in front of it, since there is considerably less angular motion in the direction of flight.