HD View

There are two main tools for converting a large image into the HD View format.  They are hdmake, a command line application, and the HD View Photoshop plugin.  The HD View format is very similar to the Deep Zoom format.  So, we are pleased to announce that we have added Silverlight Deep Zoom capabilities to our latest release of these two tools.  In either tool you can now indicate Deep Zoom output and it will generate the image tiles, Silverlight files, and an example web page.  That means that our Photoshop plug-in gives you capabilities comparable to the Zoomify export built into CS3.  See this blog post for a take on Deep Zoom's advantages over Zoomify.

The download link is: http://research.microsoft.com/research/downloads/details/345a52c3-fe44-4045-94b4-4b26a93a907c/details.aspx 

-Matt

Deep Zoom is a terrific new feature that is available in Silverlight 2.0.  The best way to describe Deep Zoom is that it allows Silverlight to natively load arbitrarily large images.  To get an idea of what Deep Zoom can do, please take a look at the initial results of the Yosemite Extreme Panoramic Imaging Project.

But, what I really want to talk about today is the Deep Zoom authoring tool - Deep Zoom Composer (DZC).  The latest beta of DZC offers a feature that my group developed - automatic image stitching.  While we've shipped this in the past in other products like Windows Live Photo Gallery, this version of the stitcher has some significant improvements.  The first improvement is a better technique for automatically finding matching control points between images.  The second is better blending between images, especially when there are significant exposure differences.

Control points are very familiar to anyone who has used manually operated stitching software.  For those who haven't used such software, the basic work flow is that you open one image, click on a point in that image that you think will likely appear in another overlapping image, then open that other image and click on the corresponding point.  You do this over and over again until all images that comprise your panorama have points matching into other images.  Recently computer vision algorithms have been invented that can automatically perform this function.  The most well known of these algorithms is the Scale-invariant feature transform (SIFT).  SIFT works so well that it has largely replaced manual control point editing and there are several automatic stitchers available, such as Autostitch, that take advantage of SIFT.

SIFT however isn't perfect at matching images and sometimes misses matches or generates erroneous matches.  Last year, Simon Winder and Matthew Brown, two researchers in our group, tackled the problem of designing a better matching technique and published their work in a paper titled Learning Local Image Descriptors.  This paper is meant for a technical audience, so I will try to summarize the findings here.  SIFT was designed by smart people testing different ideas on a large number of test images.  The new technique instead uses a much much larger number of test images (produced by the Photosynth pipeline), and uses mathematical optimization techniques to automatically 'learn' a new control point matcher.  The net result is that the new technique produces 1/3 the number of false matches when compared to SIFT on our test data.  And ultimately for users of DZC this will lead to a higher success rate for your stitching projects.

It is personally very exciting for me to see Microsoft Research (MSR) results like these moving from academic journals to actual usable products so quickly.  The MSR computer vision researchers have long been active in academic circles, but now are finding more and more outlets in Microsoft products as well.  There are other examples such as Photosynth and HD View, where preview software was available very near to technical publication^{(1, 2)}.  I look forward to this trend continuing.

That said, please give the new Deep Zoom Composer a spin.  You can learn more about it and download it from:

http://blogs.msdn.com/expression/archive/2008/08/01/download-the-new-deep-zoom-composer-preview.aspx 

Note DZC is still Beta software and there are a few rough edges such as limits on output size.

-Matt Uyttendaele

⁽¹⁾ Photosynth was published at Siggraph 2006 as "Photo tourism: Exploring photo collections in 3D" in August and made available publicly in November.

⁽²⁾ HD View was published at Siggraph 2007 as "Capturing and Viewing Gigapixel Images" and made available in conjunction with the conference.

National Geographic's web site has a great gallery of 360 panoramas from Petra, Jordan.  They have a slide show of 6 different panoramas from around the ancient city.  For each of these you can click on "View in HD" to get the full-screen HD View experience.  Check it out here: http://www.nationalgeographic.com/history/ancient/petra-360.html

-Matt

This post is all about high dynamic range imaging (HDRI).  As I mentioned in the Beta 3 announcement, HD View is the best way, that we are aware of, to interact with HDR images over the web.  We take advantage of three recent advances to make this happen.  They are HD Photo compression technology, the PC's graphic processing unit (GPU), and high contrast displays.  I will describe each of these things, but first, a quick intro to high dynamic range imaging (HDRI). 

What is HDRI?

image courtesy of Mark Fairchild

Light in the world has a very large range of intensities.  In the figure above, you can see a 100 million to 1 range (alternatively 26 stops) in intensity from starlight to direct sunlight.  An HDR image is one that contains a measure of the intensity of light in a scene.  At each pixel, it records one of the numbers on the scale in the figure above.  This means that an HDR image can represent a scene that has a very large intensity difference between the shadows and the highlights.  Cameras today can't directly record scenes that have such an extreme range.  So to create these images, photographers stack multiple exposures together, one correctly exposed for the shadows, one for the mid-tones, one for the highlights, and so on.  These separate exposures are then merged by software^{(1, 2)} to produce an HDR image.  This is quite a different representation than a JPEG image.  In terms of range, a JPEG can represent only a small portion of the  possible range of intensities in a scene, which makes JPEG a low dynamic range (LDR) format.  This is why in a backlit situation your camera can produce a good JPEG of the foreground object or one of the bright background, but not both at once.  It turns out that most computer displays are also LDR, so even if a JPEG could store a greater range, you wouldn't be able to see it on a standard monitor.  JPEGs also differ from HDR in the way they store color information, but that is a topic for my next post.

So why are there so many HDR images stored as JPEGs on flickr?  That is thanks to a technique known as tone-mapping.  In order to make an HDR image display on an LDR monitor, tone-mapping algorithms compress the wide range of intensities in the HDR image to fit in the narrow range of the monitor.  There have been terrific advances in tone-mapping, but it is a very hard problem to make tone-mapped images look natural.  In order to compress the range, dark parts of the scene need to be brightened to the point that they have similar intensity to bright parts.  This can lead to an unnatural look, because as a human observer of this image, you expect bright objects to look a certain way when compared to dark objects, but the tone-mapping process has changed the tonal relationship between these objects.  I've heard many people say that they "don't like the HDR effect," but what I think they object to is the effect of the tone-mapping process.  It is important to separate these two things.  HDR images represent the intensities of light in a scene.  Tone-mappers make HDR images presentable on the computer monitors available today.

HD Photo and JPEG XR

Now onto how we are supporting HDR in HD View.  On the High Dynamic Range page of the HD View web site we include the following diagram that describes our HDR pipeline.

The pipeline starts with a camera capturing one or more RAW images.  A single RAW image by itself contains a greater dynamic range than JPEG can store, and as described above, if multiple bracketed RAW images are captured, these can be merged into an HDR image.  If we want to deliver this over the web, today that would involve the tone-mapping process described above followed by storing the result as a JPEG.  In our pipeline, we instead store the extended dynamic range image directly in a new image format called HD Photo.   There are other file formats available today that could also be used such as, .HDR, OpenEXR, and TIFF.  But, none of these offer the sophisticated compression algorithms that HD Photo does.  This compression allows us to effectively transfer HDR images over the web.  The file sizes of these other HDR file formats are simply too large for interactive web access.  As an example, the images on our HDR Survey page were downloaded from Mark Fairchild's site in OpenEXR format and converted to the HD Photo format.  The OpenEXR files range in size from 35MB to 46MB, by comparison after conversion to HD Photo the images are about 1/10th the size (3.6MB to 6MB). 

HD Photo started as a Microsoft Research project.  It was then refined and further developed by the Microsoft Codec Group.  Microsoft subsequently presented HD Photo to the JPEG standards body, and that organization has decided to make HD Photo a standard that will be known as JPEG XR.  I am very much looking forward to the time when JPEG XR emerges from the standardization process and is in ubiquitous use.  For now though, you can start experimenting with the new technology in the form of HD Photo.  The HD View team has provided a simple command line tool and a Photoshop plugin that you can use to convert your RAW and HDR images to the HD Photo format that HD View requires for streaming over the web.  HD Photo has lots of other great features beyond HDR, please see Bill Crow's blog for many more details.

GPUs

The graphics processing unit (GPU) of almost every modern PC now has HDR capabilities.  For example, a new $300 PC from Dell uses the Intel 3100 GPU.  This chip can load and process floating point images which is the same representation that HD Photo uses and is a great representation for HDRI⁽³⁾.  Of course higher end GPUs from NVidia and ATI have these capabilities as well, but the point is that even entry level PCs can now process HDR imagery in real time.  GPUs have this capability because in order to generate realistic looking 3D graphics, game designers need to simulate real world lighting.  Thus, today's most realistic looking games are in fact doing HDR.  GPUs also have the ability to run 'shader' programs.  These shaders, as they are known, can run fairly sophisticated image processing algorithms at interactive rates.  We use these capabilities to perform real time tone-adjustment on HDR images.

Displaying HDR

HDR displays are coming but they aren't available just yet.  When they become available our HD View pipeline won't need to change and any content that you've created for HD View will look great on them.  But for the foreseeable future we will also have LDR displays.  So now that we've delivered an HDR image all the way to an LDR monitor what do we do with it?  Remember that HD View is an interactive image viewer.  So, we can create the ideal tone adjustment given the end-users display, the portion of the image they are currently viewing and any intents we let them express.  This is quite different from the tone-mapped images you see today.  Those images are processed assuming that they will be viewed full-frame on an LDR monitor or on paper.  This can lead to the distorted relationship between tonal values I described above.  It can also lead to a flattening or loss of contrast in the image.  In HD View we instead do tone adjustment on the fly as the user views the image.  For example, when zoomed all the way out on an image we might see the shadows be truly dark, but as we zoom into those dark areas HD View will auto-expose to show previously indiscernible details (the ability to retain those details in the shadows is thanks to HD Photo).  In our pipeline, we've retained all of the dynamic range of the original photographs and we have the ability to run shaders, so we offer users a variety of different ways to do tone-adjustment on the fly.  These are accessed via the toolbar in the upper right of the window and are summarized in the table below.  For some screen shots of these modes in effect see our help page.

Camera for the Web

To wrap it up, I have described a pipeline that lets you maintain all of the light information that was captured from the original scene.  A common term used to describe this is a 'scene-referred' workflow.  In the past scene-referred workflows have been advocated for the image editing process but not for delivery over the web.  For web delivery conversion to JPEG is the standard.  Here we are allowing scene-referred imagery to travel all the way to the end user's monitor.  One way that we like to think of this is that HD View is a camera for the web.  It is performing a very similar task to camera hardware today which is reading scene-referred rays of light and generating the best possible JPEG image that it can.  We instead are enabling that scene-referred data to travel over the web.  The HD View "camera" can then generate the best possible photograph using additional information that is only available at the end of this pipeline.  This includes taking into account the monitor being used, the user's current pan/zoom interaction, and any additional settings that we want to expose to the user.  Currently these additional settings are the three tone-adjustments listed above.  However, we can see expanding this to any of the settings that you find in hardware cameras today.  Beyond that, we are running on powerful PC hardware, so our software camera can perform image processing that is just not feasible in your point and shoot camera.

Try it out and we always appreciate feedback.

-Matt Uyttendaele

⁽²⁾ Wide field-of-view panoramic images are also typically HDR.  In the Capturing and Viewing Gigapixel Images paper, a pipeline to produce HDR gigapixel images is described,  PT Gui Pro and Autopano Pro both have the ability to create HDR panoramas.

⁽³⁾ See figure 3. in Greg Ward's HDR encodings paper.

We are very pleased to announce that a new version of HD View is available on our web site.  There are a lot of great new features:

• Support for 16bit/channel and 32bit/channel images.  In other words, High Dynamic Range (HDR) images can now be efficiently streamed over the web to the HD View plugin.  We offer 3 different ways of interacting with HDR images - no auto-expose, auto-expose via gain only (like a camera), or auto-tone-map. In each 'auto' case the parameters are recomputed on the fly depending on what is in view.
• Now fully color managed.  HD View source files can have embedded color profiles, and monitor profiles are respected.  In conjunction with the 16 bit and 32 bit support, this means that wide-gamut images can be effectively delivered over the web.
• A new lens - the Fisheye.  This is accessed via the rightmost button in the toolbar.  See screenshot below for Fisheye in action.
• KML-PhotoOverlay tiled image representations can now be directly loaded.
• XBox controller as an input device.
• 3DConnexion as an input device.
• Easy to use embed code for adding HD View to your blog.
• A bunch of bug fixes (a lot of these based on your feedback - thanks!).  A few of the things we addressed:
  • Firefox3 issues
  • Embedding in PowerPoint presentations
  • Now works with IE Quick Tabs.

The first two bullet items are the most significant.  HD View 3 (and associated tools) represents the first system to efficiently deliver images over the web with no limitations on dimensions (gigapixels and beyond), dynamic range (32bit/channel support), or color gamut (ProPhoto or any other wide-gamut profile support). 

There is a lot to cover here, so over the next few days I'll dive into several of these bullet points in more detail.  For now please visit the updated HD View site and try it out on some HDR content.  To install, follow this link: http://research.microsoft.com/ivm/hdview/HDInstall.htm and click the install button.  Note: HD View requires Firefox, IE, or Safari on the Windows OS.

After installing, check out the new HDR content:

http://research.microsoft.com/ivm/hdview/HDRExamples.htm

http://research.microsoft.com/ivm/hdview/HDRSurvey.htm

-Matt Uyttendaele