HD View

Deep Zoom support in HD View tools

Tue, 19 Aug 2008 23:52:36 GMT

There are two main tools for converting a large image into the HD View format. They are hdmake, a command line application, and the HD View Photoshop plugin. The HD View format is very similar to the Deep Zoom format. So, we are pleased to announce that we have added Silverlight Deep Zoom capabilities to our latest release of these two tools. In either tool you can now indicate Deep Zoom output and it will generate the image tiles, Silverlight files, and an example web page. That means that our Photoshop plug-in gives you capabilities comparable to the Zoomify export built into CS3. See this blog post for a take on Deep Zoom's advantages over Zoomify.

The download link is: http://research.microsoft.com/research/downloads/details/345a52c3-fe44-4045-94b4-4b26a93a907c/details.aspx

-Matt

Automatic Image Stitching in Deep Zoom Composer

Mon, 11 Aug 2008 22:55:12 GMT

Deep Zoom is a terrific new feature that is available in Silverlight 2.0. The best way to describe Deep Zoom is that it allows Silverlight to natively load arbitrarily large images. To get an idea of what Deep Zoom can do, please take a look at the initial results of the Yosemite Extreme Panoramic Imaging Project.

But, what I really want to talk about today is the Deep Zoom authoring tool - Deep Zoom Composer (DZC). The latest beta of DZC offers a feature that my group developed - automatic image stitching. While we've shipped this in the past in other products like Windows Live Photo Gallery, this version of the stitcher has some significant improvements. The first improvement is a better technique for automatically finding matching control points between images. The second is better blending between images, especially when there are significant exposure differences.

Control points are very familiar to anyone who has used manually operated stitching software. For those who haven't used such software, the basic work flow is that you open one image, click on a point in that image that you think will likely appear in another overlapping image, then open that other image and click on the corresponding point. You do this over and over again until all images that comprise your panorama have points matching into other images. Recently computer vision algorithms have been invented that can automatically perform this function. The most well known of these algorithms is the Scale-invariant feature transform (SIFT). SIFT works so well that it has largely replaced manual control point editing and there are several automatic stitchers available, such as Autostitch, that take advantage of SIFT.

SIFT however isn't perfect at matching images and sometimes misses matches or generates erroneous matches. Last year, Simon Winder and Matthew Brown, two researchers in our group, tackled the problem of designing a better matching technique and published their work in a paper titled Learning Local Image Descriptors. This paper is meant for a technical audience, so I will try to summarize the findings here. SIFT was designed by smart people testing different ideas on a large number of test images. The new technique instead uses a much much larger number of test images (produced by the Photosynth pipeline), and uses mathematical optimization techniques to automatically 'learn' a new control point matcher. The net result is that the new technique produces 1/3 the number of false matches when compared to SIFT on our test data. And ultimately for users of DZC this will lead to a higher success rate for your stitching projects.

It is personally very exciting for me to see Microsoft Research (MSR) results like these moving from academic journals to actual usable products so quickly. The MSR computer vision researchers have long been active in academic circles, but now are finding more and more outlets in Microsoft products as well. There are other examples such as Photosynth and HD View, where preview software was available very near to technical publication^{(1, 2)}. I look forward to this trend continuing.

That said, please give the new Deep Zoom Composer a spin. You can learn more about it and download it from:

http://blogs.msdn.com/expression/archive/2008/08/01/download-the-new-deep-zoom-composer-preview.aspx

Note DZC is still Beta software and there are a few rough edges such as limits on output size.

-Matt Uyttendaele

⁽¹⁾ Photosynth was published at Siggraph 2006 as "Photo tourism: Exploring photo collections in 3D" in August and made available publicly in November.

⁽²⁾ HD View was published at Siggraph 2007 as "Capturing and Viewing Gigapixel Images" and made available in conjunction with the conference.

Petra 360 on National Geographic

Fri, 08 Aug 2008 18:48:49 GMT

National Geographic's web site has a great gallery of 360 panoramas from Petra, Jordan. They have a slide show of 6 different panoramas from around the ancient city. For each of these you can click on "View in HD" to get the full-screen HD View experience. Check it out here: http://www.nationalgeographic.com/history/ancient/petra-360.html

-Matt

High Dynamic Range in HD View 3

Tue, 05 Aug 2008 23:46:24 GMT

This post is all about high dynamic range imaging (HDRI). As I mentioned in the Beta 3 announcement, HD View is the best way, that we are aware of, to interact with HDR images over the web. We take advantage of three recent advances to make this happen. They are HD Photo compression technology, the PC's graphic processing unit (GPU), and high contrast displays. I will describe each of these things, but first, a quick intro to high dynamic range imaging (HDRI).

What is HDRI?

image courtesy of Mark Fairchild

Light in the world has a very large range of intensities. In the figure above, you can see a 100 million to 1 range (alternatively 26 stops) in intensity from starlight to direct sunlight. An HDR image is one that contains a measure of the intensity of light in a scene. At each pixel, it records one of the numbers on the scale in the figure above. This means that an HDR image can represent a scene that has a very large intensity difference between the shadows and the highlights. Cameras today can't directly record scenes that have such an extreme range. So to create these images, photographers stack multiple exposures together, one correctly exposed for the shadows, one for the mid-tones, one for the highlights, and so on. These separate exposures are then merged by software^{(1, 2)} to produce an HDR image. This is quite a different representation than a JPEG image. In terms of range, a JPEG can represent only a small portion of the possible range of intensities in a scene, which makes JPEG a low dynamic range (LDR) format. This is why in a backlit situation your camera can produce a good JPEG of the foreground object or one of the bright background, but not both at once. It turns out that most computer displays are also LDR, so even if a JPEG could store a greater range, you wouldn't be able to see it on a standard monitor. JPEGs also differ from HDR in the way they store color information, but that is a topic for my next post.

So why are there so many HDR images stored as JPEGs on flickr? That is thanks to a technique known as tone-mapping. In order to make an HDR image display on an LDR monitor, tone-mapping algorithms compress the wide range of intensities in the HDR image to fit in the narrow range of the monitor. There have been terrific advances in tone-mapping, but it is a very hard problem to make tone-mapped images look natural. In order to compress the range, dark parts of the scene need to be brightened to the point that they have similar intensity to bright parts. This can lead to an unnatural look, because as a human observer of this image, you expect bright objects to look a certain way when compared to dark objects, but the tone-mapping process has changed the tonal relationship between these objects. I've heard many people say that they "don't like the HDR effect," but what I think they object to is the effect of the tone-mapping process. It is important to separate these two things. HDR images represent the intensities of light in a scene. Tone-mappers make HDR images presentable on the computer monitors available today.

HD Photo and JPEG XR

Now onto how we are supporting HDR in HD View. On the High Dynamic Range page of the HD View web site we include the following diagram that describes our HDR pipeline.

The pipeline starts with a camera capturing one or more RAW images. A single RAW image by itself contains a greater dynamic range than JPEG can store, and as described above, if multiple bracketed RAW images are captured, these can be merged into an HDR image. If we want to deliver this over the web, today that would involve the tone-mapping process described above followed by storing the result as a JPEG. In our pipeline, we instead store the extended dynamic range image directly in a new image format called HD Photo. There are other file formats available today that could also be used such as, .HDR, OpenEXR, and TIFF. But, none of these offer the sophisticated compression algorithms that HD Photo does. This compression allows us to effectively transfer HDR images over the web. The file sizes of these other HDR file formats are simply too large for interactive web access. As an example, the images on our HDR Survey page were downloaded from Mark Fairchild's site in OpenEXR format and converted to the HD Photo format. The OpenEXR files range in size from 35MB to 46MB, by comparison after conversion to HD Photo the images are about 1/10th the size (3.6MB to 6MB).

HD Photo started as a Microsoft Research project. It was then refined and further developed by the Microsoft Codec Group. Microsoft subsequently presented HD Photo to the JPEG standards body, and that organization has decided to make HD Photo a standard that will be known as JPEG XR. I am very much looking forward to the time when JPEG XR emerges from the standardization process and is in ubiquitous use. For now though, you can start experimenting with the new technology in the form of HD Photo. The HD View team has provided a simple command line tool and a Photoshop plugin that you can use to convert your RAW and HDR images to the HD Photo format that HD View requires for streaming over the web. HD Photo has lots of other great features beyond HDR, please see Bill Crow's blog for many more details.

GPUs

The graphics processing unit (GPU) of almost every modern PC now has HDR capabilities. For example, a new $300 PC from Dell uses the Intel 3100 GPU. This chip can load and process floating point images which is the same representation that HD Photo uses and is a great representation for HDRI⁽³⁾. Of course higher end GPUs from NVidia and ATI have these capabilities as well, but the point is that even entry level PCs can now process HDR imagery in real time. GPUs have this capability because in order to generate realistic looking 3D graphics, game designers need to simulate real world lighting. Thus, today's most realistic looking games are in fact doing HDR. GPUs also have the ability to run 'shader' programs. These shaders, as they are known, can run fairly sophisticated image processing algorithms at interactive rates. We use these capabilities to perform real time tone-adjustment on HDR images.

Displaying HDR

HDR displays are coming but they aren't available just yet. When they become available our HD View pipeline won't need to change and any content that you've created for HD View will look great on them. But for the foreseeable future we will also have LDR displays. So now that we've delivered an HDR image all the way to an LDR monitor what do we do with it? Remember that HD View is an interactive image viewer. So, we can create the ideal tone adjustment given the end-users display, the portion of the image they are currently viewing and any intents we let them express. This is quite different from the tone-mapped images you see today. Those images are processed assuming that they will be viewed full-frame on an LDR monitor or on paper. This can lead to the distorted relationship between tonal values I described above. It can also lead to a flattening or loss of contrast in the image. In HD View we instead do tone adjustment on the fly as the user views the image. For example, when zoomed all the way out on an image we might see the shadows be truly dark, but as we zoom into those dark areas HD View will auto-expose to show previously indiscernible details (the ability to retain those details in the shadows is thanks to HD Photo). In our pipeline, we've retained all of the dynamic range of the original photographs and we have the ability to run shaders, so we offer users a variety of different ways to do tone-adjustment on the fly. These are accessed via the toolbar in the upper right of the window and are summarized in the table below. For some screen shots of these modes in effect see our help page.

	Perform no automatic tone-adjustment. This is still an interesting mode for extended bit depth images because the wide-gamut color support is still in effect.
	Auto adjust the exposure value depending on what is in view. This mode is very similar to the auto gain in a camera.
	Automatically compress or expand the dynamic range. We use what is known as a global tone map operator. The adjustment curve is continuously updated depending on what is in view, it is very similar to that published by Erik Reinhard. The difference being that we transition to a contrast enhancement curve for low contrast regions.

Camera for the Web

To wrap it up, I have described a pipeline that lets you maintain all of the light information that was captured from the original scene. A common term used to describe this is a 'scene-referred' workflow. In the past scene-referred workflows have been advocated for the image editing process but not for delivery over the web. For web delivery conversion to JPEG is the standard. Here we are allowing scene-referred imagery to travel all the way to the end user's monitor. One way that we like to think of this is that HD View is a camera for the web. It is performing a very similar task to camera hardware today which is reading scene-referred rays of light and generating the best possible JPEG image that it can. We instead are enabling that scene-referred data to travel over the web. The HD View "camera" can then generate the best possible photograph using additional information that is only available at the end of this pipeline. This includes taking into account the monitor being used, the user's current pan/zoom interaction, and any additional settings that we want to expose to the user. Currently these additional settings are the three tone-adjustments listed above. However, we can see expanding this to any of the settings that you find in hardware cameras today. Beyond that, we are running on powerful PC hardware, so our software camera can perform image processing that is just not feasible in your point and shoot camera.

Try it out and we always appreciate feedback.

-Matt Uyttendaele

⁽¹⁾I used Photoshop's "Merge to HDR" for these examples on our site.

⁽²⁾Wide field-of-view panoramic images are also typically HDR. In the Capturing and Viewing Gigapixel Images paper, a pipeline to produce HDR gigapixel images is described, PT Gui Pro and Autopano Pro both have the ability to create HDR panoramas.

⁽³⁾See figure 3. in Greg Ward's HDR encodings paper.

Announcing HD View Beta 3

Fri, 25 Jul 2008 03:42:03 GMT

We are very pleased to announce that a new version of HD View is available on our web site. There are a lot of great new features:

Support for 16bit/channel and 32bit/channel images. In other words, High Dynamic Range (HDR) images can now be efficiently streamed over the web to the HD View plugin. We offer 3 different ways of interacting with HDR images - no auto-expose, auto-expose via gain only (like a camera), or auto-tone-map. In each 'auto' case the parameters are recomputed on the fly depending on what is in view.
Now fully color managed. HD View source files can have embedded color profiles, and monitor profiles are respected. In conjunction with the 16 bit and 32 bit support, this means that wide-gamut images can be effectively delivered over the web.
A new lens - the Fisheye. This is accessed via the rightmost button in the toolbar. See screenshot below for Fisheye in action.
KML-PhotoOverlay tiled image representations can now be directly loaded.
XBox controller as an input device.
3DConnexion as an input device.
Easy to use embed code for adding HD View to your blog.
A bunch of bug fixes (a lot of these based on your feedback - thanks!). A few of the things we addressed:
- Firefox3 issues
- Embedding in PowerPoint presentations
- Now works with IE Quick Tabs.

The first two bullet items are the most significant. HD View 3 (and associated tools) represents the first system to efficiently deliver images over the web with no limitations on dimensions (gigapixels and beyond), dynamic range (32bit/channel support), or color gamut (ProPhoto or any other wide-gamut profile support).

There is a lot to cover here, so over the next few days I'll dive into several of these bullet points in more detail. For now please visit the updated HD View site and try it out on some HDR content. To install, follow this link: http://research.microsoft.com/ivm/hdview/HDInstall.htm and click the install button. Note: HD View requires Firefox, IE, or Safari on the Windows OS.

After installing, check out the new HDR content:

http://research.microsoft.com/ivm/hdview/HDRExamples.htm

http://research.microsoft.com/ivm/hdview/HDRSurvey.htm

-Matt Uyttendaele

Fun with Fisheye mode, one of the great new features in HD View 3.

(shown here on xRez's Millennium Park, click images to go to the HD View)

Yosemite Extreme Pano Video

Wed, 09 Jul 2008 21:13:38 GMT

The Microsoft Pro Photo Summit is going on this week. Scoble is doing some interviews. He just posted a good one with Greg and Eric from xRez speaking about the Extreme Pano project I described in my previous post. Greg demos some of the initial stitching results streaming onto a Microsoft Surface.

Yosemite Extreme Pano Project

Wed, 04 Jun 2008 00:37:01 GMT

This past weekend several of us on the HD View team participated in the Yosemite Extreme Panoramic Imaging Project. The project was the brainchild of Eric Hanson and Greg Downing of xRez and Greg Stock, the staff geologist at Yosemite. Greg is studying rockfall behavior on the walls in the park. Today when a rockfall happens it can be difficult to piece together, from the debris, where the fall came from. Having 'before' pictures of all of the Yosemite Valley walls would give Greg a powerful rockfall forensic tool. Thus was born the project. Greg and Eric wrote up a proposal, got out their Rolodex, and rounded up sponsors and volunteer photographers to make this happen.

The plan that the xRez team hatched went beyond simply taking high resolution photographs of each wall in succession. Rather, they wanted to photograph the entire valley in as small a time window as possible. Doing this means that all of the images would have very similar lighting, thus any composite image that is generated will be more seamless. The goal is to create a seamless mosaic of all of the valley walls. Because the source images are shot from multiple vantage points, the result will be a 'multi-perspective' mosiac. To do this, xRez will project the images onto a 3D model of the valley and re-render the images using a virtual flying 'slit-scan' camera. I did a quick web search for examples of this technique. I didn't find a good example for 3D terrain, but I did find some of street-side scenes:

Capturing all of these images simultaneously required lots of gear and lots of bodies. xRez got 70 photographers to donate their time last weekend (Peter Duke has a great Flickr set of some of the Yosemite shooters). They also rounded up a good set of sponsors. Canon loaned G9 cameras, Bogen loaned tripods, Lowepro donated their Flipside 300 packs and Gigapan loaned their robotic pan tilt heads. For the shoot each G9 was outfitted with a Canon TC-DC58C tele-convertor and a home made lens hood (see Gavin Farrell's shots of the final setup here and lineup of packs here).

The Photosynth team at Microsoft was happy to help out with financial sponsorship and members of my team flew down with our large panorama rig to man one of the 15 shooting locations. Most of the photographers hiked into their locations. Fortunately, the gear I described above was nice and portable. I believe that the longest hike was done by the team that went up Half Dome. All of the photographers were in place at 1PM on Sat., May 31, and everyone started to shoot simultaneously.

Our rig, pictured to the left, didn't lend itself to hiking, so we drove it up to Glacier Point. Some technical details of our gear: Meade telescope donated a telescope mount from which we removed the optical tube. We put in its place a Canon 1Ds Mark III and a 100-400 zoom lens. This setup was controlled by a laptop.

We parked ourselves at Glacier Point for 1.5 hours. (I apologize to the tourists there that day for whom we denied the sweet corner spot) Over that time, we shot 680 x 22MPixel images at 400mm. Accounting for some overlap between images, this should result in a 10 Gigapixel composite. Since this is just one of 15 locations, the final result of everybody's photos should yield many 10s of gigapixels.

Everyone converged back at camp later that day for a nice catered dinner and keg by the campfire. From the discussion at dinner, all of the gear performed very well. It sounds like there might be 100% success rate for the shoot! Kudos go out to xRez for managing the logistics of this so well and Gigapan for the reliability of their pre-release units.

I'm really looking forward to the final product which should be ready sometime in late summer. So stay tuned to the project page here. Also check out the xRez Yosemite Shoot Flickr group for photos of the weekend.

-Matt

Pacific Northwest Panoramas

Thu, 27 Mar 2008 16:11:46 GMT

Over on the Windows Vista Blog, Brandon LeBlanc has posted a really good set of panoramic images from around the Pacific Northwest. This might be the best example yet of what the stitching technology in Windows Live Photo Gallery is capable of.

See the pictures here: http://windowsvistablog.com/blogs/windowsexperience/archive/2008/03/25/panoramic-stitches-from-around-the-pacific-northwest.aspx

For more info on stitching in Windows Live Photo Gallery see our previous blog entry. To make your own panoramic images, try out Windows Live Photo Gallery yourself: http://get.live.com (! it's free !).

Brandon is threatening to create a video tutorial about his shooting technique - we'll be sure to link to it when it becomes available.

-Matt

Amazing new Gigapixel imagery from xRez

Tue, 25 Mar 2008 16:58:57 GMT

Our friends over at xRez have written an interesting article that appears today in vrmag. The article describes a very large photography project that xRez did for move.com (a real estate web site). As Greg recounts in the article, move.com hired them to shoot Gigapixel "slice of life" images from around the country. All told the project yielded 1.08 petapixels of new imagery.

See the article here: http://www.vrmag.org/vartist/spotlight/AN_INCREDIBLE_XREZ_PRODUCTION.html
Go directly to the xRez gallery here: http://www.xrez.com/gallery/move/xRez_move.html

The links above contain some amazing new imagery. The "slice of life" aspect of this photography really comes across in this image of Gas Works Park in Seattle. Another thing that I take from this article is that xRez has demonstrated that rapid production of large immersive imagery is now feasible. One field that I think could benefit from this is Internet based journalism. The Internet news site that can publish this type of imagery in near real-time would be much better able to tell certain stories. xRez's shot of the Minneapolis I35W bridge taken 3 weeks before it collapsed brings this point home.

It turns out that this is just a small sampling of the imagery captured for move.com. Unfortunately move.com has not yet published the remainder. A quick search yields this article from July 2007 where "Gigapixel Neighborhood Views" were announced. I'm hoping to see these show up on their realtor.com site sometime in the near future.

-Matt

Hourly HD View at European Ski Resorts

Sun, 16 Mar 2008 06:14:51 GMT

The French company Shaktiware has just launched two camera rigs that automatically generate a 120MP HD View every hour. Francois from Shaktiware tells me that the robot camera scans the scene in about 2 minutes. This is followed by an automatic process that assembles the multiple source images and uploads the result to their servers.

visit the Coll Blanc resort in Andorra: http://www.neos360.com/saetde/collblanc360/ or (access HD View directly)
see a 360 view of the Punta Bagna gondola at Valfrejus ski resort in France: http://www.neos360.com/valfrejus/punta_bagna/ or (access HD View directly). You can also read the description on the Valfrejus resort site here (en Français).

I'm hoping to see more cameras from Shaktiware come online soon. If you're in that part of the world it would be fun to send us a message via the camera. Did anyone see the marriage proposal done on the time lapse camera at the Apple store in Manhattan (see here and here). Seems like you could do something similar with this setup - hold up a different sign at each camera position.

-Matt

HD View at an NFL game

Tue, 22 Jan 2008 17:55:20 GMT

Congrats to the Patriots and Giants for making it to the Super Bowl. But I'm going to rewind two weeks to relish the time that our local Seattle Seahawks were still in the playoffs. The following pictures were shot by Seattle photographer Bradford Bohonus and he tells me that this is the first time VRs have been shot on the field at an NFL game. Here is some more info from Bradford about how he shot:

"I shot it with the new Nikon D300 using 10.5mm on a monopod (tripods are not allowed on the field during a game). With some of the shots, I held the monopod over my head for a higher perspective (about 12ft)."

You can check out the HD View at: http://www.bohonus.com/pages/ms_hdview.php scroll down to the NFL Playoffs thumbnail. You can also directly access the full-screen version here: http://www.bohonus.com/hdview_pans/hdviewer.php?hdvr=seahawks I really enjoyed browsing this imagery, from the play on the field, to the scoreboard, to the skyscrapers in Seattle, to the fans, to Taima the Hawk - it's all there in this one HD View.

Bradford has also posted some other VRs from the game in Flash and QTVR format here: http://www.vrseattle.com/pages/browse.php?cat_id=912

-Matt

MSNBC uses HD View to cover the Detroit Auto Show

Tue, 15 Jan 2008 07:23:22 GMT

Today MSNBC launched a great HD View collection from the North American International Auto Show. The collection is being promoted as HD Images on the MSNBC front page or you can click here (http://www.msnbc.msn.com/id/22653887) to go directly to the slide show. John Brecher of MSNBC shot the source material which required some panoramic stitching. The media team at MSNBC then used the Windows Live Photo Gallery stitching technology in order to create the final HD View experiences.

Make sure to click on the previous/next arrows below the HD View pane to see each of the scenes. I just spent some time exploring them and I felt like I got a really good feel for the show. Some of my favorites are:

The Lexus lounge. HD View lets you zoom far enough in that you can read the small billboard describing the new Prius Plug-in Hybrid.
I enjoyed exploring the exposed chassis from both the Chevrolet Corvette ZR-1, and the Subaru Forester. In both cases you could probably zoom into more details in the HD View than if you were actually at the show.
The crowd scenes of Ford unveiling several new vehicles in the Cobo Arena and the scenes of journalists milling about the floor. Remember these HD View scenes are composed of lots of source images. I like these two examples because they show off how well the WLPG stitcher automatically seams around moving subjects when assembling the sources images.

-Matt

HD View meets Halo3

Mon, 26 Nov 2007 15:39:33 GMT

I've long been trying to figure out how to get Master Chief to promote HD View. Lucky for me, Torgeir Holm has done it using the screen shot feature on Bungie.net. He uploaded 33 screen shots of a 360 environment and stitched them in PTGui (image below), then using Photoshop added 13 more zoom-in screen shots for the more interesting details.

The HDView results can be found here: http://www.katode.no/egz/sandbox/lastresort/ , http://www.katode.no/egz/sandbox/hornet/

Bungie stores a rolling buffer of a user's last 40 screen shots. For the time being, you can still see some of the original screenshots here: http://www.bungie.net/stats/halo3/screenshots.aspx?page=0&gamertag=Doglet , http://www.bungie.net/stats/halo3/screenshots.aspx?mode=pinned&gamertag=Doglet

-Matt

PTGui used to stitch Halo3 screenshots

Windows Live Photo Gallery Image Stitching

Mon, 12 Nov 2007 19:59:32 GMT

Windows Live Photo Gallery shipped this week. You can download it for free from http://get.live.com The panoramic stitching component of the product was developed by the Interactive Visual Media group in Microsoft Research. This isn't the first time that Microsoft has shipped image stitching. So, some of the core technology is the same as what we've previously delivered, but, we have added some powerful new capabilities to the latest release. This stitcher has a very simple user interface, multi-select images and choose an output file name. It works very reliably if the photos are well shot (sufficient overlap between images and minor exposure differences). This product should make image stitching accessible to a very large audience of users.

A lot of people have asked what is the relationship between this and Photosynth. Both technologies were developed by the Interactive Visual Media Group. Several other blogs have already covered the details of how these technologies relate - see here and here.

Fully Automatic Image Matching

To use image stitching in WLPG you simply select a group of photos that have some overlap, then use the right mouse button to get the menu pictured to the right, and select "Create panoramic photo...". In this example I selected 13 input images and generated a 360 panorama, which you can view above as a flat jpg, or here as an HD View.

Over the past few years, we have refined the algorithm that performs automatic image matching. A lot of the recent work in automatically matching images was inspired by the Recognising Panoramas paper that Matthew Brown, a grad student at University of British Columbia, published in 2003. We were certainly inspired by his work and subsequently hired Matthew. His efforts combined with some technology that our group had previously developed (Szeliski & Shum, Siggraph '97) allowed us to build the automatic photo stitching feature that appeared in 2004, in the Digital Image Suite family of products. Subsequently, several other automatic stitching applications have also used Matthew Brown's work as the basis for their image matching, these include Autopano Pro, Autostitch, and Autopano-SIFT.

Intelligent Seam Selection

Matching images is only part of the problem in creating a good looking image stitch. Another important component is how the images are composed together. In WLPG we added intelligent seam selection technology. For this problem we have built on another research effort, Interactive Digital Photomontage, that was jointly developed by Microsoft Research and University of Washington. This paper outlines a general framework for doing many different kinds of image composition. In our application, panoramic stitching, this technology provides two things. First, it finds the 'most invisible' place to cut between two images. For example if a moving car is in the scene, the transition between images would not occur through the car, rather it would occur in the static parts of the images. The second thing is that even after finding the ideal transition between images there may be some remaining differences still visible at the seam line. These remaining differences are intelligently diffused through the composite image so that the seam line effectively disappears.

This isn't the first application of Interactive Digital Photomontage that we have delivered. Back in 2005 we posted Group Shot on our Research downloads page. This is a fun and useful app that you can use to select your favorite parts in each shot of a series and it will automatically build a composite image. This technology has also shown up in other products. Aseem Agarwala, the UW grad student who did this work, was hired by Adobe. Among other things he is working on, he helped ship this as the Auto-Blend Layers feature in Adobe Photoshop CS3.

HD View Compatibility

We haven't yet gotten seamless HD View creation in the Photo Gallery (then again HD View is still a research prototype). However, there is a very handy feature of WLPG that the HD View tools now leverage. When creating very wide-field-of-view panoramas, some distortion is necessarily introduced in order to get the panorama to print as a flat image. This is akin to the distortion you see when the globe is unwrapped as a flat map or the distortion you see in a fisheye lens. One noticeable artifact in both these cases is that straight lines in the world become curved in the flat image. Ideally when viewing these images in an interactive viewer, like HD View, this distortion would be removed as you zoom into the image. In order to do this the stitching application has to communicate with the viewer a few bits of information about how the projection was done. We have added this information as metadata in the stitched panoramas that WLPG generates. The details are outlined here. This details page also describes how the WLPG metadata can be mapped to the tags defined in the PhotoOverlay section of the recent 2.2 update of KML.

I generated two different panoramic images to show off the photo stitching feature. I then used the hdmake tool to read the WLPG metadata and create an HD View web page. First are some old hand (not tripod) shot images I had of the climbing pinnacle at REI Seattle. I selected the 8Megapixel source images (12 of them) in WLPG and did 'Create panoramic photo...'. I then saved the result as a jpg file when prompted to save (note in the current WLPG the result must be saved as jpeg in order to have the pano metadata). Next to convert the jpg result to an HD View web page, I used the hdmake command line utility as follows:

hdmake -src stitch.jpg -dst pinnacle 0.8 -xmlwlpg pinnacle.xml -html pinnacle.htm

Notice the -xmlwlpg argument. This tells hdmake to use the WLPG metadata in order to create the HD View xml manifest. I uploaded the generated files to our server and you can view them here. I also created a version that doesn't use the metadata, so HD View just treats it as a flat image, you can see that version here. Notice the difference? The first link feels much more immersive.

The big news around Microsoft Research Redmond today is that we are moving into a great new building. So, for my second result, I took 13 images of the lobby this morning with a 16MPix Canon 1DS MarkII. These were automatically stitched using WLPG and I've posted the HD View result here. Notice that WLPG appropriately communicated to HD View that this was a 360 environment.

I encourage everyone to download and try out the new Windows Live Photo Gallery. In addition to the panoramic stitching feature it has a host of other capabilities. Some of my favorites are the import wizard, the improved Fix functionality, and the upload to web services.

As always we appreciate feedback on any of this.

-Matt

Marc Levoy interviewed on Scoble

Thu, 25 Oct 2007 19:57:45 GMT

This is a great overview of some of the work shown at Siggraph in recent years. For those of you interested in the technology of photography, this is definitely a worthwhile watch: http://www.podtech.net/scobleshow/technology/1663/advanced-photographic-research-at-stanford-with-prof-marc-levoy Marc walks through the state of the art in "Computational Photography". Marc is a professor at Stanford, and he talks about the great research being done at Stanford, like digitally re-focusing images, the multi-camera array, and the Digital Michelangelo Project.

It was also nice to see him mention some of the work being done at Microsoft Research. In the first 10 minutes of the video MSR is mentioned several times including a reference to Image Deblurring with Blurred/Noisy Image Pairs, one of my favorite paper's from this year's Siggraph conference. I was also very happy to see him using HD View (at 36:54 in the video) on his own personal photos.

-Matt

Photoshop Plugin re-release

Wed, 24 Oct 2007 18:25:08 GMT

Thanks to all who tried the Plugin last week. Due to the feedback we are re-releasing it with a few bug fixes. The new one is available at the same place: http://research.microsoft.com/research/downloads/Details/345a52c3-fe44-4045-94b4-4b26a93a907c/Details.aspx

1) The plugin wasn't being enabled for images larger than 30K on a side. That kinda defeats the point. We want to see huge images! So, we set the right flags and PS now lights us up for images up to 300K on a side.

2) Images with additional channels beyond RGB weren't being handled correctly (not even close to correctly). This is now fixed.

3) Finally the most common complaint was hr=0x8007000E, which means out of memory. It turns out that Photoshop + the plugin together were overall trying to use too much memory. A little digging turned up that this isn't uncommon for plugins. See the "Out of memory errors" section of this Adobe Knowledge Base article: (http://kb.adobe.com/selfservice/viewContent.do?externalId=320005)

We've reworked our memory usage and now I don't think anyone should see an issue under normal PS memory usage. If you have your Photoshop Memory Usage very high (above 90%) or if Photoshop has been running for a long time and memory has gotten fragmented, then you may continue to see this. In that case we no longer generate the cryptic 0x8007000E but instead provide the following friendly dialog:

Questions, comments and issues, please use the forum: http://forums.microsoft.com/MSR/ShowForum.aspx?ForumID=1609&SiteID=37

-Matt

Photoshop Export Plugin Available

Fri, 12 Oct 2007 19:02:10 GMT

We've just made available an export plugin for Photoshop that generates HD View content. Below is a screen capture of the plugin in action. The exporter generates an HD View tile set from an RGB/8 image opened in Photoshop. It also creates the xml manifest necessary for HD View and an example web page. The download site is here, and a more detailed description of the plugin is available here. Note that this is currently only available for Windows and we recommend Photoshop 7 or above.

In the same install package we've also updated the hdmake tool. The updates to hdmake have been mainly driven by feature requests from users.

Bernhard Vogl recently posted about his "Sliced Gigapixel Panorama Rendering". We've made a corresponding feature available in hdmake. You can now specify a 'sliced' source image set instead of a single source image. See the -srcgrid argument in the help.
We've also addressed most of the requests made by george5558 on the HD view forum. (a) hdmake will now generate a web page, (b) if lossless is requested then the convert lossless pyramid step is no longer done, (c) a few more stats are output during the conversion.

Thanks for the good suggestions and do keep them coming.

-Matt

New example web pages that use HD View

Wed, 05 Sep 2007 05:01:51 GMT

Two good blog posts this week.

First Johannes Kebeck from the Virtual Earth team shows how to put HD View in a Virtual Earth info box. See the screen shot below scrapped from Johannes' blog. Click on the image to get all of the details. His example shows merging Birds Eye with an HD View he created of the Microsoft offices in London. I found it to be a very effective way of adding additional detail to VE.

Next Bernhard Vogl shows how to add a drop down list to change the view to a particular point of interest.

Both blog authors are encouraging people to download and extend their examples.

-Matt

A quick comparison of HD View v. Google Earth 4.2 Photo Viewing

Tue, 04 Sep 2007 05:54:56 GMT

When the Google Earth team announced Gigapixel and panoramic support last week I was eager to try it out. So, this weekend I spent some time exploring the new Gigapxl Photos layer. As I've mentioned before, Graham Flint from the Gigapxl project was one of our inspirations to begin creating very large immersive images, so it was great to finally be able to explore his imagery. It was also cool to hear that Michael Jones, Google Earth CTO, took some of the pictures. That certainly increases his street cred.

After playing around with the provided Gigapxl content I wanted to try it out on some of my own content. I followed the lead of the Digital Urban blog and used Gigapan to upload a panorama. I hadn't visited the Gigapan site in a few weeks and it looks like they've recently done a big update. I really like their new panorama upload and sharing service. And the snapshot feature on their site is very nicely implemented - alas they had the virtualphoto tag concept already implemented before I blogged about it last week. I'll definitely be in line to buy the Gigapan hardware when it comes out later this year. I'm looking forward to creating some great HD View content with it.

I uploaded a large 360 degree panorama of Mt. Rainier to the Gigapan site and used their tool to geocode its position. This generated a kml file which I could then open with Google Earth. I also created an HD View page for the same panorama. Below is a table summarizing what I consider to be some key features in a panoramic image viewer, and how HD View and Google Earth stack up.

Feature	Google Earth 4.2 Photo Viewing	HD View

Integrated in a world browsing application	YES	NO
Use it on your own web site	NO	YES
Handles arbitrarily large images	YES	YES
Supports panoramic images	YES *	YES
Uses GPU acceleration	YES	YES
Dynamically change projection depending on zoom level	NO	YES
Dynamic tone adjustment	NO	YES
Uses HD Photo (JPEG XR) for up to 50% faster download times	NO	YES
Has a blog dedicated to immersive images	:(	:)

But I'm clearly biased, so I won't do a full review. Rather I'll provide three links that you can try for exploring my Mt. Rainier image. Let me know what you think.

(1) Gigapan Flash Viewer, (2) from Gigapan page select View in Google Earth 4.2, (3) HD View

-Matt

*: I exchanged a few emails with Gigapan and was able to get the kml functioning somewhat. But, I'm seeing the same issues with Google Earth as the Digital Urban blog and subsequent commenters did. I'd be happy to post additional fixes to this kml file if someone can provide further improvements.

HD View Virtual Photography

Sun, 19 Aug 2007 05:32:20 GMT

During our Siggraph talk one of the key messages was that we think of HD View as a camera for the web. The slides for the talk are now posted on the Capturing and Viewing Gigapixel Images web page. I've pasted a copy of slide 9 below.

Notice that HD View lets you do many of the things you'd expect from your camera, point it in any direction you want, zoom in and out and change the exposure. Since HD View is actually software running on a powerful computer, we can also do some things that are difficult in a real camera. One example is to warp the sensor which has the effect of changing from a perspective capture to an equirectangular capture. The analogue of this in hardware would be to change the lens in your camera from a zoom lens to a fisheye lens. Another example is that we can do more sophisticated tone adjustment, unlike a hardware camera where you set a global shutter speed for the scene, we can expose the scene differently in different regions.

So, I decided to get my HD View camera out and take some pictures. I took 9 pictures and posted them to the VirtualPhoto album on this space. And also to our Flickr page, where I tagged all the photos virtualphoto. In each photo on the Flickr page I've added a comment with a permalink back to the original HD View. I had a few thoughts as I was doing this. First, in the span of 10 minutes, I took pictures of a blustery night on the Mall in Washington DC, a picture on a mountaineering trip up Mt. Rainier, a picture on a beautiful afternoon in Vienna and pictures of the new Olympic Stadium in Beijing. It was the next best thing to being there. Second, I was able to compose shots that I never noticed when I took the original pictures. For example this reflection of the Capitol. It is a zoom in on a much larger HD View experience. I took the original snapshots that made up the larger HD View but I didn't see this image at the time. I was really happy to be able to virtually take it after the fact.

If you find great photos within the HD View collection, or within Zoomify, QTVR, or other immersive imagery (of course please do respect copyrighted imagery), then I encourage you to add them to Flickr with the virtualphoto tag.

-Matt

Photo Album: Blog Images

Wed, 24 Oct 2007 18:25:06 GMT

Blog Images

More Photos...

Photo Album: VirtualPhoto

Tue, 14 Aug 2007 05:40:26 GMT

VirtualPhoto

BeijingStadium

CapitolReflection

CapitolReflection2

Klahanie

MtAdams

OBB

PHSH

WW2Flag

WW2Fountain

Photo Album: Blog Images

Tue, 12 Jun 2007 06:34:04 GMT

Blog Images

Custom List: Links

Wed, 18 Jun 2008 05:07:20 GMT

Links