Pool tests for optimized corner performance on various wide angle lenses underwater

While I was in Bonaire teaching our Digital Master class I had a rare opportunity to do some pool testing of a variety of cameras. The new Seacam D3 housing had just come out, and we had 3 on location in Bonaire, two with D3X cameras and one with a D3 (full frame, aka FX). We also had Seacam housings for Canon 1DsMKIII (full frame) and Nikon D300 and D2X (cropped sensor, aka DX). Since all housings use the same 9" mineral glass Superdome, there was an opportunity to test various lenses, diopters, and port extensions to see if we could determine optimal performance on a lot of different glass in a very short time, with a reasonable degree of standardization.

While on location I posted the results on a wetpixel.com chat that was ongoing regarding the Nikon 14-24 lens for underwater use. To view the whole 7 page thread, and see the animated and sometimes contentious replies to my testing, visit http://wetpixel.com/forums/index.php?showtopic=28444&st=0

Test procedure - Two people sat on the steps of the pool at Captain Don's Habitat, each holding an exposure slate. The one on the left was a step lower and about two feet closer to the lens, and the point of focus was on the type on the center slate. The guy holding the left slate would scoot left or right, depending on wide angle coverage of lens being tested, to assure slate would be situated in lower left corner of frame always. In every test the center slate was sharp, for any of these cameras and lenses can do a rather spectacular job getting the images sharp in the center. I was trying to replicate a multi-dimensional underwater scene, and trying to find what optical elements conspire to give enhanced corner performance. So, my final arbiter of quality was a 100% (Actual Pixels) enlargement of the exposure slate detail in the lower left corner, specifically to see how easily I could read type.

Here is the general set-up I used. The camera was not tripod mounted, but there was a lower step on the pool that I could always put my foot against to let me orient myself pretty accurately from shot to shot. Gratefully, several of the class participants were on hand through both days of testing to change ports, add diopters, and switch out lenses as necessary. Without their help (and their gear) I wouldn't have embraced such an ambitious test regime.



For the most past, I offered no opinions on the testing in the wetpixel post, but in the interest of brevity here are a few general observations on various lenses:

Nikon 14-24 - The first test I did was with a PVL55 at f-5.6 and it looked pretty horrible. Then the next day I tested a variety of different port extensions and found I could get much better performance with the PVL50. I also stopped down to F-8 and discovered pretty nice performance for the full-frame Nikon D3X and 14-24 lens so long as the precise port extension was used and smaller apertures engaged. Amazing the difference 5mm of port extension makes, which goes to show these tests must be very accurate indeed!





Nikon 12-24 on D300 (DX) camera - This lens is often maligned, but in reality it tested very well with no diopter, and extremely well with the addition of a +2 diopter (first shot). Note also the reduction in wide angle coverage once the diopter is installed. More on this angle reduction with diopter later, but the big issue here is probably DX versus FX. The FX lenses are harder to get dialed into optimal corner performance, but on the other hand it takes the additional surface area of a full-frame sensor to get the ultra-high megapixel count and corresponding resolution of the Canon 1DsMKIII and Nikon D3X. Trade-offs are to be expected.





Tokina 10-17 on Nikon D300 - Very nice, as expected, given the general quality fisheye lenses offer in terms of curved field corner performance:



Nikon 16mm on D300 - Likewise stellar, for much the same reason as with Tokina 10-17. While not tested in this series, the Canon 15mm on the 1DsMKIII and 16mm on Nikon D3X should be quite nice as well, particularly with the superdome. I would expect similar corner performance, although DX might reasonably outperform FX here as well.


Canon 16-35mm II on Canon EOS1DsMKIII - As this is my personal go-to lens for a lot of underwater applications, I have extensively tested and come up with a preferred port extension of 57.5mm. The lens tests very favorably compared to both Nikon 14-24 and 17-35 zoom lens on full frame cameras. In fact, the Canon 16-35II significantly trumped Nikon 17-35, but seemed fairly close to 14-24 Nikkor, given perfect port extension. Perhaps head-to-head comparison of these two lenses under stringently controlled conditions makes sense for some other time, as these are the premium wide angle zooms for both Canon and Nikon when used with their respective full frame cameras.

It is still interesting to see how much better it performs at F-11 versus 5.6. I tend to use ISO 200 as a base with this lens so I can use smaller apertures typically:




The Nikon 17-35 zoom on the D3X took the greatest amount of testing, because it looked so very bad after the very first test:

In fact, from that very first test test I was wondering if this lens was the underwater optical equivalent of the Ford Pinto in Ralph Nader's "Unsafe at Any Speed".

However, in the end I had little confidence in my first test and wrote the following to friend and colleague Alex Mustard:
"Alex - I didn't buy that the results could be so bad either. I know 17-35 is better than the tests suggest, so clearly the port extension we assumed was right, is not. After all, until full frame, the 1.5 crop forgave many sins. Plus, I shoot Canon, so have never obsessed over the 17-35 Nikkor. But, now that we need to support that lens with full frame Nikons, the test has to be more scientific. Today I tested 17-35 with PVL 25, 30, 35, 40, 45, 50, and 55; and with PVL 40 (the one that looked best to my eye in the field) with +1, +2, and +4 B+W diopters. I'll analyze and post those results after we wrap our class this afternoon."

Subsequent post:
"Here's the best I could do with Seacam Superdome, PVL45, full shot, 100% center, and 100% lower left corner. Almost ALL of these tests I've been doing are sharp in the center, btw. I've only been testing for ultimate corner resolution. With no diopter, this (PVL 45) is the best of PVL 25, 30, 35, 40, 45, 50, and 55"





it got better in terms of corner performance when I tested with various B+W diopters (+1, +2, and +4) but still not great. Here's the best I could do, with a +4 diopter:


Jeff Hartog (Loftus) then asked me if the 17-35 with +4 is the way to shoot 17-35. I said in reply:

"Jeff - It depends. The +4 costs significant wide angle view to achieve sharper corners. Actually, I'd probably use it without a diopter and go to a higher ISO and smaller aperture to enhance corner performance. The new cameras do so well at higher ISO, that's the direction I'd likely go first.

The 17-35mm with +4 is not wide enough, for me, for the things I'd normally shoot with a lens like that. Plus, the pincushion distortion and chromatic aberration is exacerbated with +4 so I'd probably find another way to shoot without it. But, that's just me because the wide angle view is so important."

With that the question of how diopters decrease wide angle coverage underwater got very spirited. Read thread if you have an interest in the opposing points of view, but if you want the bottom line, it does. obviously and conclusively. Alex Mustard did a great graphic to illustrate the loss of coverage at the wide end when applying an on-lens diopter behind a dome port. His words:

"Just to show how significant the reduced coverage caused by the dioptre was I have overlaid (inverted) the non dioptre shot. I think this shows Stephen's point "the greater the diopter the greater the angle of view reduction - note how many bars of color on color chart are lost as a result of adding diopter" very graphically.



Now I think that there was a little camera movement between these shots - as I had to skew the image slightly, but there is no denying that the dioptre cuts coverage by a large amount.

Alex"

My final comments on the tests:

"As far as the diopter goes, I was certain it had an effect on wide angle coverage and distance depth of focus because I really tried about everything to get optimal performance out of my Canon 17-40mm back when I was shooting that lens on a Canon 1DsMKII. But every time I used the diopter I came away thinking it was not quite wide enough and I couldn’t hold as much detail on a model in the distance. Now that I shoot the 16-35II on a 1DsMKIII, and actually can’t find a slimline 82mm diopter to try, I haven’t been too upset given the marginal success I’d had with diopters anyway.

Yet, despite my intuitive sense that I was losing coverage with the diopters, I didn’t know how significant the effect might be until this pool test. Even the people on the side of the pool watching the test were amazed at how relatively far I had to back up to be able to capture the whole peripheral slate as I added progressively stronger diopters. I think it was probably about 18-inches between the non-diopter shot and the +4. If I was shooting with strobe that would be a full stop in light loss just because of the greater water column between strobe and subject, not to mention coincident resolution decrease.

However, with certain ports, I think the diopter is the only way to go. You (Alex) mention your 4" Subal port and your 17-35. Actually, I wonder if the 17-35 would even focus on the virtual image with a port that small throughout the whole zoom range, so a diopter may be the only way to use it underwater. When our Seacam shooters use the wide port rather than the superdome with Nikon 12-24 or Canon 17-40, we recommend diopters because the virtual image is so much closer. I occasionally use my 24-70 underwater and with a superdome I use no diopter, but it also works quite well with a wide port and a +2 diopter. Actually, enhanced close focus with that lens is probably more important than achieving the widest coverage, so that's usually my preference.

I did not mean any of this as a condemnation of using diopters, for they can be useful with specific ports and specific applications. I really only tested one port, as well as I was able with the time and technology available to me on location, and the results can probably only be interpolated to wide diameter domes of 8 - 9 inches."

As a final analysis, I shot the 17-35 at a wide range of apertures, from F-2.8 through F-22, with no diopter. Around F-11 it started to get nice, and by F-16 was perfectly acceptable. If I were going to use this lens myself I'd probably kick it up to 320 or 400 for a base ISO to get a smaller working aperture, and hope to get to at least F-11. New cameras by both Canon and Nikon handle high ISO very well, so I assume this is a safe and productive solution.