|
OpticalLimits Lens Test FAQ |
OpticalLimits Lens Test FAQ
Q: What is your relationship to the lens manufacturers ?
A: As of now there're no relationships to Canon, Nikon or one of the other manufacturers.
All lenses are contributed by users like you or purchased solely for testing purposes in an
average Joe's shop.
Q: How can I help ?
A: As mentioned the tests are largely a community driven effort. If you're living within
the European Union (EU) you may consider loaning lenses for testing purposes. Have a look
at link just below the system-specific lens test overview (unless the testing queue is
filled).
Q: How do you test these lenses ?
A: The test procedure follows the IMATEST guidelines.
Imatest is a software tool that analyses various image aspects. The testing procedure is actually
quite simple - in principal at least - you're taking sample images of test charts. There're three
variants of charts:
- a MTF test chart primarily consisting of quadrants located in the center and at the borders.
Imatest analyses the blur at the edges of these quadrants.
- a grid chart for the distortion tests
- a plain white chart/screen for vignetting tests
We're using a variety of test charts sizes depending on the specific lens.
Q: Where do you measure the center, borders and extreme borders?
Here's an overview:
Q: What aspects cannot be tested ?
A: The most important aspect that cannot be tested at the moment is contrast. Subjectively a lens
with great contrast but rather mediocre resolution looks more snappy, maybe even sharper than a
low-contrast lens with a high resolution. However, living in the digital age we can boost contrast
(to some degree) via an imaging application today. Beyond contrast there's no quantitative way to classify
the characteristic of the bokeh (out-of-focus blur), the degree of flare and the performance
variations across the focus range.
Q: Are the figures comparable between cameras or different systems ?
It depends on the similarities between the image sensor system. A sensor SYSTEM contains
the image sensor with or without micro-lenses, an IR filter, a low-pass (Moire) filter and
the signal processing. As you can imagine the output quality is largely dependent on the
whole chain on not just on the amount of megapixels. The different output quality between
the Canon EOS 350D and the Olympus E-300 is a good example (despite a 8MP sensor).
The tests are a good guidance for the lens quality as long as you compare the results
WITHIN a test group (e.g. Canon).
Here're a few tendencies:
the Pentax results are a bit steeper (sharper and softer results are more pronounced). The
tangentially resolution is a bit favored due to the extremely weak vertical AA filter.
the Canon results tend to be a bit better at the extreme borders due to the smaller sensor
the Olympus results are comparatively weak due to the aggressive AA filter on the E-300
Q: Why are the quality ratings different from system to system ?
As mentioned above the lens quality is affected by the sensor "system". Every additional
step in the pipeline decreases the output quality, specifically the low-pass filter in
front of the sensor. Assuming you mount the same lens on different system its maximum
resolution will vary according to the max. quality of the sensor system. There're also
evolutions regarding the RAW converter quality so more recent system tests starts can
benefit from this - e.g. Canon/Olympus RAWs are/were converted using ACR 3.2 whereas
Pentax/Sony RAWs are/were converted via ACR 3.7 and there was an increase in converter
quality with ACR 3.4). This must all be taken into account regarding the rating system.
Q: What is the meaning of LW/PH (MTF50) ?
A: LW/PH means "Line widths per picture height" - it's basically a unit for resolution
(not sharpness which is a combination of contrast and resolution) limited by the max.
resolution of the camera's image sensor. You shouldn't take the LW/PH value itself too
seriously because it is dependent on a number of factors. The analyzing tool (Imatest)
is quite vulnerable to the quality of the source material.
The effective resolution is dependent on a number of factors. AA filter of the sensor,
A/D conversion, demosaicing, base sharpening and of course the lens resolution.
However, the "front-"factors are all near-linear. Say, e.g. the AA filter has a factor 0.7 whereas
the base sharpening has a factor of 1.5. So in the end we have a formula like
EffectiveResolution = a * b * c * d * LensResolution
Now the value of a,b,c,d is actually meaningless. You can even add another factor - say 0.5.
From a qualitative perspective this doesn't change anything. The charts will still look
the same - only the numbers will be different.
All PZ sample images are taken
as RAW files and converted via Photoshop ACR (default settings without automatic image
correction and contrast set to 0). If you convert RAWs via other imaging applications
the LW/PH figures will be lower or higher due to the different sharpening & contrast
algorithms. In the future we will probably drop the LW/PHs in favor of a school mark system
which is easier to read. Refer to the reference scale on the left of the MTF charts to
classify the quality.
Here`s a little visual reference:
| Sample portion @ 1100 LW/PH (Nikon) |
Sample portion @ 2200 LW/PH (Nikon) |
 |
 |
Q: What is the error margin for the MTFs ?
A: Well, I am human and humans are not failure-proof by nature. However, the human factor is
has probably quite a bit less impact compared to the sample-to-sample variations which can
reach a worrisome degree.
Regarding the MTF tests the procedure DOES NOT rely on the camera's AF system which tends to
be very unreliable. The principal idea is the take the AF focus distance as a rough guidance
and do several manual focus iterations within a radius of roughly +/-20% around the suggested focus
distance. The images with the highest resolution are then analyzed and presented in the final
MTF chart. From f/5.6 and up the error margin is pretty much negligible. The margin is higher
(because of a more shallow depth-of-field) towards very large apertures which is why the applied
focus bracketing gets more detailed here. My best guess is that the normal error margin increases
to ~5% here. If there's an excessive amount of CAs or purple fringing the MTF analysis will be
affected to an unknown degree. If a lens suffers from field curvature and/or residual aberrations
(see below) this is taken into account - in this case the corners are measured independently
from the center using different reference images. Depending on the degree of these problems
the error margin may increase though.
Q: Will the resolution increase with new sensor related technologies ?
To some degree - yes. Assuming the manufacturers will find a way to get rid of the sensor
low-pass filter there'll be a substantial gain for instance. Some lenses will also benefit
from higher megapixels specifically in the center. However, a lens with a mediocre border
performance will only marginally benefit from such measurements. Assuming optimal conditions
I would guess that some lenses have the potential to go up to 20mp on APS-C but
only with their center portion.
Q: What is the error margin for the distortion figures ?
The error margin regarding the distortion figures should be rather minimal - the analysis
does not rely on a high quality source material. Some lenses have a rather wavy distortion
characteristic which has no influence on the degree of barrel or pincushion distortions so
despite an unpleasant distortion characteristic the generated distortion values by Imatest
may be low. Look for a corresponding comment in the text.
Q: What is the error margin for the vignetting figures ?
For tele lenses (everything beyond ~50mm) the error margin is negligible. Wide-angles
are quite a bit more tricky because they require an evenly lit environment. Expect the
error margin peaking around +/-15% for a 10mm lens. Please note that this aspect
is tested based on straight JPEGs so there're slight dependencies regarding the default
contrast settings on the specific test system.
Q: What is the error margin for chromatic aberrations ?
The CA error margin of the testing procedure isn't really relevant - the main problem
are the CA variations of the lens itself. Look for corresponding comments in the
text.
Q: What is a centering defect ?
A: A centering defect originates in a misaligned lens element or lens group. The result
is a blurry corner, an off-coplanar projection (symmetrical blur) or object edge shadows.
A centering defect is normal to some degree - there's simply no perfect lens out there,
not even a Leica. Generally centering problems are more obvious with large aperture
lenses and they're more frequent with budget (third-party) lenses.
Q: What are residual spherical aberrations ?
This defect is actually a quite evil one. Residual spherical aberrations are focus
shifts when stopping down e.g. if the lens has a max. aperture of f/2.8 and you
stop down to f/8 the focus distance shifts from 0.80cm to 0.90cm. Remember that
you focus at working aperture (f/2.8 in the example above). At long focus distances
the defect will be compensated by the increased depth-of-field but expect focus
errors at closer focus distances. Some ultra-wide and wide angle (<35mm) lenses suffer
from this.
Q: What is the field curvature ?
Imagine that you've a perfectly flat object (e.g. a wall). If you focus using
the center AF sensor of your camera the center focus should be spot on (more or less).
If the lens suffers from field curvature the corners of the wall are projected off
the flat plane. In other words: the plane is bend to a sphere. As a consequence the
corners will have a different focus plane than the center resulting in an out-of-focus
effect. Most ultra-wide and wide angle (<35mm) lenses suffer from this.
Q: Some reviews disappeared - why's that ?
I reserve the right to remove a review if there's sufficient evidence that the tested
sample performed sub-standard.
Q: How can I find out whether my lens has a problems ?
Describing the procedure is easy. The execution is very difficult ...
Plug a couple of news paper pages to a wall with a coverage of at least 120x80cm.
Place a print with a black square in the center (say 10x10cm).
Mark all four corners with a red point and connect the points with a line - this should result in a PERFECT rectangle.
The camera-to-wall distance should be sufficient to cover the newspaper in the viewfinder but not bigger.
The four red points MUST BE VISIBLE in the extreme-most corners of the viewfinder or better in the TFT after taking a shot.
The rectangle must not be tilted! Check this with a couple of sample shots at f/11.
Forget water-level devices - these are not accurate enough because they don´t measure the optical path.
Then do _numerous_ fine-stepped manual focus iterations around the suggested focus distance supplied by the AF - at
least 20 iterations between the two points where the image gets out-of-focus on either side. You need to do that at
wide-open aperture and maybe 1 stop down. In the analysis you should be able to find one image where the center is
perfectly in-focus and another one where all four corners are in focus (due to the quite common field curvature you
will usually not find a perfectly sharp center and corner on the same image). In case of a zoom it should be sufficient
to run this test at the most extreme focal lengths.
There´re a couple of indications whether your lens has a centering problem:
- the black square. If one or more lens groups are "loose" they tend to follow gravity. This is visible via the black square. The horizontal edges will be blurred or show a halo effect whereas the vertical edges are sharp(er). This is a frequent problem with VR/IS lenses or zooms that extend significantly.
- one of the corners or a whole side is blurred whereas the other corners are not. Usually you will be able to find an in-focus frame for the "problem" spot in another focus iteration or none at all. If the corner remains blurred across all focus iterations you should send the lens straight to the service department. If the "problem" spot is sharp on a neighboring focus iteration this is still Ok and not field relevant. Counter-check with the stopped down image in this vicinity.
If you find the sharp focus in a "distant" iteration consider servicing.
Please be aware that this analysis REQUIRES an absolutely perpendicular setup and this is the problem. If the setup is sub-optimal you run into simply out-of-focus scenarios that are no indication for a problem whatsoever. This is specifically a problem with fast lenses (f/2.8 and faster) where the DOF is extremely shallow wide-open.
Please note that it is absolutely unrealistic to have a perfectly centered lens - it just does not exist. In case of a slight centering problem you should not send it to the service department simply because there´re limits of what they can do. They may even make it worse after all (not unheard of) or simply return it arguing that the lens is within factory limits.
If I was a normal user I would run this procedure only if there´s reason to be suspicious. Otherwise you just risk loosing your sanity.
|
|
