Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
01-20-2013, 02:15 PM   #1
Site Supporter




Join Date: Jan 2013
Location: Texas
Posts: 32
Biased Ratings in Lense Reviews

Based on 40 years of experience with Pentax lenses, I see problems in our sharpness ratings:

- Large aperture lenses have higher scores for 'sharpness' that they should. I thing it is due to the 'prestige' (price) factor, but also the fact that photos are typically taken at shorter shutter speed with them.
- Older lenses are rated on their sharpness from corner to corner of a 24x36 mm image; newer lenses are used with APS-C format cameras, and the sharpness rating is more weighted on the centre of the image.
- There are lenses with a terrible full aperture sharpness, improving a lot at, say, 5.6, that get the same score than lenses with a good full aperture, but less stunning maximum sharpness

Another thing I don't like is 'Aberrations'. It should be divided into 'Geometrical Distortion', 'Color rendition' and 'Vignetting'.

We could have the following categories:

- Sharpness full aperture 24x36
- Sharpness full aperture APS-C
- Optimum Sharpness
- Vignetting 24x36
- Vignetting APS-C
- Distortion 24x36
- Distortion APS-C
- Color Rendition
- Bokeh
- Autofocus
- Manual focus

Cheers,

Yves

01-20-2013, 02:58 PM   #2
Pentaxian




Join Date: Mar 2010
Location: Frankfurt am Main
Posts: 1,055
With the exception of 'Aberrations', I agree with all you said.

But I don't think your sharpness ratings would lead to a more realistic overall judgement. Only few reviewers have really tested the lenses in regard to all the present categories. This would be ok, if they would just not attach a rating to such a category. But my feeling is, that many reviewers just make a guessing for categories they haven't really tested. Following your suggestions, this affect may increase.

One should really not just look at the average rating, but read the text part of the reviews. Most times this will give you an impression about how to rate the rating of the reviewer.
01-20-2013, 03:01 PM   #3
Loyal Site Supporter




Join Date: Mar 2009
Location: Southern Indiana
Photos: Gallery
Posts: 15,337
QuoteOriginally posted by Belgarchi Quote
Based on 40 years of experience with Pentax lenses, I see problems in our sharpness ratings:

- Large aperture lenses have higher scores for 'sharpness' that they should. I thing it is due to the 'prestige' (price) factor, but also the fact that photos are typically taken at shorter shutter speed with them.
- Older lenses are rated on their sharpness from corner to corner of a 24x36 mm image; newer lenses are used with APS-C format cameras, and the sharpness rating is more weighted on the centre of the image.
- There are lenses with a terrible full aperture sharpness, improving a lot at, say, 5.6, that get the same score than lenses with a good full aperture, but less stunning maximum sharpness

Another thing I don't like is 'Aberrations'. It should be divided into 'Geometrical Distortion', 'Color rendition' and 'Vignetting'.

We could have the following categories:

- Sharpness full aperture 24x36
- Sharpness full aperture APS-C
- Optimum Sharpness
- Vignetting 24x36
- Vignetting APS-C
- Distortion 24x36
- Distortion APS-C
- Color Rendition
- Bokeh
- Autofocus
- Manual focus

Cheers,

Yves
Well, I think overall, these categories are probably really tough for all but the most experienced users to truly evaluate. It is better to actually read the narrative of someone's review and see if they have much experience with other, similar lenses to truly evaluate how a particular lens performs.

Currently, Pentax doesn't have a full frame camera of any sort, in production, so I think unless otherwise specified, most of the reviews will be based on an APS-C sized sensor.
01-20-2013, 03:38 PM - 1 Like   #4
Pentaxian




Join Date: Oct 2011
Location: Savannah, GA
Posts: 4,989
They are impressionistic reviews, not standardized tests -- you can only hope for a general impression. Everybody likes it, everybody agrees it is kind of crappy, or results are mixed. The really valuable parts are the quirks you discover that actually matter to usage -- it is unbalanced and you need a certain type of tripod collar, the focus throw is too short, a teleconverter won't fit, bokeh looks like Phyllis Diller, etc etc

01-20-2013, 03:47 PM   #5
Veteran Member
ihasa's Avatar

Join Date: Jul 2009
Location: West Midlands
Photos: Gallery
Posts: 2,054
I agree that reading the comments about a lens is key, and gives you most of the information you require without some sort of cumbersome ratings system.
01-20-2013, 03:48 PM - 1 Like   #6
Site Supporter
jatrax's Avatar

Join Date: May 2010
Location: Oregon
Photos: Gallery | Albums
Posts: 10,444
While I agree with you I am afraid it is asking too much for casual users to properly evaluate a lens in that much detail. After all these are user reviews, not professional ones with proper standards and procedures for testing. As noted above, read the narrative and throw out the nonsense.

The review database is a tremendous resource and I applaud any effort to improve it. And I wish all reviewers would understand the purpose and use it for what it is instead of treating it like the reviews on Amazon. A recent review gave the DA 17-70 a '1' rating because: "It does not focus". That's not a review, that's a complaint.
01-20-2013, 04:03 PM   #7
Loyal Site Supporter




Join Date: Jun 2009
Location: Tumbleweed, Arizona
Photos: Gallery | Albums
Posts: 5,035
Without quantitative testing everything is subjective. Even with quantitative testing - like DXO, there are folks pointing out that its not perfect - the holy grail. As most have pointed out, its the written opinions that matters the most. Even the reviews on Amazon, what I find most interesting and helpful, are not the 98 opinions with 5 stars - its the 1 star ratings with an explanation that provides the insight to the product's weakness or shortcomings.

A lot of the written reviews do include experience from film bodies, however the vast majority of them are with the more recent digital bodies, that really do just use the center of the glass. Where this will matter the most will be when (or if) Pentax provides a full frame body, then its going to be a large culling process in terms of glass that will perform well in that environment. As folks that have either migrated or expanded their cameras to the D600/D800 attest, the only way to get the most out of the sensor is with excellent glass.

01-20-2013, 05:15 PM   #8
Site Supporter




Join Date: Jul 2011
Location: Auckland, New Zealand.
Posts: 328
QuoteOriginally posted by interested_observer Quote
Without quantitative testing everything is subjective
Pretty much say's it all

01-21-2013, 05:28 AM   #9
Pentaxian




Join Date: Nov 2011
Photos: Gallery | Albums
Posts: 3,658
Professional testing of lens sharpness is almost always done in the mid range, a couple of meters.
That doesn't tell you about sharpness at infinity, or close up.
So written user reviews that address these issues can offer help that you won't get from the professional tests.
01-21-2013, 02:58 PM   #10
Pentaxian
Just1MoreDave's Avatar

Join Date: Feb 2007
Location: Aurora, CO
Photos: Gallery | Albums
Posts: 4,838
Even if I collected the data properly and interpreted the results correctly to satify everyone (I didn't), what do I do in my review today, for the Pentax-M 120mm f2.8? Do I compare it to the one other 120mm K-mount lens, which I haven't used? I compared it to 135mm lenses instead, skipped the IQ questions, and hope it helps someone.
01-21-2013, 03:44 PM   #11
Veteran Member
demp10's Avatar

Join Date: Jun 2011
Location: Atlanta
Photos: Albums
Posts: 602
No user review or lab test can fully account for the performance of a lens under all circumstances. A lens is just a tool, and as such it will be great in certain situations and rather poor in others.

Lab tests are just that; if you are shooting test targets in a lab environment, then the results will apply correctly. User reviews are by nature subjective and usually the more one pays for a lens, the higher the expectations, and an overly optimistic review will be the result.

A good way to skim through the user reviews is to ignore the ratings (too subjective) and read the comments referring to usage examples. Pay particular attention to negative issues as people tend to be critical for thinks they do not like. Also reports about ergonomics are quite useful and informative.

Finally, the perceived lens performance will depend of other factors that potentially can be far more influential than the lens itself, for example a stable platform (tripod) or good lighting (flash, lamps, etc.).
01-22-2013, 12:41 AM   #12
Pentaxian
kh1234567890's Avatar

Join Date: Sep 2010
Location: Manchester, UK
Photos: Gallery
Posts: 2,357
QuoteOriginally posted by Belgarchi Quote
Based on 40 years of experience with Pentax lenses, I see problems in our sharpness ratings:
The review system should make it easier to include full size example shots with EXIF info. Then the user could decide for themselves how a particular lens performs in real life. At present it is possible to do this by for example including a link to a Flickr hosted image, as I do, but it is a bit of a pain.
01-22-2013, 01:06 AM   #13
Pentaxian
Not a Number's Avatar

Join Date: Mar 2012
Location: Venice, CA
Posts: 4,030
Lab tests at least provide an objective standard that has reproducible results that can be applied between lenses. Back in the 80s Modern Photography magazine sold a len's testing kit using the same resolution targets, methodology and criteria they used in their testing labs. Modern Photography would test several of the same model lenses as there often was noticeable variation.

I recently found the booklet and USAF resolution charts and scanned them. A friend of mine in the UK cleaned them up and is hosting them on his website. I scanned the resolution charts at the highest native resolution of my scanner - 1200 dpi. You can purchase other resolution targets which can be adapted to Modern Photography's methodology. Distortion measurement is covered.

Page 9

People can use Modern Photography's methods for their reviews or perhaps Pentax Forums can adapt them for standardized testing.
01-22-2013, 05:56 AM   #14
Pentaxian
bdery's Avatar

Join Date: Mar 2008
Location: Quebec city, Canada
Photos: Gallery
Posts: 5,695
QuoteOriginally posted by vonBaloney Quote
The really valuable parts are the quirks you discover that actually matter to usage -- it is unbalanced and you need a certain type of tripod collar, the focus throw is too short, a teleconverter won't fit, bokeh looks like Phyllis Diller, etc etc
I agree with this.

Make the rating system too complex and no one will use it. Or use it poorly. The reviews are guidelines, user comments, feedback on each person's personal experience. The ratings give a guideline, and get more useful when a large number of people enter their input. But like anything else, you take some and you leave some.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
aperture, aps-c, corner, distortion, image, k-mount, lenses, pentax lens, rendition, sharpness, slr lens
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
An observation, or am I just biased? amc654 Pentax DSLR Discussion 25 01-12-2012 11:38 AM
Error Different ratings in lens reviews Jonathan Mac Site Suggestions and Help 4 11-08-2011 09:58 PM
How accurate do you think the average ratings in the Lens Reviews are? OutOfFocus Pentax SLR Lens Discussion 47 08-06-2011 07:18 PM
The IMF incompetent, biased and culpable jeffkrol General Talk 4 02-15-2011 09:55 PM
Fox News: DEAD LAST In Online Ratings johnmflores General Talk 27 09-18-2010 03:45 AM



All times are GMT -7. The time now is 12:48 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top