Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
01-13-2020, 11:09 PM - 1 Like   #1
Site Supporter
Site Supporter
EssJay's Avatar

Join Date: Jul 2019
Location: Kamloops B.C
Photos: Gallery
Posts: 70
Lens reviews 10/10 ... Really?

I'm sure everyone can agree that the lens review section of this forum is one of the most useful tools to us Pentax user. I have used the resource for years to find information on lenses and it has been invaluable in regards to choosing the right lenses for my desired photography kit ... thank you PentaxForums!!


So I decided to begin posting reviews of my most used lenses, but only lenses that I have had years of experience using, as well as having had experienced using other lenses of a similar type to compare with - this seems to be the only fair way to write a review. The first few reviews that I wrote were definitely influenced by the rating that the lens was already given by previous reviewers, however the more I thought about it the more wrong it seemed. When filling out the required "post a review" form I noticed that the scale used is out of 10, but 5 is rated as "good". So according to the reviews, nearly every lens Pentax has made falls between "good" and "excellent" - most closer to excellent.

Now don't get me wrong, I'm a loyal user and lover of Pentax lenses for umpteen years, but do the reviews seemed way too skewed to the right to be accurate? Does it seem to anybody else that 10's are given out way too freely? Should 10's not be reserved for only the best of the best?


I have used a few real stinkers, but according to the rating they have they are above "good". The FA 28-90 f3.5-5.6 for example was one of the most horrendous pieces of garbage I've ever put on my camera but I ended up rating it 5. Why? because I felt that it was in line with the other reviews. But a 5 rating indicates a "good" lens. If anybody ever asked me if this was a good lens, without hesitation I'd say NO. That was a mistake on my part and I will not rate lenses this way in the future.


I believe the lens ratings would be more accurate if there were no option to give the lens an overall rating, and the overall rating was instead an average of the individual rating categories. I noticed that many lenses get rated 10 even though the reviewer gives low values in the individual categories. This makes no sense to me. How can a lens be described as "soft" and have a "cheap plastic build" with a rating of 7 for aberrations, 8 for sharpness, 9 for bokeh, autofocus, and handling, and a 10 for value end up with an overall rating of 10? If the categorical ratings were averaged out it would come in as an 8.6 rating. Round it up to 9 and it would be more accurate than the given 10.


I feel that many cheap yet decent performing lenses get a rating way too high. It's as if value trumps all other categories and it doesn't matter if the lens has a shoddy build and poor auto-focus because it was only $20 ... 10/10. I also see the same thing on very expensive lenses. The reviewer will name a bunch of things that they really dislike about the lens yet give a 10/10 overall. Where's the logic?


But since lens reviews are completely subjective who am I to argue


I would love to hear what other forum members think on the subject and I believe it would be a good discussion topic.


Do you think there could be changes made to the review section to make it more accurate? Or perhaps you believe that since reviews are so subjective the review section is perfect as is? Chime in!!

01-13-2020, 11:39 PM - 1 Like   #2
Moderator
Site Supporter
Loyal Site Supporter
Sandy Hancock's Avatar

Join Date: Feb 2012
Location: Adelaide Hills, South Australia
Photos: Gallery | Albums
Posts: 8,785
This has been discussed many times. The review section is just fine.

I can't remember who first suggested it, but I subtract 5 from the average rating and consider the remainder to be a score out of five. That makes the score for the FA28-90 - 5/10 (or 0/5 by my formula) - fit with your observation

Frankly, whatever the original intent of the scoring, I consider a lens with an average Pentax Forums rating below 9/10 (modified with the above formula to 4/5) to be "good". Below 8 (or 3/5) is mediocre, and below 7 isn't even worth considering.
01-14-2020, 01:10 AM   #3
Site Supporter
Site Supporter
EssJay's Avatar

Join Date: Jul 2019
Location: Kamloops B.C
Photos: Gallery
Posts: 70
Original Poster
QuoteOriginally posted by Sandy Hancock Quote
This has been discussed many times. The review section is just fine.

I can't remember who first suggested it, but I subtract 5 from the average rating and consider the remainder to be a score out of five. That makes the score for the FA28-90 - 5/10 (or 0/5 by my formula) - fit with your observation

Frankly, whatever the original intent of the scoring, I consider a lens with an average Pentax Forums rating below 9/10 (modified with the above formula to 4/5) to be "good". Below 8 (or 3/5) is mediocre, and below 7 isn't even worth considering.
Ah shoot, my bad. I guess I should have searched harder for a similar thread. Just thought it might be a stimulating conversation had it not taken place already.

anyhow I guess that's one vote for for the review section being perfect as is if one applies a simple mathematical equation
01-14-2020, 01:24 AM   #4
PJ1
Site Supporter
Site Supporter
PJ1's Avatar

Join Date: Jul 2012
Location: Mosquito Creek, Australia
Photos: Gallery | Albums
Posts: 3,923
I use the ratings in different ways. First I look at the overall ratings in relative terms; e.g for the same type of lens, a 9.8 should be better than a 9.4, and that should be better than an 8.6, etc Then if I am seriously considering a lens, I look at the individual ratings and at any "outliers" in particular. If someone rates a lens much lower than most reviewers I expect to see images to back up their assessment; no pics, I ignore the rating. Most lenses these days are very good and ratings are subjective assessments. They tell us how others feel about the lens which - if we read all the reasons they give for their opinions - is the best indicator of how we might feel about it ourselves.

01-14-2020, 01:50 AM - 1 Like   #5
Moderator
Site Supporter
Loyal Site Supporter
Sandy Hancock's Avatar

Join Date: Feb 2012
Location: Adelaide Hills, South Australia
Photos: Gallery | Albums
Posts: 8,785
QuoteOriginally posted by EssJay Quote
Anyhow I guess that's one vote for for the review section being perfect as is
Thank you for your sarcasm. I'm not saying it's perfect. Just that it's fine for what it is - a reference and guide for those who want to learn about and/or buy Pentax lenses. In that regard I'd suggest it is without peer. The absolute numbers are largely irrelevant as long as the relativities are valid.

Anyway, there's far too much information already embedded to contemplate a forced overhaul.
01-14-2020, 02:12 AM - 2 Likes   #6
Pentaxian




Join Date: May 2019
Posts: 1,516
I've always felt that numerical reviews are inherently flawed anyway, so it's best to ignore the "final score" and go read through the better written reviews. Thorough descriptions and images are the only way to accurately assess anything. An easy example is the Pentax-M 20mm f/4 pseudo-limited. I got one a month ago, and for what I want it to do, which is a miniature UWA with decent optical qualities, it is perfect. I couldn't care less that the corner performance is mushy even at f/8, or that it's only f/4 wide open, because I don't need UWAs often enough to toss a 600 g lens in a bag instead of in a pocket "just in case".
How am I to condense such an interesting, specific lens into a number?
01-14-2020, 02:31 AM - 1 Like   #7
The one and only
Loyal Site Supporter
iheiramo's Avatar

Join Date: Nov 2016
Location: Espoo
Photos: Albums
Posts: 1,624
I have been gathering informantion and experience over M series primes last years. At first PF reviews were great help in deciding the lenses to get. Later bought almost all of them to get personal experience. In retrospect I can confirm that the reviews guided me to right direction, but nowadays IMO there are few lenses with too high scrore and few with too low, but in general they you some idea on what is worth acquiring, but you also need to read the reviews, as numbers only can be missleading.
01-14-2020, 02:39 AM - 3 Likes   #8
Pentaxian
Dartmoor Dave's Avatar

Join Date: Aug 2012
Location: Dartmoor, UK
Photos: Gallery | Albums
Posts: 2,404
I just ignore the numerical ratings. I read the review and dismiss it as nonsense if it does things like criticise a manual focus lens for not having autofocus. If the written review seems coherent I look carefully at any photos included, and if the photographer's work is competent then I give his opinions a lot of credit. If the photos are awful because of the photographer's obvious incompetence rather than because of the lens or camera, it's another reason to just ignore that review.

I'd never buy a piece of gear just based on the reviews though. I think the sample archive here on Pentax Forums is a much more reliable resource for judging whether a lens will suit me, and for a camera I'd look at the model specific thread on this site since there usually is one. Flickr is useful too.

01-14-2020, 03:21 AM - 2 Likes   #9
Senior Moderator
Loyal Site Supporter
Loyal Site Supporter
BigMackCam's Avatar

Join Date: Mar 2010
Location: North-East of England
Posts: 14,680
QuoteOriginally posted by EssJay Quote
Do you think there could be changes made to the review section to make it more accurate? Or perhaps you believe that since reviews are so subjective the review section is perfect as is? Chime in!!
QuoteOriginally posted by EssJay Quote
I guess that's one vote for for the review section being perfect as is if one applies a simple mathematical equation
QuoteOriginally posted by Sandy Hancock Quote
I'm not saying it's perfect. Just that it's fine for what it is -
...
Anyway, there's far too much information already embedded to contemplate a forced overhaul.
The user reviews section is far from perfect, but as Sandy points out there's too much information embedded already. The scores are completely unrealistic, and yet not without value... It's possible to get a broad concensus, for instance, on general sharpness, fringing, AF performance, build quality etc.

The problem with any scoring system is that it will be applied subjectively by each user. In fact, even without scoring, the commentaries are highly subjective, based on each user's frame of reference, personal preferences and use cases. What one person likes and considers good in a lens, another sees as flaws. A lens that's sharp across the frame and has no obvious aberrations may considered excellent by some users, but others may feel it produces flat and lifeless images. Field curvature might not matter to some users, and be absolutely critical to others. Then there's the use of the same lens on APS-C versus FF... Should lenses only be reviewed based on the format for which they were intended?

There are so many factors, I think it's all but impossible to come up with a reviewing approach and scoring system that strips away variables between reviewers, not least their subjectivity...

Last edited by BigMackCam; 01-14-2020 at 04:33 AM.
01-14-2020, 03:34 AM   #10
Site Supporter
Site Supporter
EssJay's Avatar

Join Date: Jul 2019
Location: Kamloops B.C
Photos: Gallery
Posts: 70
Original Poster
This is great stuff here. Seems all feel that the qualitative assessment of the reviewer takes precedent over the quantitative number assigned - the ratings are close enough to be useful but not perfect. I love this stuff because for the past few years i've been doing survey design, many times using a Likert Scale which is basically what the rating system is. A challenge is assessing whether or not the data collected from a Likert Scale is relevant enough to use that style of survey. I guess the real question here is do people see any value in collecting more precise numerical data, and the answer is no.
01-14-2020, 05:11 AM   #11
Senior Moderator
Loyal Site Supporter
Loyal Site Supporter
BigMackCam's Avatar

Join Date: Mar 2010
Location: North-East of England
Posts: 14,680
QuoteOriginally posted by EssJay Quote
I guess the real question here is do people see any value in collecting more precise numerical data, and the answer is no.
I'd love it if we could record more precise and better-baselined numerical data... but setting and enforcing a consistent objective approach across a wide variety of members with different use cases, rendering preferences, experience of comparable lenses, sensor formats, general technical experience, expectations etc. would be a truly Herculean effort, IMHO, and would likely exclude members who feel unable to contribute in the strict manner required.

For what it's worth, I look at the user reviews to canvas a broad majority opinion on performance aspects in the field. I tend to give more weight to opinions from certain members that have demonstrated some level of experience and common sense in the forums or in the reviews themselves (though I of course appreciate all reviewers' contributions). Then, I look at sites such as Optical Limits, LensTip, ePHOTOzine, dxomark and others to see how the lenses perform under more stringent test-bench conditions. Looking at multiple sources, it's usually possible to get a very good idea of how a lens should perform. What's interesting is, not all sources always reach consensus... often, there's an outlier or tests and conclusions are split; which may be due to sample variation in some cases, but I also believe even the most technical and disciplined testers can be unconsciously subjective, despite efforts to the contrary.

Last edited by BigMackCam; 01-14-2020 at 06:12 AM.
01-14-2020, 08:01 AM - 3 Likes   #12
Pentaxian
photoptimist's Avatar

Join Date: Jul 2016
Photos: Albums
Posts: 4,052
Perfect data is perfectly impossible.

Some people have bad copies of lenses, bad shooting technique, challenging subject matter, or suffer from excessive pixel-peeper syndrome. For them, every lens sucks.

Some people have low standards, never shoot in challenging conditions, or have ego issues that make them want to claim they own the best lens ever. For them, every lens they own is a 10/10.

And lots of people bias their opinions by the price: a $5 lens that works at f/8 is a great lens and a $2,000 lens that isn't pixel-peeper-pinpoint-perfect, corner-to-corner, and wide-open is a low-rated lens.

Knowing one person's opinion of their copy of the lens on their subject matter provides little insight into what my opinion of my copy of the lens on my subject matter is likely to be.

That said, if several people review a lens, the average is likely to be closer to the truth and a bit of reading of the reviews can help determine who's opinions might be closer to my own.
01-14-2020, 10:56 AM   #13
Site Supporter
Site Supporter
luftfluss's Avatar

Join Date: Jun 2011
Location: NJ
Photos: Gallery | Albums
Posts: 6,824
QuoteOriginally posted by Sandy Hancock Quote
I can't remember who first suggested it, but I subtract 5 from the average rating and consider the remainder to be a score out of five.
I believe it was @normhead who came up with this.
01-14-2020, 11:32 AM   #14
Site Supporter
Site Supporter




Join Date: Mar 2009
Location: Gladys, Virginia
Photos: Gallery
Posts: 21,573
The best reviews should have a nice narrative description of the number scores and photos that illustrate said narrative.

I don't find the number score is nearly as important as the narrative and photos and can tell pretty quickly if the person making the review has enough experience for me to take their opinion seriously. The one thing I would add is that I do wish there was a way to sort reviews based on the number of people who felt like they added something. Currently, I think the only way to sort the reviews is with regard to when they were made.
01-14-2020, 12:05 PM   #15
Pentaxian
35mmfilmfan's Avatar

Join Date: Oct 2016
Location: Norfolk, UK
Posts: 1,095
In the days of my youth (late sixties / early seventies), the UK magazine Amateur Photographer ran many tests on manual focus lenses (all that we had in those days) using charts that helped them to test edge and centre sharpness, barrel and pincushion distortion and probably much else, giving results in LPPM (Line Pairs Per Millimetre, IIRC). The actual test images, however, were printed on fairly poor quality paper, doubtless to keep costs down, so the finer points of the images were somewhat less than helpful.

However, even with years of inexperience behind me, I felt that these 'paper' tests were somewhat irrelevant in the real world, fascinating though they may have been to the then equivalent of pixel-peepers (grain grumblers, perchance?). Adding in the introduction of possible errors caused by processing, I never treated each individual review as a stand alone set-in-stone standard.

I did, however, use to keep copies of all of these reviews, with a view to comparing the results from different lenses (not that I could afford them !), both marque lenses and independents, although in those days there seemed to be less information concerning the actual manufacturers of these lenses. From what I recall, there was a definite tendency to score 'name' lenses higher, even though, to my eyes, there seemed very little difference, and the published figures seemed to bear this out. I found more help was obtained from the standard photos included with each review, of the clock in a church tower some distance away (might have been a spire), but even these I felt could be affected by atmospheric conditions and again processing and printing. As comparison tests, they had value, but as indicators of each lens's worth and capabilities, less so.

With the PF reviews, I feel that the 'real world' aspect of published images is of worth for comparisons, despite the fact that poor copies of some lenses do exist and may skew peoples' perceptions.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
categories, lens, lens reviews, lenses, pentax, review, reviewer, reviews, section, value
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Night Really, really big Snowflake built by a local High School Jpeachee Post Your Photos! 3 01-15-2019 06:57 PM
Really, really expired stuff Smolk Pentax Medium Format 9 04-06-2014 09:27 PM
Lens Reviews - Camera used not listed on 3rd party lens reviews? ak_kiwi Pentax SLR Lens Discussion 3 02-15-2014 01:34 PM
Really-really cheap sub-2.8 AF lens minahasa Pentax SLR Lens Discussion 55 09-14-2011 10:52 PM
Reviews really missed the mark on Kr AWB in tungsten light LRB21 Pentax K-r 3 09-06-2011 03:12 PM



All times are GMT -7. The time now is 08:14 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top