[OutOfFocus]

I was comparing the average ratings of some lenses in the Lens Review section and I saw what I think are some anomalies. Comparing relative ratings is one influence on my purchasing decisions, so I would like to know to what extent you believe these average ratings are accurate, whether the anomalies I think exist are real or not, whether there are other examples, and the extent to which you care.

There are some technical reasons I can think of that might explain why these could occur, even if everybody involved is acting in good faith, but the explanation is involved so I don't want to go into it if there isn't any significant issue.

The first anomaly I noticed is that the DA 18-55 F3.5-5.6 AL WR is rated substantially worse than the DA 18-55 F3.5-5.6 AL II (7.63 from 27 reviews versus 8.26 from 35 reviews, respectively, at the time of writing). Since these are supposed to have the same optics, why the big discrepancy? In fact, the WR is only slightly better rated than the original version (7.42 from 64 reviews), and everything I've read suggests it has much better optics, and it's WR as well.

The second anomaly I noticed is that the DA 18-135 F3.5-5.6 WR is rated slightly lower than the DA 18-55 AL II mentioned above (8.14 from 14 reviews versus 8.26 from 35 reviews, respectively, at the time of writing). All the reviews of the DA 18-135 that compare it to the DA 18-55 suggest it's comparatively much sharper, plus it has WR and the new DC motor which people seem to like. So seeing it rated slightly lower is a bit surprising. (It is rated higher than the original and WR versions of the 18-55 lens though.)

So are these valid observations, or is there something I've missed? Are their any other examples where the consensus of opinion is that lens A is better than lens B, but the average rating of B is higher than that of A? How big a deal is this to you, if at all?

[twitch]

Price plays a large part in peoples ratings so it makes it impossible to really compare lenses across price brackets. That's where Pentax K / Samsung NX Lens Tests comes in.

[dgaies]

The actual average numerical rating are pretty much useless in my opinion for a variety of reasons. The primary reason is that one person's "9" might be another person's "7" since there's not universal method by which individual rate the lenses. Also, some people rate things relative to the value. So something like the DA35/2.4 might get a "10" because it's a great lens for the money while the FA31 might get a "9" from the same person because for $1000 they expected a little more. That doesn't mean the person thinks the DA35/2.4 is a better lens than the FA31.

Bottom line, you're best off reading the text of the reviews and various threads on the forum and using that information to make a decision.

[Deimos]

I think there is something to be said for relative consistency of a high rating in the reviews. Say a lens gets pretty much nothing but 9s and 10s over a large volume of reviews. Its likely a winner

[Lowell Goudge]

I would agree with dgaies. When I rate a lens I consider and I think I state rating is relative to price paid. Most $30 lenses get an 8 which means if you learn the lens it cn take exceptional photos

But consider a $200 lens as an 8 is a much better lens than a $30 lens as an 8

The other thing I do is try to put honest show stoppers in the review.

[jimr-pdx]

Expectations also play a part; if a cheap lens really does well you can assume a point was added for the amazement! I'm curently a victim of that, with a Rikenon zoom that is nearly a 9 to me despite immense old-school flare issues. Another smaller issue might be that a bad lens gets more reviews as it's sold among the members; it's probably a better measure to go by how frequently a lens actually appears in the FS forum, and perhaps for how long

But I agree with others here - read the text to learn what the numbers mean to the reviewer.
And reviewers: please add more text! It really helps buyers determine if your problems are relevant to their shooting.

[Digitalis]

I point out the show stoppers in my reviews also - though that does depend on whether the lens has any issues I really consider to actually be show stoppers*

*And so far there really aren't, though Voigtlander and Nikon have a few lens design bloopers that are really annoying.

[Gareth.Ig]

For me it depends on the number of reviews. For example, the SMC M 50mm 1.7 has 90 reviews and a rank of 9.16 and the SMC M 50 f2.0 has 43 reviews and a rating of 7.72. Those are both rankings you can depend on, and the higher the ranking the more dependable the ranking.

If there are less reviews (less than 10) and the ranking is under 8, I always look to see if the ranks are consistently below 8 or if there are a bunch of happy people and one disgruntled owner, which appears to happen quite often. I read disgruntled owner(s) review to then determine if I should disregard the ranking or not.

Also good to look at the % Recommended, which should give you a sense of the value.

Thinking you can compare one lens to another by rankings is not the best way to think about the lens. People buy different lens for different purposes. The most you can expect is that you understand your purpose for the lens and then use the rankings to confirm you're not buying a lemon.

[jimr-pdx]

Another point that is relevant (to me at least ) - which camera you are using. I gave up on the 55-300 because I could not get satisfactory images in fairly low light - my fault of course, not the lens', and note that I did not enter a review for it. That was with my K-7, which is a fine but iso-constrained camera. Now I own a k-x and can bump iso up another stop or more without discomfort, and the 55-300 regains some appeal.

Funny to look back - I really liked the 16-45 and 55-300 lens combo, but with the K-7 became disillusioned and sent them on while trying other lenses (especially WR types). A year and a camera later, I've gathered both those lenses back into my fold. Thankfully they were regained with exceptional deals, so other than time & sanity I lost very little..

[pinholecam]

Like all such numbers, I'd look at the mean as well as the median.
There can be 5 folks giving '10pts' each to a lens and have a very disgruntled fella giving a '0' and the mean number then becomes '8.3'
'
But looking at the median, I know that out of 6 ppl, 5 were very satisfied with the lens which means something.

The one or few that gave low scores deserve a through read of their comments to understand the reason for the score.
It can be due to reasons that has affected them, but may not be applicable to every lens (eg. faulty lens; too expensive; can't compare to his/her FA ltd, etc)

[JinDesu]

I'd at least read the reviews, to see if they know what they are talking about when they submitted their number

[magkelly]

It varies, but i think the reviews are pretty good. Price-wise I think that the database can be a little off. The Sigma 70-300 1:4-5.6 I just bought is rated 8.60 in the review section here. That's about the same as the reviews I saw elsewhere for it. I make a point of getting a second opinion but overall things can be good review-wise. Price-wise I often see things listed for a lot more, or a lot less, but then again you always have to consider lens condition plays a part too...

[Ash]

User ratings won't be gospel or comparative, but they're at least more useful than the forum's image ratings as voted by members...

[bjake]

Have found some ratings to be very accurate and some not so.There is one lens that I really like that has one of the lowest ratings.The thought seems to be widespread because the lens can be has for very cheap which is good as I let my son have my copy but will replace it.
Jake

[OutOfFocus]

Thanks to everyone who replied for their comments, they've been very helpful. I'll quote dgaies here, but I realize many others made similar points.

The actual average numerical rating are pretty much useless in my opinion for a variety of reasons. The primary reason is that one person's "9" might be another person's "7" since there's not universal method by which individual rate the lenses. Also, some people rate things relative to the value. So something like the DA35/2.4 might get a "10" because it's a great lens for the money while the FA31 might get a "9" from the same person because for $1000 they expected a little more. That doesn't mean the person thinks the DA35/2.4 is a better lens than the FA31.

Bottom line, you're best off reading the text of the reviews and various threads on the forum and using that information to make a decision.

I agree with nearly everything you've said. And please note, I did say that comparing relative ratings was one influence, not the only one.

The two reasons you give for the numerical rating being useless are valid, but by themselves are not the reason for the average rating being useless. One person's 7 being equivalent to another's 9 just means you can't assign an absolute value to a rating. Lowell says that for him a $30 lens rated at 8 means "capable" of taking exceptional photos if you know how to use it, but there's probably at least some people who have a slightly different scale. (If you take an average over lots of people of what a rating 8 implies for a $30 lens, he might be spot on. That's not the point.)

The fact that different people determine a rating based on different mixtures of quality and value is also, by itself, not a major issue. It just means the rating is neither a pure quality score nor a pure value score, but some unspecified mixture of both. I don't have any problem with that. It's probably a good thing.

I think that the real problem with the average rating arises from the fact that, in combination with the above differences between individuals, there are different sets of individuals rating the different lenses. (That's clearly inevitable.) This can lead to the very counter-intuitive result that everyone who rates both lenses gives lens A a higher rating than lens B, using whatever personal rating scale they have and whatever balance between quality and value they prefer, but the average rating for lens A ends up less than that for lens B, because of the influence of those reviewers who rated only one lens or the other. (Edit: Apologies if you saw my first two edits. I was being dumb. I put the text back the way it was originally. I need sleep.)

So getting back to the two specific examples I mentioned originally, if you had to rate both the 18-55 WR (average reviewer price $134.14) and the 18-55 AL II (average reviewer price $99.29), would you really rate the former 0.63 points worse than the latter? Possibly, but I'd be really surprised if that was common.

Now, the 18-135 is a lot more expensive than the 18-55, but when I read the reviews for the 18-135 I didn't get the impression from those that compared it to the 18-55 that it was of lesser value. In fact, I got the distinct impression they were quite happy with their purchase. I did just go back and read the reviews for the 18-55 AL II, and many of those used a phrase something like "great lens for the price", so perhaps exceptional value explains the slightly higher rating for the 18-55.

Overall, I got the impression most people don't care much for the average ratings. Although I don't either, it's hard to ignore completely. I can't mentally compute my own average for the "believable" ratings only, so it's tempting to think of it as a valid summary.