[gofour3]

I think that sometimes people will rate a lens lower because they expect it to do something that it wasn't designed for. I've seen more than enough remarks about f/4 - f/5.6 telephoto zooms being "bad in low light". No crap! People should know a little about what they are buying before they complain. I think that sometimes there is some wishful thinking involved with cheaper lenses, that maybe I can get by with this and save a few hundred and they are upset when it doesn't work out. I bought my DA 10-17 for a very good price in the Marketplace from an owner who hated it because of the distortion. It was cheaper than a 12-24 and he assumed he could easily de-fish it with software.

Yeah or rate an old manual focus lens lower because it's not AF!

Phil.

[Jonathan Mac]

There are also people who badly score lenses by giving it too high a score. This is probably more of a problem than under-scoring. When I look at the ratings, assuming there are more than a handful of reviews, anything over a 9 is excellent, and anything less than a 7.5 is terrible, because the ratings are hugely skewed upwards. Of course, that's a generalisation and is to be treated with caution.

If you want a good idea of how a lens performs before using it, then one overall rating is not very useful. You need to read all (or at least many) of the reviews, keeping an eye open for recurring comments (purple fringing, sharp wide open etc) and bearing in mind that not everyone looks for the same good/bad points in a lens. Looking at pictures produced will also help, especially if you know which camera and settings were used.

[Sol Invictus]

There are also people who badly score lenses by giving it too high a score. This is probably more of a problem than under-scoring. When I look at the ratings, assuming there are more than a handful of reviews, anything over a 9 is excellent, and anything less than a 7.5 is terrible, because the ratings are hugely skewed upwards. Of course, that's a generalisation and is to be treated with caution.

If you want a good idea of how a lens performs before using it, then one overall rating is not very useful. You need to read all (or at least many) of the reviews, keeping an eye open for recurring comments (purple fringing, sharp wide open etc) and bearing in mind that not everyone looks for the same good/bad points in a lens. Looking at pictures produced will also help, especially if you know which camera and settings were used.

This is exactly what I do too. I look for patterns in the written review and largely ignore the numerical score. The scores are so over-inflated that if I see a score around 7, a score which should be indicative of a better than average lens, I assume the lens is actually around a 3 or 4 and people are excited because they got it for $0.75 or something.

[Class A]

What do you say Adam ? Drop the lowest number !

As others have pointed out, inflated ratings are a problem as well.

If there is to be anything else but a straight average -- a distribution being the ideal -- I suggest a weighted mean, for instance, the average of all ratings without the bottom and top 2.5% ratings.

[Jonathan Mac]

Everyone reviews their lenses differently. As such, there is no way to accommodate this into the numerical rating, and it should be left alone.

The people reading the reviews need to learn to interpret them properly (as with any data) and move away from the desire to know everything about an item by looking at a single number, which is the total of assessments made using entirely different criteria.

Using only the total rating is like having $100, €100 & £100 all in your wallet, and saying that you have 300. 300 what?

[tim60]

After reading the first page of this thread I notice that many seem to be concerned about numerical ratings.

This is perhaps a misapplication of Lord Kelvin's dictum that measurement and numbers are necessary to true knowledge. The page where that quotation comes from is comparing scientific knowledge which has advanced to the stage of numerical expression rather than just the observation of relationships of things leading to the development of theories which can be investigated in more formal methods.

My interest is in the descriptive parts of the lens reviews, where users describe their experience of the lens. some provide some quite interesting insight into what the user tried to do witht he lens and what happened.

I like the richness of descriptive knowledge of things.

The price information about lenses is very hard to use, and not very interesting to me. When I started collecting the Takumars I kept a spreadsheet of the price of every auction sale on eBay until I bought one. This gave me inforamtion abotu the market value of each based on what the real market of people buying in a competitive market were doing. That, in turn, gave me a basis for deciding how much to bid for a particular one that came available. Therefore I got most of my collection for average or below price - based on that data source.

Actually, there are several times I have seen reviews by people who tell the reader that their lens was seriously damaged, and then give a review including the impact of the damage as part of their judgement. Would I base my opinion of anything else on its performance when serious impaired?

I like Jonathon Mac's comments - demonstrates significant questioning of some of the basic aspects of the theory of measurement - - what I di my PhD in.

[Lowell Goudge]

The rating system has always been flawed, in terms of numerically

There are no rules or guidelines, so how do people rate a lens, absolute quality, value for money, ease of use, their personal experience or lack thereof?

A score without reasons is useless, the e views should discuss pros and cons, ease of use, optical quality, and anybody flaws etc... The scores should in all likelihood be dropped, or a simpler scale 0 or 1. Basically would you recommend the lens for what you paid or not.

[lytrytyr]

Actually, there are several times I have seen reviews by people who tell the reader that their lens was seriously damaged, and then give a review including the impact of the damage as part of their judgement. Would I base my opinion of anything else on its performance when serious impaired?

If a fault is typical for a particular lens in the market,
like FF/BF issues in a modern AF lens,
wobbly build in an FA 24-90,
or sticky blades on an old M42 lens,
it may be perfectly valid to review a copy with that fault.

For example, it could help someone contemplating
a BG or UG lens on KEH.

[lytrytyr]

The rating system has always been flawed, in terms of numerically

There are no rules or guidelines, so how do people rate a lens, absolute quality, value for money, ease of use, their personal experience or lack thereof?

In doing a lens review, I've tried to match my rating
to some sort of perceived average of how lenses are rated on the site.

For example, I gave the M100/2.8 a 10, despite its occasional PF,
and gave the M200/4 an 8, because it is harder to focus and not quite as sharp.

After several years of these reviews, a baseline has now been established,
and it is usually easy to spot and discount the outliers.

Of course, the system isn't perfect,
when the 55/1.8 Tak gets 9.35,
while the essentially identical 55/2 only gets 8.65.

[Rondec]

In doing a lens review, I've tried to match my rating
to some sort of perceived average of how lenses are rated on the site.

For example, I gave the M100/2.8 a 10, despite its occasional PF,
and gave the M200/4 an 8, because it is harder to focus and not quite as sharp.

After several years of these reviews, a baseline has now been established,
and it is usually easy to spot and discount the outliers.

Of course, the system isn't perfect,
when the 55/1.8 Tak gets 9.35,
while the essentially identical 55/2 only gets 8.65.

I think the bigger issue is not under-rating of lenses, but folks that habitually over-rate their lenses. I also think that price plays way too big a factor in how people rate a lens. A lousy lens that you paid 25 dollars for is still a lousy lens, it is just a cheap, lousy lens. Finally, I think it is really useful if those rating lenses mention other lenses that they have used and briefly compare them. If you are rating the FA 50 and the only lens you have to compare it to is the kit lens, then you will tend to rate it higher than if you have shot a lot of different lenses in that focal length.

[rbefly]

Well, I agree with most of what's been posted so far. There are certainly problems with the system, nothing's perfect, especially when you add the subjective human factor to the mix.
In regard to only allowing members with a certain # of posts to contribute to the reviews; Sounds good in theory, and would solve the OP's complaint about the low-ball score of 1.
But, without getting into the "elitism" issue, a new member may not be a novice at all. They just joined us, but COULD have years of experience in photography. I don't believe that is true in the original case cited, but how many times have you read a "Welcome" post (1st post) describing their extensive past photo background?
Having said that, I still think an automatic function that kicks out the lowest and highest score (once, for any given lens) is a good idea. Now, if that lens continues to receive scores of 1 or 2, it's probably a dog or has major issues.
Same on the high end. The first "10" would skew a small sample, but if it goes on to receive a dozen more?
Personally, if a lens only has 3 reviews, I take the average score (and price, for that matter!) with a large grain of salt. 20 reviews? Different story.
So, a chart or histogram of the scores would help, as would eliminating the highest/lowest single scores, assuming the lens has at least 5 reviews.
Last suggestion; Although sending readers to other sites for additional info is usually considered bad manners or at least disloyal to the original site, in this case it may be worthwhile. A short list of other "Recommended Reading" lens-evaluation links could be handy. After all, we're not the only people on the planet with strong views on PK lenses!
JMO
Ron

[lytrytyr]

After all, we're not the only people on the planet with strong views on PK lenses!

At least we don't dis them because they don't fit Nikon!

[Sagitta]

As far as prices go in the ratings, I just take them as a general idea, and go in and check to see what the norm is manually just in case someone paid $20 for a $200 lens, or vice versa.

The big, BIG thing I look for is the red NOs under 'Would you recommend?'. If I see more than a few of those, its probably the biggest clue that theres something awry with the ens, either in being overpriced or in being poor quality or both. It would be nice to see a sortable ranking just using strictly that yes/no criteria so you can skip right to the 'nos' and see why the reviewer hated that lens.

[Blue]

The rating system has always been flawed, in terms of numerically

There are no rules or guidelines, so how do people rate a lens, absolute quality, value for money, ease of use, their personal experience or lack thereof?

A score without reasons is useless, the e views should discuss pros and cons, ease of use, optical quality, and anybody flaws etc... The scores should in all likelihood be dropped, or a simpler scale 0 or 1. Basically would you recommend the lens for what you paid or not.

How difficult is it to rate something on a scale of 1 o 10? Farah Fawcett used to be considered The Perfect 10 but not everyone agreed on that. However, there are currently 6 categories for rating plus an overall rating and it has been that way for a while. Some people have gone back and updated there ratings pre-dating that an some haven't. On the rating, it shows that 1 is poor and somewhere between 5 and 6 is good with 10 being excellent. I think if anything, half number increments would be something of an improvement. Otherwise, just dealing with the odd outliers like a person that gives a lens a 1 just because some one else gave it a 10.

[tim60]

If a fault is typical for a particular lens in the market,
like FF/BF issues in a modern AF lens,
wobbly build in an FA 24-90,
or sticky blades on an old M42 lens,
it may be perfectly valid to review a copy with that fault.

For example, it could help someone contemplating
a BG or UG lens on KEH.

Thanks for the refinement of my comment. I was thinking of a small number of reviews where the damage was things like badly scratched elements, and even at least one with a cracked element. Such faults caused by misadventure of the lens mean that any observations are not a fair indication of the product. But the faults you refer to are the result of design or manufacture issues, and so are embedded in the product as shipped, and so are fair to include.

When I was young there was an Australian magazine, Choice, which published consumer reviews of products. They bought them retail anonymously, and so got the normal retail service (including after sales) and they tested operation under conditions that normal people would use the product, and also did some potentially destructive tests such as tugging on power cables. If they were testing a lens they might have included a drop test onto concrete (such as from table top height) because that is an event likely to happen during the life of a real lens. Also, reviews were of a set of competing products in a comparative format. Such a contrast to the magazines that do one at a time reviews of products that distributors have suppolied to them, and would never think of actually paying for a product over the counter.

The problem that most of us have for most products is that we are forced to rely on the information that suppliers choose to make available to us, or 'sponsored' reviews, and we can only afford to buy one of anything at a time, so we cannot buy all, try them to see which suits our need best, and dispose by way of secondhand trade, all the others.

I think the reviews here are useful, but like everything, we need to read the descriptive part to try to read between the lines to determine how to let the review influence our thinking.