[Blue]

Since the ratings consist of just a single number, you have to read the reviews to get an accurate sense of the optical quality of a lens. It would be better if we had multiple scores, i.e. one for each parameter as we have in the in-depth reviews.

I agree. Something like 5 criteria with each worth points.

1. Build Quality
2. Optical Quality & Performance
3. Mechanical (speed, internal focus, internal zoom, focus, focus reliablity, aperture rings etc)
4. Price to Performance rating
5. "character" (Bokeh, blurr etc. rareness, unique ability such as tilt)

In addition, I think allowing a rating of 7.5, 8.5 and 9.5 would help a lot. However, I think a scale of 0-5 would be more accurate given the subjective nature of a rating. I think going to 10 implies more resolution than can realistically be expected.

---

Last edited by Blue; 08-05-2011 at 09:30 AM.

[Blue]

. . .
The first anomaly I noticed is that the DA 18-55 F3.5-5.6 AL WR is rated substantially worse than the DA 18-55 F3.5-5.6 AL II (7.63 from 27 reviews versus 8.26 from 35 reviews, respectively, at the time of writing). Since these are supposed to have the same optics, why the big discrepancy? In fact, the WR is only slightly better rated than the original version (7.42 from 64 reviews), and everything I've read suggests it has much better optics, and it's WR as well.

The second anomaly I noticed is that the DA 18-135 F3.5-5.6 WR is rated slightly lower than the DA 18-55 AL II mentioned above (8.14 from 14 reviews versus 8.26 from 35 reviews, respectively, at the time of writing). All the reviews of the DA 18-135 that compare it to the DA 18-55 suggest it's comparatively much sharper, plus it has WR and the new DC motor which people seem to like. So seeing it rated slightly lower is a bit surprising. (It is rated higher than the original and WR versions of the 18-55 lens though.)

So are these valid observations, or is there something I've missed? Are their any other examples where the consensus of opinion is that lens A is better than lens B, but the average rating of B is higher than that of A? How big a deal is this to you, if at all?

Regarding the 18-55, I think the biggest problem is that not everyone has used all 4 versions of that lens. I myself have used the 3 quick shift versions. The optical formula was changed with the release of the AL II and I could tell a difference. I currently have the WR and while it is better due to the sealing, I can't say it is better or worse than the II. I can't say about the DA L version. However, in my above suggested draft scheme, the mechanical rating would allow things like quick shift, WR and metal mount to differentiate the ratings. Optical performance and build quality would be addressed in a different rating.

As far as the 18-135, I believe what shocks a lot of people is the near $500 price point of that lens and I have to question whether it is nearly 4x better than what I paid for my DA 18-55 WR. Plus, it is larger especially when extended.

[bjake]

More reviews would be good.One part of the reviews that I like is the size-weight as smaller and lighter work better for me.
Jake

[Lowell Goudge]

at one point I suggested that the rating shoud be relitive to the price, i.e. an 8 is meeting all your expectations for the price, 9's and 10's would only be given if the lens exceeded expectations for the price.

In fact I think this is still true, but for overall ranking, I think the overall ranking, or comparison of 2 completely differently priced lenses should be something like the

lens score = rating (for price on a scale of 0-10) x Square root of price
the reason in my opinion is that for improved quality of image and other lens performance points, the price curve rises very sharply indeed.

[Gareth.Ig]

I agree. Something like 5 criteria with each worth points.

1. Build Quality
2. Optical Quality & Performance
3. Mechanical (speed, internal focus, internal zoom, focus, focus reliablity, aperture rings etc)
4. Price to Performance rating
5. "character" (Bokeh, blurr etc. rareness, unique ability such as tilt)

In addition, I think allowing a rating of 7.5, 8.5 and 9.5 would help a lot. However, I think a scale of 0-5 would be more accurate given the subjective nature of a rating. I think going to 10 implies more resolution than can realistically be expected.

What would be really cool is if you could then develop your own weightings for each of the categories, you could then have an aggregate ranking customized to your personal preferences.

[Lowell Goudge]

What would be really cool is if you could then develop your own weightings for each of the categories, you could then have an aggregate ranking customized to your personal preferences.

but the problem with the top three things is they are a function of price. therefore should all be lumped into item 4 or individually ranked for price.

As an example, the samyang 85/1.4 has a plastic lens barrel. but at about 329 new, should get ranked potentially higher than a more expensive pentax DA lens with a plastic barrel, because plastic is cheaper to manufacture with than metal, but others don't see it that way.

as for blue's 5th item bokeh, this is one area where a separate ranking can help. but the thing is every one views this differently.

[paperbag846]

as for blue's 5th item bokeh, this is one area where a separate ranking can help. but the thing is every one views this differently.

Yeah. The more I read about bokeh the less sure I am of what it even IS. There are people who would, for example, consider the FA 77's cream good bokeh, while others would prefer the pointillism-inspired abstraction that the FA 43 tends to render. Others might prefer the general softness of the FA 50 (and then you could discuss whether you prefer the 1.7 or 1.4). Others still would like the subtle bokeh of the DA 40.

Its just so subjective I'm wondering if we should just have a rating for "rendering" and leave it at that. Sure it would be ambiguous, but anything more specific might be misleading.

[Blue]

Yeah. The more I read about bokeh the less sure I am of what it even IS. There are people who would, for example, consider the FA 77's cream good bokeh, while others would prefer the pointillism-inspired abstraction that the FA 43 tends to render. Others might prefer the general softness of the FA 50 (and then you could discuss whether you prefer the 1.7 or 1.4). Others still would like the subtle bokeh of the DA 40.

Its just so subjective I'm wondering if we should just have a rating for "rendering" and leave it at that. Sure it would be ambiguous, but anything more specific might be misleading.

That's why I used the title "character" because there are other things besides bokeh that can give a lens "character." One example here might be the aperture blades in the D FA 100 vs the D FA 100 WR. Another example may be the unique optical design of the Takumar 58mm/2.4 (preset) in addition to its uniqueness and rarity.

[Blue]

but the problem with the top three things is they are a function of price. therefore should all be lumped into item 4 or individually ranked for price.

As an example, the samyang 85/1.4 has a plastic lens barrel. but at about 329 new, should get ranked potentially higher than a more expensive pentax DA lens with a plastic barrel, because plastic is cheaper to manufacture with than metal, but others don't see it that way.

as for blue's 5th item bokeh, this is one area where a separate ranking can help. but the thing is every one views this differently.

No, the top 3 things are NOT a function of price as far as the rating goes. That's what #4 is for. Build quality and mechanical design were also separated. A brick sh--house is better built than a marine plywood one which in turn is better built than a standard plywood one. Therefore, the brick one should come out better rated in build quality. The flapper inside would come under mechanical and is a separate evaluation. The price to performance is a separate evaluation. If a dump is a very pleasant experience in the brick building but not so much in the cheapest plywood one, perhaps the money was just thrown down the crapper on the lesser one. Perhaps the one in the middle is a better compromise in situations with dimensioning returns. Every one privy to such things know that a brick s---house is built better than a cheap grade plywood one.

[RioRico]

Factoring-in price is inconsistent for new vs old vs used lenses. Does an F35-70 I paid US$10 for last year rank higher than one someone bought new in 1990 for US$150 or whatever? How do I relatively rank a couple of used SuperTak's I got a couple years ago, the 55/2 for US$20 and the 55/1.8 for US$7? (For those who don't know: these have EXACTLY THE SAME OPTICS AND BUILD, just a microscopically different maximum aperture.) If I were to plot price-performance, all my "good deals" (that I persevere to obtain) certainly gain, vs their bloated market 'value' -- like, do I overvalue my SuperTak 105/2.8 because I got it for US$18 instead of US$180? And the few dogs for which I overpaid, and sold for too little -- do I let my bitterness over my losses influence their ratings? Just wondering...

[Sagitta]

Echoing reading words over numbers, but if you see a lens with 40 reviews and a rating of 6, that number is going to have more impact and accuracy than a lens with 5 reviews and a rating of 6.

If I had gone strictly by the numbers, I'd never have dropped that whopping $10 for the FA 28-90. But for 10 bucks, well.. I skewed the score up almost a point with *my* rating because unless the lens crawled off the camera and started poking me in the eye, there is no way its going to get a low score from me for the output it generates.

EDIT: Same goes for a lot of manual lenses. People low-score them because of the lack of autofocus. IMHO you can't lowball a lens made in an age when a technology did not exist for not having said technology. If I'm buying a lens made in 1968 I'm not buying it expecting it to autofocus on my camera manufactured in 2008.

[pacerr]

I've never considered the 10-point rating system to be a useful description of the desirability of acquiring a given lens for the many good reasons noted above. Sample images do provide a general example of the potential results possible with a specific lens and meaningful verbal comments may highlight specific, physical characteristics such as build and handling qualities, an abnormal tendency for CA/PF/flare, etc.

But if a rating system seems necessary and irresistible to many users, I'd prefer one such as this:

If you had significant experience with a given type lens (why review it if you don't?) and a friend asked to borrow it, would you describe the expected potential results to be obtained with it as being --
1. unsatisfactory
2. satisfactory
3. exceptionally noteworthy

Ten or more reviews would have to be published before a simple, subjective rating of un-sat / sat / exceptional would be displayed and those would be provided in descriptive form, i.e., U=1/S=7/E=2/tot=10. This could be extracted from/displayed as a poll rather than a running statistical average.

The comments section would categorically describe specific traits that can be objectively compared to an accepted standard or to a comparable lens and point out specific traits for better or worse.


IMO, averaging the cost of a lens over time serves no reliable purpose and is best left to research of actual recent sales. Unverifiable valuations can contribute to manipulation of the used market and astute buyers and sellers tend to ignore them anyway. If the community actually wants a valuation list, retaining the final sales prices in the Marketplace paired with public auction site results would provide a more relevant and reliable resource.

H2

[magkelly]

Some things can be awfully subjective. The Sigma 70-300 I just bought is a little soft full out at 300MM at some apertures, as most of the reviews I saw stated, but that's pretty typical of most long lenses from what I have seen and I didn't automatically think that made the lens bad, though some reviews I read disagreed. Nor do I think the lens is built cheaply and there were several people on another board who were dissatisfied with that apparently.

Every lens has it's sweet spot it's drawbacks as well as it's strengths, ditto most cameras. I've actually seldom run across a truly bad lens that was totally useless. Once or twice, but overall, not many. Most of the time I know better than to blame a lens for my own failure to use it as well as I could. It takes time to learn to use a lens well sometimes. I have had a couple of lenses that at first I thought were not so great actually turn out to be a lot better than I thought. My Tamron 28-200MM for one.

On my *ist I seemingly couldn't get much decent out of that lens. But unbeknownst to me my *ist had issues. On my K-x, that same lens is a whole other beast. I've actually gotten a number of images with it now that I really like. It's a pretty good lens actually on the right camera. If I had gone with my initial impression on that lens I'd have rated it pretty badly. As it turns out that wouldn't have been fair at all because ultimately it wasn't the lens but my old *ist that simply wasn't up to snuff.

Human error and unrealistic expectations can make for a lot of unjustly bad reviews from what I've seen and let's face it some people are just way too picky and critical too. There are a lot of people who expect perfection out of mechanical devices that simply have their limits. When they don't get it? They blame everything but themselves for not getting it.

A lens is just a lens, a camera is just a camera. If you're not really that good of a photographer yet having the most expensive equipment in the world won't help you with that. Quite the contrary actually if you are using equipment that you don't know well or don't have the skills to use yet it can make you feel like a total idiot until you get the hang of it. But blaming the camera, or the lenses? That's just what a lot of people do. They don't want to stop to think that maybe it's more lens or camera than they are capable of using at that moment and they review things accordingly.

I always read as many reviews for whatever I want to buy as I can, but ultimately I also look at a lot of pics taken with it, preferably by a good photographer too. Words can be subjective, but a few good pics will usually tell me what I want to know far better than most reviews...

[SpecialK]

that is a little harsh.

Perhaps so, as some Amazoners have no clue :-)

[EsBee]

I don't think they represent any level of accuracy unless the number of reviews is high -- but even then it is debatable.

Since there are no real guidelines for reviewers to follow, I now think most individual reviews are random in the grand scheme of things -- some based on emotion, some on impulse and others based on some real intelligent and hard work that went into assessing the product's capability for which they were built. Even then, the final review provided may still be contaminated with intangibles such as price, packaging, delivery and service by the vendor.

However, if the population of reviewers is large enough, statistically you could take the median to be somewhat representative of the overall assessment.

In the beginning I used to very enthusiastically provide reviews on products that I purchased and tried, and even go back and change them at times. Later I realized that I probably did no good to prospective buyers by contaminating already less than accurate set of reviews. Then I also noticed, more than a few times, excellent reviews showing up for products by reviewers who soon after had the same products for sale in the Marketplace!

That to me was the end of PF reviews as a consumer and reviewer, except in cases where the number of reviewers is very large and contains reviews by those whom I know as experts -- based on my own observations.