Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
08-11-2011, 10:26 PM   #1
Senior Member




Join Date: Aug 2009
Location: Houston, TX
Photos: Gallery
Posts: 141
Experiment with method to compute average ratings

This a follow-on to my earlier thread How accurate do you think the average ratings in the Lens Reviews are?. But since Adam has done a major overhaul on the lens rating system since then and added a number of sub-ratings I thought I would start a new thread to avoid confusion.

I think the new system is a major step forward. Hopefully it will end (or reduce) the confusion about how value relates to the rating. While conducting the experiment described below I came across one user who rated a kit zoom lens higher than a limited prime lens. Value has its place, but I think that's a bit extreme!

Although the new system is a big improvement, it doesn't address the issue of different reviewers having intrinsically different scales. At one end, there are people who have only reviewed the kit lens, and at the other are people who have only reviewed limited lenses (lucky devils). There's also "ballot stuffing". I came across people who said (paraphrasing) "I'm giving an extreme score to counteract the scores I don't agree with" and there are people who submitted multiple reviews for the same lens all with the same high score.

To see if there's a smarter way to compute averages that minimizes these effects, I conducted a little experiment. I scraped the reviews for a small number of lenses (listed below) and extracted the reviewer and their rating for each lens. You might want to ignore the following details on first reading. Reviewers who only rated one lens or who gave the same rating to every lens they reviewed were ignored. For the others I scaled their ratings to the range 0 (for the lowest rating they assigned) to 1 (for the highest). From these scaled ratings I then computed an average lens rating. I then computed an average user rating by averaging the lens ratings for all the lenses they reviewed. I then iterated this procedure also taking into account the user rating when determining the lens ratings, until the lens ratings converged. The lens ratings thus obtained are not on a natural scale, so I transformed them to have the same mean and variance as the original average lens ratings.

The simple average rating and the new rating for the small subset of lenses I applied it to are shown in the following table:

HTML Code:
Lens                                          New rating  Old rating
SMC-Pentax-FA-31mm-F1.8-Limited 10.08 9.71
SMC-Pentax-FA-77mm-F1.8-Limited 10.03 9.77
SMC-Pentax-DA-Star-50-135mm-F2.8-SDM-Zoom 9.67 9.59
SMC-Pentax-FA-43mm-F1.9-Limited 9.57 9.52
SMC-Pentax-DA-40mm-F2.8-Limited-Pancake 9.27 9.67
SMC-Pentax-DA-70mm-F2.4-Limited 9.06 9.42
SMC-Pentax-FA-Star-24mm-F2 8.98 9.37
SMC-Pentax-FA-50mm-F1.4 8.94 8.86
SMC-Pentax-DA-18-135mm-F3.5-5.6-ED-AL-IF-DC-WR 8.74 8.19
SMC-Pentax-DA-18-250mm-F3.5-6.3-Zoom 8.40 8.65
SMC-Pentax-DAL-55-300mm-F4-5.8-Zoom 8.30 8.47
SMC-Pentax-DA-18-55mm-F3.5-5.6-II-Version-2-Zoom 8.25 8.23
SMC-Pentax-DAL-50-200mm-F4-5.6-Zoom 7.56 7.40
SMC-Pentax-DAL-18-55mm-F3.5-5.6-Zoom 7.29 7.31
Overall, I'm pretty surprised about how similar the two scores are. I'm happy that the DA 18-135 has moved up relative to the 18-55 kit lenses. I feel happier knowing that the different rating scales of the various reviewers doesn't make much of a difference. But does it change anything significantly enough to be worthwhile?

08-11-2011, 11:22 PM   #2
Pentaxian
calsan's Avatar

Join Date: Nov 2008
Location: Perth, Australia
Photos: Gallery
Posts: 1,370
Mode

I try and guess the "mode" score, if there are enough ratings: for example if a lens is rated by ten people as:

8, 8, 7, 5, 1, 8, 9, 10, 10, 8

The average would be 7.4, dragged down by the score of 1, but the mode (most commonly rated score) would be 8. More people think it's an 8 than a 7.4, so in probability, it's actually an 8.

08-11-2011, 11:25 PM   #3
Pentaxian




Join Date: Dec 2007
Location: Prague
Photos: Gallery
Posts: 1,198
It is interesting to have score better than 10 for few exceptional lenses. But I'd rather wait and see what the new system Adam installed is doing. Let's give it time to settle before making further modifications.
08-12-2011, 05:33 AM - 1 Like   #4
Site Supporter
boriscleto's Avatar

Join Date: Dec 2009
Location: Liverpool, NY
Photos: Gallery
Posts: 12,469
xkcd: TornadoGuard



08-12-2011, 06:25 AM - 1 Like   #5
Forum Member




Join Date: Jul 2011
Location: New Mexico
Photos: Albums
Posts: 74
QuoteOriginally posted by calsan Quote
I try and guess the "mode" score, if there are enough ratings: for example if a lens is rated by ten people as:

8, 8, 7, 5, 1, 8, 9, 10, 10, 8

The average would be 7.4, dragged down by the score of 1, but the mode (most commonly rated score) would be 8. More people think it's an 8 than a 7.4, so in probability, it's actually an 8.

I second this. Plus, I like to quickly compute and/or estimate the median value: the value at which 50% of the ratings are above and below. In the example cited, with 10 ratings, sorted from low to high, the 5th rating is an 8 --- hence, the median.

If the ratings are normally distributed, the mean, median, mode will all be equal. But, as calsan has pointed out, if you have one or more "outlier" values (like the 1) central tendency measures other than the mean are helpful in my eyes.

Aged statistician here who can't help himself!
08-12-2011, 07:58 AM   #6
Senior Member




Join Date: Aug 2009
Location: Houston, TX
Photos: Gallery
Posts: 141
Original Poster
Let's say we have three reviewers.

Albert is a cranky old guy who only ranks top quality lenses and will downrate a lens for the slightest difference from absolute perfection. He rates a few quality lenses as follows: FA31, 9, FA77, 8, FA43 7, and FA24, 6. (I said he was fussy.)

Charlie has just switched to a Pentax SLR from his Canikon P&S and thinks the kit lenses are fantastic. He rates them as follows: DAL50-200, 10, DAL18-55, 9.

If there are enough Alberts and Charlies in the population, their ratings can also skew the mode and median of the lens ratings. I think there are lots of Charlies and quite a few Alberts (perhaps not quite so extreme) out there, but to get some real data I will compute the mode and median ratings for the lenses I looked at and post them in the next day or so. Neither the Alberts or Charlies are "wrong", they just have different perspectives.

So, to understand just what an average (or mode or median) rating for a new lens really means, you have determine whether the reviewers of that lens are dominated by the Alberts or the Charlies. That is a lot of effort to do manually and a "best guess" would probably be pretty subjective.

However, it is what the rating calculation described above is attempting to do computationally, by also considering the Brendas who rate all types, for instance FA77 10, FA24 9, and DAL18-55, 7, to put the ratings from the Alberts and Charlies into better perspective. I think it would be wrong to classify the rating calculation above as a kind of average.

Conceptually, I think the issue is very similar to that of college grade inflation, with the lenses equivalent to the students, and the reviewers equivalent to the courses/professors. The rating calculation above is much simpler than the ones suggested in the statistics literature for addressing grade inflation.
08-12-2011, 08:03 AM   #7
Senior Member




Join Date: Aug 2009
Location: Houston, TX
Photos: Gallery
Posts: 141
Original Poster
QuoteOriginally posted by elho_cid Quote
But I'd rather wait and see what the new system Adam installed is doing. Let's give it time to settle before making further modifications.
I agree, but I did the experiment and thought I would post the results while they were still relatively fresh.
08-12-2011, 09:47 AM   #8
Pentaxian




Join Date: Feb 2010
Location: Eureka, CA
Photos: Gallery | Albums
Posts: 1,875
QuoteOriginally posted by OutOfFocus Quote
While conducting the experiment described below I came across one user who rated a kit zoom lens higher than a limited prime lens. Value has its place, but I think that's a bit extreme!
That's a good point. The value category does tend to favor cheap or inexpensive glass at the expense of costlier glass. But there are problems with some of the other categories as well, particularly relative to other categories. Is how well a lens handles, or how well it auto-focuses, really as important as its sharpness? How important is bokeh in a wide angle f4 prime, or in a fisheye lens, compared to its importance in an f1.4 or f1.2 prime? For me, the most important factor in a lens, the factor which makes all other wilt into insignificance in comparison, is the quality of the images that it produces; and of the categories making up the sub-ratings, only sharpness and bokeh really bear on that issue in a significant way (since most "abberations" can be removed or significantly reduced in post). Nor is there anything about color rendition or contrast, which, in a lens like the DA 10-17, may play a more important role in image quality than abberations or bokeh.

08-12-2011, 11:06 AM   #9
Veteran Member
RioRico's Avatar

Join Date: Jun 2008
Location: Limbo, California
Posts: 11,264
I HEREBY PROPOSE: That the 'average' user ratings display mean, median, and mode. That's simple, and will give us a quick eyeball overview of the ratings. (Don't include standard deviation, that'll explode too many heads.)
08-12-2011, 12:12 PM   #10
Pentaxian
Lowell Goudge's Avatar

Join Date: Jan 2007
Location: Toronto
Photos: Gallery | Albums
Posts: 15,342
QuoteOriginally posted by OutOfFocus Quote
This a follow-on to my earlier thread How accurate do you think the average ratings in the Lens Reviews are?. But since Adam has done a major overhaul on the lens rating system since then and added a number of sub-ratings I thought I would start a new thread to avoid confusion.

I think the new system is a major step forward. Hopefully it will end (or reduce) the confusion about how value relates to the rating. While conducting the experiment described below I came across one user who rated a kit zoom lens higher than a limited prime lens. Value has its place, but I think that's a bit extreme!

Although the new system is a big improvement, it doesn't address the issue of different reviewers having intrinsically different scales. At one end, there are people who have only reviewed the kit lens, and at the other are people who have only reviewed limited lenses (lucky devils). There's also "ballot stuffing". I came across people who said (paraphrasing) "I'm giving an extreme score to counteract the scores I don't agree with" and there are people who submitted multiple reviews for the same lens all with the same high score.

To see if there's a smarter way to compute averages that minimizes these effects, I conducted a little experiment. I scraped the reviews for a small number of lenses (listed below) and extracted the reviewer and their rating for each lens. You might want to ignore the following details on first reading. Reviewers who only rated one lens or who gave the same rating to every lens they reviewed were ignored. For the others I scaled their ratings to the range 0 (for the lowest rating they assigned) to 1 (for the highest). From these scaled ratings I then computed an average lens rating. I then computed an average user rating by averaging the lens ratings for all the lenses they reviewed. I then iterated this procedure also taking into account the user rating when determining the lens ratings, until the lens ratings converged. The lens ratings thus obtained are not on a natural scale, so I transformed them to have the same mean and variance as the original average lens ratings.

The simple average rating and the new rating for the small subset of lenses I applied it to are shown in the following table:

HTML Code:
Lens                                          New rating  Old rating
SMC-Pentax-FA-31mm-F1.8-Limited 10.08 9.71
SMC-Pentax-FA-77mm-F1.8-Limited 10.03 9.77
SMC-Pentax-DA-Star-50-135mm-F2.8-SDM-Zoom 9.67 9.59
SMC-Pentax-FA-43mm-F1.9-Limited 9.57 9.52
SMC-Pentax-DA-40mm-F2.8-Limited-Pancake 9.27 9.67
SMC-Pentax-DA-70mm-F2.4-Limited 9.06 9.42
SMC-Pentax-FA-Star-24mm-F2 8.98 9.37
SMC-Pentax-FA-50mm-F1.4 8.94 8.86
SMC-Pentax-DA-18-135mm-F3.5-5.6-ED-AL-IF-DC-WR 8.74 8.19
SMC-Pentax-DA-18-250mm-F3.5-6.3-Zoom 8.40 8.65
SMC-Pentax-DAL-55-300mm-F4-5.8-Zoom 8.30 8.47
SMC-Pentax-DA-18-55mm-F3.5-5.6-II-Version-2-Zoom 8.25 8.23
SMC-Pentax-DAL-50-200mm-F4-5.6-Zoom 7.56 7.40
SMC-Pentax-DAL-18-55mm-F3.5-5.6-Zoom 7.29 7.31
Overall, I'm pretty surprised about how similar the two scores are. I'm happy that the DA 18-135 has moved up relative to the 18-55 kit lenses. I feel happier knowing that the different rating scales of the various reviewers doesn't make much of a difference. But does it change anything significantly enough to be worthwhile?
this is all well and good, BUT what is needed is to compare the pentax lenses, which many are prone to rate highly (being pentax snobs to some extent), against other brands, or against older pentax legacy lenses.

the question will not be as much whether the FA77 is better than the DA 70 for example, but how these two compare with the A 85/1.4, the samyang 85/1.4 and perhaps a super tak 85/1.9.

If there is no clear winner with that spread of samples then clearly the criteria is still too prone to personal preferences.
08-13-2011, 06:49 AM   #11
Senior Member




Join Date: Aug 2009
Location: Houston, TX
Photos: Gallery
Posts: 141
Original Poster
Here is the expanded table I promised above. I also included a few popular non-Pentax lenses. Note the 'new' ratings have changed for the lenses previously included, even though I didn't update the original scores on which they were based. This is a feature, but perhaps a confusing one.

HTML Code:
Lens                                                N    New    Avg  TMean Median  Mode(s)
sigma-70mm-f2-8-ex-dg-macro 10 10.20 9.80 10.00 10.0 [10]
SMC-Pentax-FA-31mm-F1.8-Limited 38 10.04 9.71 9.91 10.0 [10]
SMC-Pentax-FA-77mm-F1.8-Limited 52 10.00 9.77 10.00 10.0 [10]
SMC-Pentax-DA-Star-50-135mm-F2.8-SDM-Zoom 66 9.77 9.59 9.85 10.0 [10]
SMC-Pentax-FA-43mm-F1.9-Limited 44 9.48 9.52 9.69 10.0 [10]
SMC-Pentax-DA-40mm-F2.8-Limited-Pancake 72 9.28 9.65 9.77 10.0 [10]
SMC-Pentax-DA-70mm-F2.4-Limited 50 9.06 9.42 9.50 9.5 [10]
SMC-Pentax-FA-50mm-F1.4 93 8.96 8.86 8.98 9.0 [9]
SMC-Pentax-FA-Star-24mm-F2 19 8.89 9.37 9.64 10.0 [10]
tamron-af-17-50mm-f2-8-xr-ld-aspherical-if-sp 32 8.86 8.59 8.95 9.0 [9]
sigma-105mm-f2-8-ex-dg-macro 19 8.86 9.32 9.36 9.0 [9]
tamron-70-200mm-f2-8-di-ld-macro 29 8.84 8.93 9.24 9.0 [9]
tamron-28-75mm-f2-8-af-xr-di-ld-macro-sp 36 8.73 8.72 9.00 9.0 [9]
SMC-Pentax-DA-18-135mm-F3.5-5.6-ED-AL-IF-DC-WR 16 8.68 8.19 8.60 9.0 [9]
sigma-70-200mm-f2-8-ii-ex-dg-apo-macro-hsm 21 8.44 8.62 8.85 9.0 [9]
SMC-Pentax-DA-18-250mm-F3.5-6.3-Zoom 31 8.42 8.65 8.63 9.0 [9]
SMC-Pentax-DAL-55-300mm-F4-5.8-Zoom 15 8.38 8.47 8.56 9.0 [9]
SMC-Pentax-DA-18-55mm-F3.5-5.6-II-Version-2-Zoom 35 7.98 8.23 8.24 8.0 [8]
tamron-70-300mm-f4-5-6-af-di-ld-macro 30 7.93 7.50 7.72 8.0 [8]
sigma-70-300mm-f4-5-6-apo-dg-macro 20 7.74 7.95 8.00 8.0 [8]
SMC-Pentax-DAL-50-200mm-F4-5.6-Zoom 5 7.70 7.40 7.33 7.0 [7]
SMC-Pentax-DAL-18-55mm-F3.5-5.6-Zoom 26 7.31 7.31 7.56 8.0 [8]
The columns in the table are:

Lens name
Number of ratings used (differs from number of reviews because of N/As)
New method
Average rating
Trimmed average (sort, remove top and bottom 20%, mean of what's left)
Median
Mode(s) in general can have more than one, but not in any of these cases
08-13-2011, 08:37 AM   #12
Senior Member




Join Date: Aug 2009
Location: Houston, TX
Photos: Gallery
Posts: 141
Original Poster
Something like one of the little utilities I wrote to evaluate the sensibility of the above rankings might be a useful addition to the reviews.

For any specific lens, it outputs a table of all those people who have reviewed both it and at least one other lens and shows all of their ratings in a table grouped by reviewer.

This could be pretty easily added as an additional link on the top of the review page for each lens, and the entries in the table could link back to the individual reviews concerned.

This would let you easily compare the lens's ratings to others by the same reviewers, but do the hard work of collecting all the relevant reviews.
08-14-2011, 07:42 AM   #13
Forum Member




Join Date: Jul 2011
Location: New Mexico
Photos: Albums
Posts: 74
QuoteOriginally posted by RioRico Quote
I HEREBY PROPOSE: That the 'average' user ratings display mean, median, and mode. That's simple, and will give us a quick eyeball overview of the ratings. (Don't include standard deviation, that'll explode too many heads.)
Ditto. I agree.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
experiment, k-mount, kit, lens, lenses, pentax lens, people, ratings, reviewers, reviews, slr lens, system, user
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
How accurate do you think the average ratings in the Lens Reviews are? OutOfFocus Pentax SLR Lens Discussion 47 08-06-2011 07:18 PM
Average height Nesster General Talk 5 04-29-2011 10:37 PM
Average time spent? Leana Pentax DSLR Discussion 18 02-20-2011 08:32 PM
K-7 and below average dynamic range harleynitelite Troubleshooting and Beginner Help 9 01-18-2010 01:05 AM
Nature Not your average seagull - M400 F5.6 goddo31 Post Your Photos! 11 11-12-2009 05:47 AM



All times are GMT -7. The time now is 04:28 PM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top