Originally posted by BrianR A Summary:
You've got a bunch of data points. If you try to summarize this bunch of data into a single value, you lose information that may or may not be important to the reader. However you try to summarize this bunch of data someone will complain about how you did it, even if you explain how you chose to summarize it. In particular, expect complaints when your method of summarizing doesn't support the conclusion that a particular person would like to see, especially when they can find a different method of summarizing that does (which is quite often possible if you try hard enough).
How to avoid this problem:
If you want the full, unbiased data to make a more informed decision, look at the full, unbiased data.
I guess you haven't seen the Lensrentals page where they scientifically tested sample variations on at least 10 lenses and plotted the results. Without standard deviations printed for each test, their results are pretty much meaningless. In fact any statistical analysis that doesn't include the standard deviation is meaningless. They haven't even established what the ranges are for each sample of the lens you intend to buy. They tell you what one lens they tested is like as far as I can tell. That doesn't tell you what the worst sample you might get or the best sample you might get would test. It tells you what one test was like, and therefore can't tell you what you're experience might be.
But is a lens is well rated on every platform, Pentax, Canon, NIkon, Nikon APS-c, Canon APS-c, like say the Sigma 35 1.4, then you know they;ve tested at least 3 lenses, making their tests a little more reliable. But in most cases you only find 3rd party lenses on non-Pentax cameras, and if you look at the ratings, lenses are rated lower on Pentax systems than on other systems.
There is no way these ratings can be considered in any way accurate, beyond the fact that on one camera, probably not a Pentax, one lens tested better than another lens. If everyone took DxOs advice, everyone would own Nikon D-800s. The best lenses have the highest scores on D-800s. The Pentax 645D is rated down with a K-5. These findings do not conform to the experiences of forum members. I personally cannot take DxO over the words and examples of forum members. Their measurements are too imprecise and based on the flawed notion, that the copy of the lens they have represents an average of the lens of the type being tested.
With no published standard deviation, no sample size listed, no range based on best and worst copies of the lens, no explanations for the different ratings on different sensors, no attempt to rationalize why the scores are so different on different bodies etc, their work is interesting but ultimately meaningless. I would ask people on the forum about a lens based on their numbers, but wouldn't go any further than that. I certainly wouldn't buy a lens or camera body, solely on what they publish.
They simply do not publish enough information for you to make an informed decision.