[Belnan]

I am sure this topic has been beat to death but I just wanted to put this out there again. Looking at the lens rating of the Nikkor 85 1.8g on the d600 vs the da* 55 1.4 on the K-3. The d600 would appear to pummel the k-3. Even the d7100 sensor/lens combo would appear to be better than the K-3. However real world images would seem to suggest that DxO is out to lunch. I also noticed that Chromatic Abberations with the sigma 35 1.4 are higher on the K-3 than they are with the d7100. How is this possible? Is there some difference in the sensor design that affects this?

Pentax smc DA Star 55mm F1.4 SDM on Pentax K-3 versus Nikkor AF-S NIKKOR 85mm f/1.8G on - Side by side lens comparison - DxOMark

Nikon D600 vs. Pentax K-3 Image Comparison: Pentax Sweeps Nikon

[Rondec]

I am sure this topic has been beat to death but I just wanted to put this out there again. Looking at the lens rating of the Nikkor 85 1.8g on the d600 vs the da* 55 1.4 on the K-3. The d600 would appear to pummel the k-3. Even the d7100 sensor/lens combo would appear to be better than the K-3. However real world images would seem to suggest that DxO is out to lunch. I also noticed that Chromatic Abberations with the sigma 35 1.4 are higher on the K-3 than they are with the d7100. How is this possible? Is there some difference in the sensor design that affects this?

Pentax smc DA Star 55mm F1.4 SDM on Pentax K-3 versus Nikkor AF-S NIKKOR 85mm f/1.8G on - Side by side lens comparison - DxOMark

Nikon D600 vs. Pentax K-3 Image Comparison: Pentax Sweeps Nikon

My guess is that they have the same issue that PHotozone has, in that they just test one copy of a lens, or at most two copies. If they get a really good copy, then the tests look awesome and if they get an average copy then it doesn't look so good. That's the only explanation I can think of, particularly for the Sigma 35mm, since the lens itself, sensor should be the same.

My experience with the DA *55 was certainly quite a bit better than what some of the reviews say.

[normhead]

DxO seems to be a few guys who exist in a bubble. IN fact most of the testing sites weakness is, sample variation. Whereas lens rentals has contributed data that would suggest with many lenses, the sample variation can mean a sample of one lens can test better than a sample of another lens even though on average the second lens may be better.

Or in the case of DxO the same lens tests differently on different bodies, because they have a different copy of the lens.

To a person like myself, publishing data like this without an established sample variation and standard deviation is next to lying. It leads people to believe they can predict what kind of lens they are going to get, and which will be better to buy, based on faulty data. In DxO, every lens tested tests worse on a Pentax than it does on any other system.

But remember DxO is a commercial company. Their goal is to sell product. They aren't going acquire 10 copies of every lens ever made so they can do accurate comparisons. Unfortunately in this case, since they sell lens corrections etc. it makes you wonder if they actually have enough data about each lens to do what they think they are doing. Short answer, they are going to say they are doing a great job. More likely answer, with more money and time they could probably do a lot better.

Only answer that means anything to me..
They have done some testing, which is better than someone who hasn't done any testing, but their results must be viewed with a healthy degree of scepticism. One piece of information, but not the whole picture.

[Tarragon]

Hmmm, why can't camera and lens manufacturers just license this technology and provide the fixes 'in house' so we all benefit? I would love to see noise reduction automatically implemented within a pentax program. Maybe not in-camera as its too processor intensive, but surely as a plug-in perhaps? Surely a manufacturer that offers the best picture enhancement wins in sales??

[stevebrot]

Whereas lens rentals has contributed data that would suggest with many lenses, the sample variation can mean a sample of one lens can test better than a sample of another lens even though on average the second lens may be better.

You don't need to read many reviews on photozone.de to come to the same conclusion. Klaus frequently encounters lenses having centering defect or other problems that are not present on a second copy. He publishes the best result and makes a note of the other lens in his comments.


Steve

[monochrome]

My guess is that they have the same issue that PHotozone has, in that they just test one copy of a lens, or at most two copies. If they get a really good copy, then the tests look awesome and if they get an average copy then it doesn't look so good. That's the only explanation I can think of, particularly for the Sigma 35mm, since the lens itself, sensor should be the same.

My experience with the DA *55 was certainly quite a bit better than what some of the reviews say.

Who chooses the actual sample lens DXO tests? Does Canon/Nikon/Sigma/Tamron/Leica/Pentax send a lens or does DXO go out and randomly buy one or two in a store?

[Rondec]

Who chooses the actual sample lens DXO tests? Does Canon/Nikon/Sigma/Tamron/Leica/Pentax send a lens or does DXO go out and randomly buy one or two in a store?

Probably the second, but who knows? Certainly the tests are not sanctioned by the major brands and they aren't interested in selling DXO Mark's software. Doesn't seem like they are particularly interested in the DXO Mark tests as a marketing ploy, either. When the K5 was number one APS-C for dynamic range, I didn't see a whole lot of ads for it.

[derekkite]

This is sloppy sales rep stuff. These are essentially advertising for your product, so you have someone who is the contact and gives top quality lenses for testing. Yes from the standpoint of the buyer it makes these things useless, but they are a data point for buyers and if your people aren't there you lose.

Didn't dpreview get a K3 with kit zoom that was pretty terrible for testing? It came from Pentax iirc.

Sloppy.

[MJSfoto1956]

love 'em or hate em, one thing that is unmistakeable is that DxO testing does reveal TRENDS and data mining their published data can really help you to learn how your particular lens + body combination behaves as a system and how different f-stops change the behavior of the system. Viewed that way, what they have done is a real service.

[normhead]

love 'em or hate em, one thing that is unmistakeable is that DxO testing does reveal TRENDS and data mining their published data can really help you to learn how your particular lens + body combination behaves as a system and how different f-stops change the behavior of the system. Viewed that way, what they have done is a real service.

Or you could take pictures and finds out how changing the f-stops affects the system, like with your camera and your lenses. I don't think DxO publishes anything that saves you any time on the road to learning your own camera gear, but it's an interesting diversion down the road of "related information" if you choose to go that way.

Given that the lenses I've read all test worse on Pentax equipment than on there brands, in the few instances when the same lens is rated on two systems, the most logical conclusion would be don't buy Pentax. You'll get better performance with the same lens on another system.

There is one lens where the same Sigma or Tamron lens is rated for both the k-5 and D7000. The K-5 is rated a slightly better body, the lens is rated lower on the Pentax than on the D7000. DxO's data is not internally consistent. Until it is, it's like reading a historical novel, the facts are real, but the story line is made up. The story line being their ratings and commentary.

[MJSfoto1956]

which is why I am a big promoter of ALWAYS renting your gear (and carefully comparing it to a known quantity) before you buy. Thank goodness we have choices these days with LensRentals and BorrowLenses both offering lots of choices. In the past you had to cross your fingers and trust that your vendor had your best interests in mind. As far as how I test gear: I could care less about charts. I am more interested in how the gear handles and I always compare the results with a system that I know like the back of my hand. If the new gear can match the quality then I know that it will work for me. If the quality is less than my known system, then I also know that gear is not for me. Simple really. In the case of recently testing a Nikkor AF-S 80-400mm lens I had too many "non keepers" over three days of testing. All the Nikon guys I talked with said the same thing: the problem is with your technique. It can't be the LENS!! But the same scenes shot with the Pentax 60-250mm delivered a significantly higher number of keepers at the same settings. So which is it: bad photographer technique?, or finicky/touchy lens? Frankly, it doesn't matter: After 35 years as a professional photographer I'm not going to change the way I shoot and if I get better results with one system over another, then the choice is easy. IMHO, this is the best way to test equipment.

YMMV

Michael

[CDW]

I've found that no test site gets it right 100% of the time. I check out DXO, Photozone, DPR and Imaging Resource, as a bare minimum. I find Photozone lens reviews to be the easiest to interpret and they are good about letting the reader know how many samples they went through to get their results.

The one thing I find bothersome about noise graphs and DXO in particular, is the fact they don't give any details of the distribution of noise, simply a S/N figure that is the aggregate. In my former career in the high end audio industry we viewed noise components with a spectrum analyzer. However, it was often possible on a test bench to judge by listening whether the noise was low, mid or high frequency in nature. In the case of camera sensors and the related amplification circuitry around the sensor, the nature of the noise spectra determines the look of the noise (coarse, fine grain, film-like, etc.). We are left with JPG comparisons unless we take the time to download files, such as those from Imaging Resource, and do our own test prints. High frequency noise is analogous to the region that reveals detail in an image. Too much noise being generated in that region will obscure or reduce fine detail. A graphical representation of the noise spectrum would be nice to have from DXO reviews.

[normhead]

I've found that no test site gets it right 100% of the time. I check out DXO, Photozone, DPR and Imaging Resource, as a bare minimum. I find Photozone lens reviews to be the easiest to interpret and they are good about letting the reader know how many samples they went through to get their results.

As long as we keep that in perspective, everything is cool. If I lived in Europe, I'd be happy to send Klaus at photo zone a lens every now and then , and it would be cool to get organized and send him some money for Pentax testing. I don't think one person's donation makes a lot of difference. Despite what I consider to be his shoddy treatment of the 18-135, I still go to his site first, most of the time, after this site... it's probably not cool to organize a drive for a site that runs on donations, on another site that competes for those donations, so, that's pretty much a non-starter.

[Rondec]

As long as we keep that in perspective, everything is cool. If Iived in Europe, I'd be happy to send Klaus at photo zone a lens every now and then , and it would be cool to get organized and send him some money for Pentax testing. I don't think one person's donation makes a lot of difference. Despite what I consider to be his shoddy treatment of the 18-135, I still go to his site first, most of the time, after this site... it's probably not cool to organize a drive for a site that runs on donations, on another site that competes for those donations, so, that's pretty much a non-starter.

I think Klaus did his best with the 18-135. He had a bad copy, but he sent it to Pentax Germany and they said that it was up to standard.

All a reviewer can do is test things and then report the data that they get. And Klaus is really transparent about where his lenses come from and his overall testing procedure.

[Tarragon]

The one thing I find bothersome about noise graphs and DXO in particular, is the fact they don't give any details of the distribution of noise, simply a S/N figure that is the aggregate. In my former career in the high end audio industry we viewed noise components with a spectrum analyzer. However, it was often possible on a test bench to judge by listening whether the noise was low, mid or high frequency in nature. In the case of camera sensors and the related amplification circuitry around the sensor, the nature of the noise spectra determines the look of the noise (coarse, fine grain, film-like, etc.). We are left with JPG comparisons unless we take the time to download files, such as those from Imaging Resource, and do our own test prints. High frequency noise is analogous to the region that reveals detail in an image. Too much noise being generated in that region will obscure or reduce fine detail. A graphical representation of the noise spectrum would be nice to have from DXO reviews.

Well, I too come from a hi tech audio background, and at the end of the day you work with the gear you have. Ok, in audio there is expensive pro gear, good gear and cheap rubbish. But the difference between pro and good isnt that much and pros use "good" sometimes!

So, really, at the end of the analysis, a picture serves a purpose. If its good enough for a client/photographer who wants to pixel count? Unless you work for NASA where this stuff is ultra critical, just take a good photo and don't worry about it?

I love reducing ISO noise and reducing distortion, but I wouldn't loose sleep over a 1% difference in quality!