Pop Photo July issue

flyer · 06-14-2008, 04:46 AM

In the july issue, Pop Photo has a comparison test of high ISO cameras. In the test results, they rated the noise level of the K20D at ISO 3200 and 6400 as unacceptable. What ticks me off is the fact (stated in the article) that they left the NR off on the K20D while all the other contenders had high ISO N.R. on. It shows that Pop Photo has a strong dislike of Pentax cameras to use such tricks to discredit a model. They should use the same conditions for all the cameras they test for a particular issue. They implied that they left the camera in the state it comes in the box (default setting), but they turned the high ISO on on the Olympus E-3. If you look only at the results without reading the whole article, it could mislead a lot of readers. Why turn high ISO n.r. on for the Olympus and not do it for the K20D? POP PHOTO, YOU STINK

**monochrome** · 06-14-2008, 05:31 AM

I don't take their (or any) photography magazine for just this reason, but I'd bet Pentax doesn't advertise enough (pay enough bribes) to be treated fairly - this is a form of blackmail.

RiceHigh · 06-14-2008, 05:52 AM

So, what's the big deal?

Even Dpreview do tests at the default factory settings. It is just impossible to try each combination for they to carry out the review. Does an individual setting require an individual test report?

If it is the factory default, it is fair enough to believe that it is the best setting as recommended by the maker, isn't it?

Anyway, I would say those noise tests nowadays by the measurbators are not so meaningful as long as noises are measured along instead of the Signal-to-Noise (aka S/N) ratio.

Take the following DPR posted high ISO shootout image for example:-

https://www.pentaxforums.com/forums/pentax-dslr-discussion/29317-14-mpix-high-iso-shoot-out.html

I must say I am impressed with the details and rich colours retained by the K20D at high ISO, although it is obviously noisy from ISO 800 already. (The 450D and A350 are far less noisy, but there are no good IQ at all (as Phil Askey and his guy seem to be right in what they are presenting))

Originally posted by flyer

In the july issue, Pop Photo has a comparison test of high ISO cameras. In the test results, they rated the noise level of the K20D at ISO 3200 and 6400 as unacceptable. What ticks me off is the fact (stated in the article) that they left the NR off on the K20D while all the other contenders had high ISO N.R. on. It shows that Pop Photo has a strong dislike of Pentax cameras to use such tricks to discredit a model. They should use the same conditions for all the cameras they test for a particular issue. They implied that they left the camera in the state it comes in the box (default setting), but they turned the high ISO on on the Olympus E-3. If you look only at the results without reading the whole article, it could mislead a lot of readers. Why turn high ISO n.r. on for the Olympus and not do it for the K20D? POP PHOTO, YOU STINK

RiceHigh · 06-14-2008, 05:55 AM

Also, we actually don't need to read the new issue to compare their *old* results. A quick research on the old published issue will give us the info (which I did that already):-

RiceHigh's Pentax Blog: First K20D Production Camera Formal Test

Originally posted by flyer

In the july issue, Pop Photo has a comparison test of high ISO cameras. In the test results, they rated the noise level of the K20D at ISO 3200 and 6400 as unacceptable. What ticks me off is the fact (stated in the article) that they left the NR off on the K20D while all the other contenders had high ISO N.R. on. It shows that Pop Photo has a strong dislike of Pentax cameras to use such tricks to discredit a model. They should use the same conditions for all the cameras they test for a particular issue. They implied that they left the camera in the state it comes in the box (default setting), but they turned the high ISO on on the Olympus E-3. If you look only at the results without reading the whole article, it could mislead a lot of readers. Why turn high ISO n.r. on for the Olympus and not do it for the K20D? POP PHOTO, YOU STINK

jsherman999 · 06-14-2008, 06:28 AM

Originally posted by RiceHigh

...
Anyway, I would say those noise tests nowadays by the measurbators are not so meaningful as long as noises are measured along instead of the Signal-to-Noise (aka S/N) ratio.
...

You're right about that, RH, so many people seem to not understand the S/N ratio concept - and I seriously mean really do not understand it. I'm starting to come to the dismaying conclusion that even some semi-pro reviewers for publications like Pop Photo don't quite get the concept - which would be equivalent to a writer for Car & Driver not quite getting the concept of, say, fuel injection.

If you're a reviewer, it's your responsibility to know this stuff, and to not misrepresent measured results by using out-of-context frames of reference.

The S/N ratio of the K20D is nearly class-leading, if not at the very top of it's class. The fact that this has not been spelled out in big bold letters means that reviewers either a) discount S/N inappropriately, b) don't really fully understand the concept, and thus skim over it, or c) assume everyone already understands it, so it doesn't need to be discussed. Meanwhile, it continues to be misunderstood by a good portion of the buying public as well as a good number of secondary reviewers/ review referrers.

.

cupic · 06-14-2008, 06:38 AM

You cant compare apples to pears

cheers

flyer · 06-14-2008, 06:41 AM

Originally posted by RiceHigh

So, what's the big deal?

The BIG DEAL is that the Olypus E-3 comes off with N.R. off as the default setting and they turned it on. They can't even respect their own protocol, which makes them as biased as you. That is the point.

Maxington · 06-14-2008, 08:08 AM

I never understood the point of this "TEST AT OUT OF BOX DEFAULT SETTINGS" business. Ok, from a practical point of view, they might not have time or ability to test EVERY setting, sure.

But its a DSLR. It is absolutely meant to have its settings changed, fiddled with, adjusted to suit the person, the shot, the place, the lighting, you name it. That's the entire point of the camera.

You find a major flaw, you go look if there is a setting to change. This is obvious, its what you do with anything you buy. Don't you change the setting of the seats in a new car? Fiddle with the settings on your new mobile phone?

And if its as blindingly obvious as a setting as noise reduction factor in a NOISE TEST, then you fail in life if you don't check that.

I hope they don't test waterproof cameras one day, and one is sent as factory default with the card door open or something.

The Underwatermagictron 2000 failed completely in our underwater testing, water poured in and destroyed the camera. Unacceptable!

RiceHigh · 06-14-2008, 08:11 AM

Originally posted by flyer

The BIG DEAL is that the Olypus E-3 comes off with N.R. off as the default setting and they turned it on. They can't even respect their own protocol, which makes them as biased as you. That is the point.

So what then? Do you expect every measurebation test conducted by each measurbator to be the same each time? Ask yourself if the test results are favourable to Pentax, will you make the complaint? (which the case is just of no big deal IMO)

As long as the reviewers let you know about the settings, I think it is already fair enough. Whether their published results are meaningful or not is another story.

falconeye · 06-14-2008, 09:35 AM

Originally posted by RiceHigh

I would say those noise tests nowadays by the measurbators are not so meaningful as long as noises are measured along instead of the Signal-to-Noise (aka S/N) ratio.

RH, well said.

But things are improving, slowly.

Recent German magazine "Color Foto" decided they must redo all camera tests (29 DSLRs retested) because they "forgot" to notice the disappearance of texture detail in some models.

Some of those "preservation of texture" values read (bigger is better, max. 15, ISO 100/400/800/1600):

Pentax K20D(Samsung GX20): 15/15/15/14.5(15) (or 100%!)
Pentax K200D: 15/15/15/13.5
Pentax K100Dsp: 13/13.5/13.5/12
Nikon D300: 14.5/14.5/14/14 (or 98%)
Nikon D3: 14.5/14.4/14/14.5
Leica Digilux 3: 15/14.5/14.5/14.5
Canon EOS 40D: 10/10.5/10.5/8.5 (or 66%)
Sony Alpha 700: 9/8.5/9/10.5 (or 62%)

So, if you just would weight category "preservation of texture" high enough, Pentax would win the entire competition. Of course, they didn't. But they ended up at a very respectful position.

flyer · 06-14-2008, 10:09 AM

[QUOTE=RiceHigh;268196]So what then? Do you expect every measurebation test conducted by each measurbator to be the same each time? Ask yourself if the test results are favourable to Pentax, will you make the complaint? (which the case is just of no big deal IMO)

As long as the reviewers let you know about the settings, I think it is already fair enough. Whether their published results are meaningful or not is another story.[/QUOTE
Comparison tests, to be valid, should ALWAYS be done under the same conditions, not the conditions that suit the tester. By the way, what makes you an "expert" on the K10D/K20D. As far as I know, you never owned or used any of the two mentioned cameras, yet you keep bringing them down. If those cameras are as bad as you imply, how come the K10D won so many awards? As far as I'm concerned, you wouldn't know a good camera if it bit you on the a**.

P.S. Answer if you want' I don't care and won't answer back. You always need to be right, even when you're wrong.

morfic · 06-14-2008, 09:10 PM

falconeye: those are impressive numbers, no wonder i love the K20D so much over the 30D. Makes me wanna see if i can subscribe to Color Foto from the US.

George Lama · 06-14-2008, 10:35 PM

Testing with settings out of the box certainly shows that most of these "reviewers" are catering to the masses who think SLRs are nothing more than fancy point and shoots with lots of extra buttons.

Who in their right mind would assume that default settings are the best settings? Well, except maybe Microsoft.

cputeq · 06-15-2008, 07:03 AM

I don't see a problem with this methodology.
------------------

Let's assume Popphoto does alter camera settings -- Guess what happens? You have people complaining "we all know that Setting X is better than Setting Y in the camera but they used the worse setting!"

PopPhoto is a mass-market photography magazine -- they are to cameras as Ziff-Davis publishing is to computers, so their reviews should be taken with a grain of salt.

That being said, testing cameras in factory defaults is about the only fair way to do things -- but, in a case such as this (all cameras having NR on except the Pentax), a special note should be mentioned.

falconeye · 06-15-2008, 11:45 AM

Originally posted by cputeq

That being said, testing cameras in factory defaults is about the only fair way to do things

cputeq, you're right for a tester with average (read insufficient) qualification.

Let me give an example from the testing methodology of German computer magazine c't.
IT news, features and forums - heise online UK
and
c't - magazin fr computertechnik
They have an outstanding reputation, not met by any photo magazine, or any other computer magazine I am aware of, on a worldwide scale...

What do they do?

First, they live with the products long enough to know them.

Second, they reconfigure the products to bring them to their best value, even sometimes making the manufacturer give them an updated model following their input. This means, exchanging drivers, running updates, maybe exchanging too loud fans etc. They do what an actual buyer would have done.

Third, they judge the individual aspects. They do not give an overall score!

I don't see why a photo mag couldn't do this.

06-14-2008, 04:46 AM	#1
flyer Veteran Member Join Date: Jan 2008 Location: Ste-Anne des Plaines, Qc., Canada Posts: 2,013	Pop Photo July issue In the july issue, Pop Photo has a comparison test of high ISO cameras. In the test results, they rated the noise level of the K20D at ISO 3200 and 6400 as unacceptable. What ticks me off is the fact (stated in the article) that they left the NR off on the K20D while all the other contenders had high ISO N.R. on. It shows that Pop Photo has a strong dislike of Pentax cameras to use such tricks to discredit a model. They should use the same conditions for all the cameras they test for a particular issue. They implied that they left the camera in the state it comes in the box (default setting), but they turned the high ISO on on the Olympus E-3. If you look only at the results without reading the whole article, it could mislead a lot of readers. Why turn high ISO n.r. on for the Olympus and not do it for the K20D? POP PHOTO, YOU STINK

06-14-2008, 05:52 AM	#3
RiceHigh Veteran Member Join Date: Nov 2006 Location: Hong Kong Posts: 1,934	What's the big deal here? So, what's the big deal? Even Dpreview do tests at the default factory settings. It is just impossible to try each combination for they to carry out the review. Does an individual setting require an individual test report? If it is the factory default, it is fair enough to believe that it is the best setting as recommended by the maker, isn't it? Anyway, I would say those noise tests nowadays by the measurbators are not so meaningful as long as noises are measured along instead of the Signal-to-Noise (aka S/N) ratio. Take the following DPR posted high ISO shootout image for example:- https://www.pentaxforums.com/forums/pentax-dslr-discussion/29317-14-mpix-high-iso-shoot-out.html I must say I am impressed with the details and rich colours retained by the K20D at high ISO, although it is obviously noisy from ISO 800 already. (The 450D and A350 are far less noisy, but there are no good IQ at all (as Phil Askey and his guy seem to be right in what they are presenting)) Originally posted by flyer In the july issue, Pop Photo has a comparison test of high ISO cameras. In the test results, they rated the noise level of the K20D at ISO 3200 and 6400 as unacceptable. What ticks me off is the fact (stated in the article) that they left the NR off on the K20D while all the other contenders had high ISO N.R. on. It shows that Pop Photo has a strong dislike of Pentax cameras to use such tricks to discredit a model. They should use the same conditions for all the cameras they test for a particular issue. They implied that they left the camera in the state it comes in the box (default setting), but they turned the high ISO on on the Olympus E-3. If you look only at the results without reading the whole article, it could mislead a lot of readers. Why turn high ISO n.r. on for the Olympus and not do it for the K20D? POP PHOTO, YOU STINK

06-14-2008, 06:28 AM	#5
jsherman999 Veteran Member Join Date: Dec 2007 Photos: Gallery Posts: 8,237	Originally posted by RiceHigh ... Anyway, I would say those noise tests nowadays by the measurbators are not so meaningful as long as noises are measured along instead of the Signal-to-Noise (aka S/N) ratio. ... You're right about that, RH, so many people seem to not understand the S/N ratio concept - and I seriously mean really do not understand it. I'm starting to come to the dismaying conclusion that even some semi-pro reviewers for publications like Pop Photo don't quite get the concept - which would be equivalent to a writer for Car & Driver not quite getting the concept of, say, fuel injection. If you're a reviewer, it's your responsibility to know this stuff, and to not misrepresent measured results by using out-of-context frames of reference. The S/N ratio of the K20D is nearly class-leading, if not at the very top of it's class. The fact that this has not been spelled out in big bold letters means that reviewers either a) discount S/N inappropriately, b) don't really fully understand the concept, and thus skim over it, or c) assume everyone already understands it, so it doesn't need to be discussed. Meanwhile, it continues to be misunderstood by a good portion of the buying public as well as a good number of secondary reviewers/ review referrers. . Last edited by jsherman999; 06-14-2008 at 06:37 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
K-7 has graced the cover of the September issue of Pop Photo as a "Best Buy!"	Matjazz	Pentax DSLR Discussion	33	11-04-2012 11:57 AM
Photo of the Week #122, July 11 - July 25, 2010	VaughnA	Weekly Photo Challenges	47	07-24-2010 10:45 PM
Photo of the Week #121, July 4 - July 18, 2010	jmschrei	Weekly Photo Challenges	50	07-18-2010 10:01 AM
Pop Photo K20d review	jms698	Pentax News and Rumors	136	02-25-2008 01:00 PM
Pop Photo	roy	Pentax News and Rumors	1	01-30-2008 03:42 PM

06-14-2008, 05:31 AM	#2
monochrome Loyal Site Supporter Join Date: Mar 2008 Location: Working From Home Photos: Gallery \| Albums Posts: 26,276	I don't take their (or any) photography magazine for just this reason, but I'd bet Pentax doesn't advertise enough (pay enough bribes) to be treated fairly - this is a form of blackmail.

06-14-2008, 05:55 AM	#4
RiceHigh Veteran Member Join Date: Nov 2006 Location: Hong Kong Posts: 1,934	Also, we actually don't need to read the new issue to compare their old results. A quick research on the old published issue will give us the info (which I did that already):- RiceHigh's Pentax Blog: First K20D Production Camera Formal Test Originally posted by flyer In the july issue, Pop Photo has a comparison test of high ISO cameras. In the test results, they rated the noise level of the K20D at ISO 3200 and 6400 as unacceptable. What ticks me off is the fact (stated in the article) that they left the NR off on the K20D while all the other contenders had high ISO N.R. on. It shows that Pop Photo has a strong dislike of Pentax cameras to use such tricks to discredit a model. They should use the same conditions for all the cameras they test for a particular issue. They implied that they left the camera in the state it comes in the box (default setting), but they turned the high ISO on on the Olympus E-3. If you look only at the results without reading the whole article, it could mislead a lot of readers. Why turn high ISO n.r. on for the Olympus and not do it for the K20D? POP PHOTO, YOU STINK

06-14-2008, 06:38 AM	#6
cupic Veteran Member Join Date: Apr 2008 Location: Australia-NSW Photos: Gallery Posts: 4,824	You cant compare apples to pears cheers

06-14-2008, 06:41 AM	#7
flyer Veteran Member Join Date: Jan 2008 Location: Ste-Anne des Plaines, Qc., Canada Posts: 2,013 Original Poster	Originally posted by RiceHigh So, what's the big deal? The BIG DEAL is that the Olypus E-3 comes off with N.R. off as the default setting and they turned it on. They can't even respect their own protocol, which makes them as biased as you. That is the point.

06-14-2008, 08:08 AM	#8
Maxington Senior Member Join Date: Sep 2007 Location: Australia Posts: 255	I never understood the point of this "TEST AT OUT OF BOX DEFAULT SETTINGS" business. Ok, from a practical point of view, they might not have time or ability to test EVERY setting, sure. But its a DSLR. It is absolutely meant to have its settings changed, fiddled with, adjusted to suit the person, the shot, the place, the lighting, you name it. That's the entire point of the camera. You find a major flaw, you go look if there is a setting to change. This is obvious, its what you do with anything you buy. Don't you change the setting of the seats in a new car? Fiddle with the settings on your new mobile phone? And if its as blindingly obvious as a setting as noise reduction factor in a NOISE TEST, then you fail in life if you don't check that. I hope they don't test waterproof cameras one day, and one is sent as factory default with the card door open or something. The Underwatermagictron 2000 failed completely in our underwater testing, water poured in and destroyed the camera. Unacceptable!

06-14-2008, 08:11 AM	#9
RiceHigh Veteran Member Join Date: Nov 2006 Location: Hong Kong Posts: 1,934	Originally posted by flyer The BIG DEAL is that the Olypus E-3 comes off with N.R. off as the default setting and they turned it on. They can't even respect their own protocol, which makes them as biased as you. That is the point. So what then? Do you expect every measurebation test conducted by each measurbator to be the same each time? Ask yourself if the test results are favourable to Pentax, will you make the complaint? (which the case is just of no big deal IMO) As long as the reviewers let you know about the settings, I think it is already fair enough. Whether their published results are meaningful or not is another story.

06-14-2008, 09:35 AM	#10
falconeye Veteran Member Join Date: Jan 2008 Location: Munich, Alps, Germany Photos: Gallery Posts: 6,871	Originally posted by RiceHigh I would say those noise tests nowadays by the measurbators are not so meaningful as long as noises are measured along instead of the Signal-to-Noise (aka S/N) ratio. RH, well said. But things are improving, slowly. Recent German magazine "Color Foto" decided they must redo all camera tests (29 DSLRs retested) because they "forgot" to notice the disappearance of texture detail in some models. Some of those "preservation of texture" values read (bigger is better, max. 15, ISO 100/400/800/1600): Pentax K20D(Samsung GX20): 15/15/15/14.5(15) (or 100%!) Pentax K200D: 15/15/15/13.5 Pentax K100Dsp: 13/13.5/13.5/12 Nikon D300: 14.5/14.5/14/14 (or 98%) Nikon D3: 14.5/14.4/14/14.5 Leica Digilux 3: 15/14.5/14.5/14.5 Canon EOS 40D: 10/10.5/10.5/8.5 (or 66%) Sony Alpha 700: 9/8.5/9/10.5 (or 62%) So, if you just would weight category "preservation of texture" high enough, Pentax would win the entire competition. Of course, they didn't. But they ended up at a very respectful position.

06-14-2008, 10:09 AM	#11
flyer Veteran Member Join Date: Jan 2008 Location: Ste-Anne des Plaines, Qc., Canada Posts: 2,013 Original Poster	[QUOTE=RiceHigh;268196]So what then? Do you expect every measurebation test conducted by each measurbator to be the same each time? Ask yourself if the test results are favourable to Pentax, will you make the complaint? (which the case is just of no big deal IMO) As long as the reviewers let you know about the settings, I think it is already fair enough. Whether their published results are meaningful or not is another story.[/QUOTE Comparison tests, to be valid, should ALWAYS be done under the same conditions, not the conditions that suit the tester. By the way, what makes you an "expert" on the K10D/K20D. As far as I know, you never owned or used any of the two mentioned cameras, yet you keep bringing them down. If those cameras are as bad as you imply, how come the K10D won so many awards? As far as I'm concerned, you wouldn't know a good camera if it bit you on the a**. P.S. Answer if you want' I don't care and won't answer back. You always need to be right, even when you're wrong.

06-14-2008, 09:10 PM	#12
morfic Veteran Member Join Date: Feb 2008 Location: San Antonio, TX Photos: Gallery Posts: 428	falconeye: those are impressive numbers, no wonder i love the K20D so much over the 30D. Makes me wanna see if i can subscribe to Color Foto from the US.

06-14-2008, 10:35 PM	#13
George Lama Veteran Member Join Date: Jun 2007 Location: Buffalo/Rochester, NY Photos: Albums Posts: 1,133	Testing with settings out of the box certainly shows that most of these "reviewers" are catering to the masses who think SLRs are nothing more than fancy point and shoots with lots of extra buttons. Who in their right mind would assume that default settings are the best settings? Well, except maybe Microsoft.

06-15-2008, 07:03 AM	#14
cputeq Inactive Account Join Date: Apr 2008 Posts: 464	I don't see a problem with this methodology. ------------------ Let's assume Popphoto does alter camera settings -- Guess what happens? You have people complaining "we all know that Setting X is better than Setting Y in the camera but they used the worse setting!" PopPhoto is a mass-market photography magazine -- they are to cameras as Ziff-Davis publishing is to computers, so their reviews should be taken with a grain of salt. That being said, testing cameras in factory defaults is about the only fair way to do things -- but, in a case such as this (all cameras having NR on except the Pentax), a special note should be mentioned.

06-15-2008, 11:45 AM	#15
falconeye Veteran Member Join Date: Jan 2008 Location: Munich, Alps, Germany Photos: Gallery Posts: 6,871	Originally posted by cputeq That being said, testing cameras in factory defaults is about the only fair way to do things cputeq, you're right for a tester with average (read insufficient) qualification. Let me give an example from the testing methodology of German computer magazine c't. IT news, features and forums - heise online UK and c't - magazin fr computertechnik They have an outstanding reputation, not met by any photo magazine, or any other computer magazine I am aware of, on a worldwide scale... What do they do? First, they live with the products long enough to know them. Second, they reconfigure the products to bring them to their best value, even sometimes making the manufacturer give them an updated model following their input. This means, exchanging drivers, running updates, maybe exchanging too loud fans etc. They do what an actual buyer would have done. Third, they judge the individual aspects. They do not give an overall score! I don't see why a photo mag couldn't do this.