Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
03-16-2014, 03:36 AM   #1
New Member




Join Date: Feb 2014
Posts: 18
The problem with "Professional" Reviews

Hello All,

What I wanted to comment on is the disparity I see between professional reviews of cameras and real life experience of consumers. Not so much regarding sterile studio test reviews versus actually going outside, taking pictures and producing photographs; but rather, the often glaring disparity between professional reviews and end user findings regarding a product defect. How often do we see glowing reviews for products that make no mention of well-known manufacturing defects (hardware or software) discovered by consumers. Not even in follow up reviews if there are any. Examples include spots on the sensor of the Nikon 7000 (maybe it was the 7100) and perhaps Pentax had a similar issue with the K5 maybe. Sample variation is one explanation, as I am sure manufacturers do their best to hand pick the models sent to the reviewer. Also, pre-production models are sent out as well (we have all seen the Pentax K3 with Pentax logo on the back).

I understand some issues may not reveal themselves immediately; and only after some time (excluding normal wear and tear) will a defective part be revealed. This might not be noticed by an initial review and that makes sense, but even so should there not be some expectation of a long term review? We do see these types of long-term reviews for sporting equipment- camping gear is a good example.

Perhaps I am trying to come to terms with my own expectations of a product review. I find these reviews often carry themselves out like extended advertisements for a product as opposed to independent investigations. Reviewers tell us they are not biased and carry out their sterile tests (wearing white lab coats I am sure) and we are expected to take this as some standard of objectivity. Even the most well meaning reviewers I think are fooling themselves along with the public they hope to inform. The ceaseless drone and battery of test results carried out almost prove nothing anymore. That well-documented, repeatable and crippling product defects discovered by endusers do not make into reviews is damning to the credibility of any review, and any reviewer worth their salt should be ashamed.

What should be the standard for review of a camera? Should it be accepted what we have so far from DPreview, EPhotozine, CNet and the like? Conclusions of a reviewer for a camera (hand-picked by the manufacturer) based solely on graphs, charts and the same studio images over and over again? Or even worse, Youtube video statements from some person (read Senior Managing Photo Reviewer) merely holding a camera and occasionally pushing a button or navigating a menu with some air of authority? What is the point of going through these tedious tests if the product has some major manufacturing flaw rendering it a de facto paper weight?

If you are still reading, I apologize. I cannot give you back those 2 minutes.

There is a criticism among some educators regarding standardized tests. I remember a million years ago when I was in elementary or middle school we had to take the California Achievement Test. The test is suppose to measure against some standard of reading, writing, and arithmetic aptitude. One of the criticisms is that the teachers are restricted in their curriculum and end up teaching the student how to pass the test as opposed to educating us. Assessment based on multiple guess arithmetic problems is questionable if no work is shown, and teachers are right I think for this kind of criticism. Likewise reviews from the likes of DPreview and so on, if there is no evaluation beyond the standard test they perform, then it is only half a job at best.


I would like to think the following should become standard for any review- 1) sample variation testing should be a part of a review; 2) Long-term testing should be performed to ferret out any product defects over time. And honestly, some folks notice issues straight out of the box and so I wonder how is it the professional reviewer missed it. Additionally, any review for a product based on a pre-production model should be automatically disregarded by us, the consumer; and, any reviewer with any integrity should not accept such a sample to be used as the subject for a review. And, Any review that is a mind-numbing reiteration of the product overview page from the manufacturer can also be thrown out. Any review released the day of production release should also be ignored as nothing other than free advertising for the manufacturer.



Yes, I can choose to ignore all the reviews and buy based on my own assessment of the promises of the manufacturer, but these reviews are meant to be on our side. However, currently, they seem to be run as a business. DPreview is run by Amazon and reviews are made to and prioritized to products that sell or traditionally have the marketshare. If that is how they want to run themselves then that is their business, but we should not accept any review they carry out. They have lost credibility. This is not specifically a diatribe against Dpreview as such (though it sounds like one a bit). I do not care specifically if DPReview ever does a review or not on any particular camera or manufacturer, but we should be wary of these reviews.


Caveat Emptor – Let the buyer beware. This is no longer a caution to the consumer against the manufacturer; but equally well, we should be cautious to what the reviewer is selling, especially if they are leaving a job undone. That reviewers want to run a camera through a battery of lab tests is fine, go for it. Obviously people love that stuff, and it can be useful information especially in evaluating performance against manufacturer claims. I would like to see at least some more inspired subjects, but that is neither here nor there I suppose. Reminds me though during a programming course I followed at university. The professor said while testing code that if we did not find any issue with our code there was something wrong with our test. All products these days are going to have some issue, I understand. The job of the reviewer, however, is suppose to find them and pass them along to the potential consumer. There is an implied trust in a product review between the reviewer and the consumer; and for that, we should hold the reviewer to some degree accountable for their review. I think it is time we invoke the principle Caveat venditor, Let the seller beware, and apply it to not only the manufacturer for false advertising claims (as does happen sometimes) but to the "professional' reviewer as well for their negligence at the very least. That it is solely left up to the consumer to find fault and carry out the due diligence to document, reproduce, corroborate with others to try to get the manufacturer to correct the issue. We should expect some support from our trusted professional reviewers.



For the purposes of full disclosure, this started because of my slavish obedience to the sea of reviewers selling me on the virtues of the Pentax K3 and my discover through teh good folks here on the forum that there does seem to be an issue regarding the mirror flap. I was able to back out of my purchase before potentially being stuck with a dud; but I find it interesting that I have yet to come across any reviewer - DPreview, Ephotozine, Pentax Forums, Digital Resource who had this issue either during initial testing or at least in some kind of follow-up statement. Correction, it seems Adorama briefly discussed this in a conclusion section; but obviously they did not do their homework as they reported it was resolved from a firmware fix; but we know that is not true and they should know better to make such a statement.


Apologies again for wasting everyone’s time.

03-16-2014, 04:33 AM - 2 Likes   #2
Site Supporter
Site Supporter




Join Date: Jan 2008
Location: Middle of Everywhere
Photos: Gallery
Posts: 1,180
What's with the hyper-cynicism?

Do you pursue a similar analysis (consult trip adviser, the local health department, gather the experience of willing strangers, etc.) before you sit down for dinner in a restaurant ???... Hey... it's only a camera, after all... not your health that's being put at risk...

So you make a mistake by over-relying on some stranger's observations and get burned.. consider it tuition, learn and move on...

Maybe Pentax is too much for you... just pass on it (and you'll never know... )

Sometimes you've just got to trust the soup...

Cheers... M

Last edited by Michaelina2; 03-16-2014 at 04:39 AM.
03-16-2014, 04:40 AM   #3
Veteran Member
JimmyDranox's Avatar

Join Date: Oct 2012
Location: Ploiesti, Romania
Photos: Gallery | Albums
Posts: 1,630
Simply, reviewer are paid by stores, who are paid for ads by manufacturers, and only when a problem is really big, like the Nikon D600 stains on sensor, they take notice. Of course, there are differences, from site to site. Recently IR has discussed openly about light leaks for Sony A7/R, and Fuji X-T1, while other sites ignores it.

So, all we can do is to take those review with a grain of salt, and better, to wait a little when a new camera comes to market. But it is unfortunate, that we cannot trust them.
03-16-2014, 04:47 AM - 1 Like   #4
Veteran Member
Barry Pearson's Avatar

Join Date: Jul 2013
Location: Stockport
Posts: 951
QuoteOriginally posted by Little Ray Quote
For the purposes of full disclosure, this started because of my slavish obedience to the sea of reviewers selling me on the virtues of the Pentax K3 and my discover through teh good folks here on the forum that there does seem to be an issue regarding the mirror flap. I was able to back out of my purchase before potentially being stuck with a dud; but I find it interesting that I have yet to come across any reviewer - DPreview, Ephotozine, Pentax Forums, Digital Resource who had this issue either during initial testing or at least in some kind of follow-up statement. Correction, it seems Adorama briefly discussed this in a conclusion section; but obviously they did not do their homework as they reported it was resolved from a firmware fix; but we know that is not true and they should know better to make such a statement.
I got an early K-3 by pre-ordering it. I have never suffered from mirror flap. I assume the vast majority of K-3 users don't either. And probably it is so rare that reviewers typically won't get it on their review cameras.

I suspect the likelihood that you would have suffered from it had you not backed-out is very low. Not zero, presumably. (I've certainly read that some people have suffered). But very low.

03-16-2014, 04:50 AM   #5
New Member




Join Date: Feb 2014
Posts: 18
Original Poster
QuoteOriginally posted by Michaelina2 Quote
What's with the hyper-cynicism?

Do you pursue a similar analysis (consult trip adviser, the local health department, gather the experience of willing strangers, etc.) before you sit down for dinner in a restaurant ???... Hey... it's only a camera, after all... not your health that's being put at risk...

So you make a mistake by over-relying on some stranger's observations and get burned.. consider it tuition, learn and move on...

Maybe Pentax is too much for you... just pass on it (and you'll never know... )

Sometimes you've just got to trust the soup...

Cheers... M
Yep,

I fully admit to hyperbolizing the whole thing; and yes, Pentax is probably too much for me anyway. Most of the cameras in this class are. I do have aspirations though.

Cheers,

Little Ray
03-16-2014, 05:33 AM   #6
Veteran Member




Join Date: Jan 2012
Location: Slovenia
Photos: Gallery
Posts: 2,184
QuoteOriginally posted by Michaelina2 Quote
not your health that's being put at risk...
Umm... Well... Canon did have an allergenic rubber problem...
03-16-2014, 06:38 AM   #7
Senior Member




Join Date: Oct 2008
Location: Prince George, British Columbia, Canada.
Posts: 135
If it doesn't work right send the darn thing back simple! isn't that what the warranty is all about.
03-16-2014, 06:39 AM   #8
Loyal Site Supporter
Loyal Site Supporter




Join Date: Nov 2013
Location: Alabama
Posts: 2,450
QuoteOriginally posted by Giklab Quote
Umm... Well... Canon did have an allergenic rubber problem...
A what?

03-16-2014, 06:50 AM   #9
Veteran Member
oxidized's Avatar

Join Date: Dec 2009
Location: USA - Delaware
Photos: Albums
Posts: 435
There is ups and downs to either type of review. Having a standardized review obviously is good because it allows you to measure cameras (one against another) in a very defined way. It also explores all areas (usually) that someone may be interested in before buying a camera. However there is the problem of sample variation or prioritization of the products. Another problem is that the reviewer simply uses the camera for like a week (or two) and then writes the review and discards it. They are too focused on the aspects that they test with all other cameras and could miss an array of problems. For instance dpreview did not catch the light leaks / shutter vibration problems of the a7r. they did not notice the sensor stains of the d600 etc..

More user reviews obviously has the advantage of sample size. Bigger sample (as opposed a case study) provides stronger and more confident results. The biggest issue is that the "quality" of the datapoints is somewhat inferior/questionable. Anyone can go online and write a review and most often people base their results (or quantify them at least) relative to what they have seen/used in the past. For instance, if all you have used in the past has been an 18-200mm lens and suddenly you get yourself a prime (say 50 1.7mm KA) you will be BLOWN AWAY by the sharpness, the added bokeh etc. You get on some site and you give it five stars or 10/10. Not saying it is a bad lens, but 10/10 to me is equivalent to all around perfection and there are plenty of 50mm lens out there that are all unique in their own away and very possibly better than the 50/1.7. The problem is that "users" unlike "profesisonal reviewers" (usually) dont have access to more than one item in the same class/league and it is hard for them to measure performance relative to another camera/lens the product is actually competing against. The just compare it to some standard set in their mind from to the previous equipment they have used. This is unfortunate bcs you as a buyer, whenever you want to buy a 50mm you comparare all 50mm lens out there on bhphoto and buy the "best" one for the money.

The easiest way to observe this phenomena is if you go say the review database. Check out some of the reviews of the prime lenses out there. You can easily find plenty of lenses where a bunch of users put "sharpness" and as pro and a bunch of other users consider the same lens as "too soft". How is this possible? I highly doubt it is sample variation (even though a few lenses have been known to have such problems). What is sharp for me may not be sharp for you and it is all based on what other equipment i have/have used.

I think that either type of reviews have their purpose/uses. Just have to know how to use them.

Last edited by oxidized; 03-16-2014 at 07:00 AM.
03-16-2014, 06:54 AM - 1 Like   #10
Loyal Site Supporter
Loyal Site Supporter




Join Date: Nov 2013
Location: Alabama
Posts: 2,450
QuoteOriginally posted by Little Ray Quote
Hello All,

What I wanted to comment on is the disparity I see between professional reviews of cameras and real life experience of consumers. Not so much regarding sterile studio test reviews versus actually going outside, taking pictures and producing photographs; but rather, the often glaring disparity between professional reviews and end user findings regarding a product defect. How often do we see glowing reviews for products that make no mention of well-known manufacturing defects (hardware or software) discovered by consumers. Not even in follow up reviews if there are any. Examples include spots on the sensor of the Nikon 7000 (maybe it was the 7100) and perhaps Pentax had a similar issue with the K5 maybe. Sample variation is one explanation, as I am sure manufacturers do their best to hand pick the models sent to the reviewer. Also, pre-production models are sent out as well (we have all seen the Pentax K3 with Pentax logo on the back).

I understand some issues may not reveal themselves immediately; and only after some time (excluding normal wear and tear) will a defective part be revealed. This might not be noticed by an initial review and that makes sense, but even so should there not be some expectation of a long term review? We do see these types of long-term reviews for sporting equipment- camping gear is a good example.

Perhaps I am trying to come to terms with my own expectations of a product review. I find these reviews often carry themselves out like extended advertisements for a product as opposed to independent investigations. Reviewers tell us they are not biased and carry out their sterile tests (wearing white lab coats I am sure) and we are expected to take this as some standard of objectivity. Even the most well meaning reviewers I think are fooling themselves along with the public they hope to inform. The ceaseless drone and battery of test results carried out almost prove nothing anymore. That well-documented, repeatable and crippling product defects discovered by endusers do not make into reviews is damning to the credibility of any review, and any reviewer worth their salt should be ashamed.

What should be the standard for review of a camera? Should it be accepted what we have so far from DPreview, EPhotozine, CNet and the like? Conclusions of a reviewer for a camera (hand-picked by the manufacturer) based solely on graphs, charts and the same studio images over and over again? Or even worse, Youtube video statements from some person (read Senior Managing Photo Reviewer) merely holding a camera and occasionally pushing a button or navigating a menu with some air of authority? What is the point of going through these tedious tests if the product has some major manufacturing flaw rendering it a de facto paper weight?

If you are still reading, I apologize. I cannot give you back those 2 minutes.

There is a criticism among some educators regarding standardized tests. I remember a million years ago when I was in elementary or middle school we had to take the California Achievement Test. The test is suppose to measure against some standard of reading, writing, and arithmetic aptitude. One of the criticisms is that the teachers are restricted in their curriculum and end up teaching the student how to pass the test as opposed to educating us. Assessment based on multiple guess arithmetic problems is questionable if no work is shown, and teachers are right I think for this kind of criticism. Likewise reviews from the likes of DPreview and so on, if there is no evaluation beyond the standard test they perform, then it is only half a job at best.


I would like to think the following should become standard for any review- 1) sample variation testing should be a part of a review; 2) Long-term testing should be performed to ferret out any product defects over time. And honestly, some folks notice issues straight out of the box and so I wonder how is it the professional reviewer missed it. Additionally, any review for a product based on a pre-production model should be automatically disregarded by us, the consumer; and, any reviewer with any integrity should not accept such a sample to be used as the subject for a review. And, Any review that is a mind-numbing reiteration of the product overview page from the manufacturer can also be thrown out. Any review released the day of production release should also be ignored as nothing other than free advertising for the manufacturer.



Yes, I can choose to ignore all the reviews and buy based on my own assessment of the promises of the manufacturer, but these reviews are meant to be on our side. However, currently, they seem to be run as a business. DPreview is run by Amazon and reviews are made to and prioritized to products that sell or traditionally have the marketshare. If that is how they want to run themselves then that is their business, but we should not accept any review they carry out. They have lost credibility. This is not specifically a diatribe against Dpreview as such (though it sounds like one a bit). I do not care specifically if DPReview ever does a review or not on any particular camera or manufacturer, but we should be wary of these reviews.


Caveat Emptor – Let the buyer beware. This is no longer a caution to the consumer against the manufacturer; but equally well, we should be cautious to what the reviewer is selling, especially if they are leaving a job undone. That reviewers want to run a camera through a battery of lab tests is fine, go for it. Obviously people love that stuff, and it can be useful information especially in evaluating performance against manufacturer claims. I would like to see at least some more inspired subjects, but that is neither here nor there I suppose. Reminds me though during a programming course I followed at university. The professor said while testing code that if we did not find any issue with our code there was something wrong with our test. All products these days are going to have some issue, I understand. The job of the reviewer, however, is suppose to find them and pass them along to the potential consumer. There is an implied trust in a product review between the reviewer and the consumer; and for that, we should hold the reviewer to some degree accountable for their review. I think it is time we invoke the principle Caveat venditor, Let the seller beware, and apply it to not only the manufacturer for false advertising claims (as does happen sometimes) but to the "professional' reviewer as well for their negligence at the very least. That it is solely left up to the consumer to find fault and carry out the due diligence to document, reproduce, corroborate with others to try to get the manufacturer to correct the issue. We should expect some support from our trusted professional reviewers.



For the purposes of full disclosure, this started because of my slavish obedience to the sea of reviewers selling me on the virtues of the Pentax K3 and my discover through teh good folks here on the forum that there does seem to be an issue regarding the mirror flap. I was able to back out of my purchase before potentially being stuck with a dud; but I find it interesting that I have yet to come across any reviewer - DPreview, Ephotozine, Pentax Forums, Digital Resource who had this issue either during initial testing or at least in some kind of follow-up statement. Correction, it seems Adorama briefly discussed this in a conclusion section; but obviously they did not do their homework as they reported it was resolved from a firmware fix; but we know that is not true and they should know better to make such a statement.


Apologies again for wasting everyone’s time.
I'm invoking "The Rule" in case op returns to delete this exceedingly loquacious diatribe.

I'm adding a TL;DR for y'all: OP is upset that pro reviewers got good copies of the K-3 with no CAMS. He almost bought one, but didn't because he heard on the Internet that a LOT of K-3s have CAMS.

---------- Post added 03-16-14 at 06:56 AM ----------
CAMS = crazy *** mirror sickness


03-16-2014, 07:13 AM   #11
Pentaxian




Join Date: Mar 2010
Location: Frankfurt am Main
Posts: 1,182
Catching the significance of sample variation will never be possible with such tests, no matter how many samples you would test. You will not get information about possible variations in production or QC processes, nor which of the employees were responsible at the time of production.

What you will get may be some (or none) bandwidth of results, which would mean nothing, as in both cases it could either be the rare exception, or the standard. You would have to pick the samples yourself right at the production line (I worked for nearly 30 years as advicer for quality control of machinery parts in Europe, Asia, and South Africa).

To get more realistic results in the sense the OP is suggesting, you would need a near to complete poll of the existing user base. But then you would not know how biased or experienced these users have been.
03-16-2014, 07:22 AM   #12
Pentaxian
johnmflores's Avatar

Join Date: Apr 2009
Location: Somerville, NJ
Photos: Gallery | Albums
Posts: 5,264
I review motorcycles. I was reviewing a BMW that had notchy throttle response at particular RPMs. I debated writing about it in the review. I looked around for complaints online-there was none that I could find. I wrote about in an early draft but pulled it before publication. I'm glad I did. A couple of months later I got another press bike-the same BMW model but a different copy. Throttle response was smooth as butter.

In other words, I will only write about a defect if I can write about it with some authority, not just a one-off issue. Knowing that tens of thousands of people may read what you write makes you more considered than noting personal experiences on a forum like this. They are both valuable, of course.

Regarding the K-3...some people are having issues, but I'd buy another one if I needed it. That's what warranties are for, and thanks to the forum members I'd know how to test for the defect.

If I only purchased products that had no reported defects then I'd have nothing.

Last edited by johnmflores; 03-16-2014 at 07:31 AM.
03-16-2014, 07:36 AM   #13
Veteran Member




Join Date: Jan 2012
Location: Slovenia
Photos: Gallery
Posts: 2,184
QuoteOriginally posted by dansamy Quote
A what?
Canon issues allergy warning for EOS 650D/Rebel T4i: Digital Photography Review

I'm surprised you haven't heard of this, considering the whole internet went LOLOLOL CANON when they announced this.
03-16-2014, 07:37 AM - 1 Like   #14
Site Supporter
Site Supporter




Join Date: Jan 2008
Location: Middle of Everywhere
Photos: Gallery
Posts: 1,180
QuoteOriginally posted by Little Ray Quote

Pentax is probably too much for me anyway. Most of the cameras in this class are. I do have aspirations though.
I understand... no problem... Wanna have a ton of photography fun?

Consider this...

PentaxWebstore.com:PENTAX Q7 Complete Kit PENTAX Q7 Complete Kit

... you get everything... a complete system for less than the price a K-3 (body only) AND much less than you might pay for each part individually... looks like a killer deal.... It's sooooo stealthy AND has an upgrade path that will not break the bank...

Next, dedicate (say) six months to becoming a post processing wizard. With a little experience, soon you should be able to match (or better) the output of most D4, or D800 toting photographers.

Once you know your gear inside and out, and know how to shoot to its strengths, add the OEM K mount adapter followed by some of KEH camera's (cheap) BGN grade K mount manual focus lenses.... and on you'll go...

Easy... no problem!

Cheers... M
03-16-2014, 07:48 AM   #15
Pentaxian
Site Supporter
normhead's Avatar

Join Date: Jun 2007
Location: Near Algonquin Park
Photos: Gallery | Albums
Posts: 30,234
My only problem with professional reviews, is the amount of crazy stuff I have to read through, to get to the little bits I really want to know about.

I doubt many of us are looking for much in a review, but we want the thing we want, since the reviewer has to anticipate what everybody might want, they have to write a lot I guess. Speed read and skim is the only answer.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
camera, consumer, dpreview, issue, manufacturer, photo industry, photography, product, review, reviewer, reviewers, reviews, test
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
what happen if i shoot with the "back" of the film instead of the "front" ? aurele Film Processing, Scanning, and Darkroom 20 07-03-2013 08:21 AM
One reason why Canon is considered "Professional" Jodokast96 Photographic Industry and Professionals 17 08-09-2012 05:05 PM
"This image cannot be displayed" problem with K5 sb in ak Troubleshooting and Beginner Help 6 06-14-2012 10:08 PM
The Problem With Da* zoom "weather-sealing" Andrew Lang Pentax SLR Lens Discussion 14 11-07-2010 07:33 AM
Is the K10 considered a "professional" camera?? deludel General Talk 21 05-27-2008 01:08 PM



All times are GMT -7. The time now is 07:10 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top