Hi, Technical Editor over at DPReview. I'd like to set the record straight by copying and pasting the actual exchange that occurred on DPR:
 Quote: @Rishi: "Can you upload them to your own gallery? Full resolution please.
At this point several members who've repeated our test get exactly our hit-rate. I've even received PMs saying 'I did your test and you were essentially right - thanks for pointing this out and I hope Pentax takes this critical feedback seriously.'
Ironically, thecamerastoretv video many have linked to as somehow representative of how focus works better than in our tests shows a whole lot of out-of-focus shots in the sequence of Chris running, visible even at the ridiculously low resolutions of vertical frames in a 1080p video (< 1MP resolution). They also provide no full-resolution images.
So we'd be very curious to see the actual data from someone who claims they're getting drastically better results in a similar test/scenario." To which, instead of submitting your images ("providing evidence"), you replied:
Quote: @pentaust: "@Rishi: I'm sad of your attitude, "At this point several members who've repeated our test get exactly our hit-rate. I've even received PMs saying 'I did your test and you were essentially right - thanks for pointing this out and I hope Pentax takes this critical feedback seriously.'
This is the most dishonest comment I've ever read. I'm disgusted. Because, you are not honest, you find an excuse to any result because you don't want to see results as they are. If you have this attitude at a court, the judge will hate you and no one will trust your saying. You are totally biased.
You retain only bad results and you ignore the good ones.
You should be sued at the court for broadcasting false information.
At DPReview company, please hire personnel who has decent professionalism when responding to users of DPreview.. In other words, you didn't "provide evidence" at all, despite us asking you to. You instead responded with an uncivil, denigrating message directed at us, claiming I was dishonest and wanted to ignore your data when I said the exact opposite: that I was curious to see your data. Here, you've suggested to the Pentax Forum members that you've provided us with this evidence, when you've done no such thing. You've claimed you were banned because we didn't want to see your evidence, when the opposite is true: we wanted to see it, and you didn't provide it, instead resorting to ad hominem derogatory insults - for which you weren't even banned, you were temp banned for only 2 weeks, asking you to please be civil on our site.
I'll let this exchange speak for itself, but I will add one thing: as a trained scientist, I am always interested in the opposing viewpoint, because I want to always check and re-check if I'm right or wrong. Being wrong keeps me up at night. So when I said I was curious to see your results, I was 100% honest and sincere. Even if your results were valid - I'd be curious in the explanation for why we'd be getting different results. I wouldn't want to look at results that disagree with mine and simply come up with excuses for why we're right and you're wrong... rather, I'd want to see if there's something we missed, or you missed, to try and get at the reasons for the perceived differences.
Objective analysis, that is. But passionate, knee-jerk reactions leave no room for objective analysis, instead opting to immediately discredit the other so you don't have to change your own belief. That's not how we work - just look at my exchange with MightyMike, for example, where we're actually trying to discuss why we have slightly different results. That's how these discussions should play out. When MightyMike was wondering why his SEL33 (subject tracking) results were so vastly different from our 15% hit-rate, we problem-solved it: his subject took up the entire AF area throughout his sequence, so there was no chance of the system getting confused about which AF point to use - any one it used would still register the correct distance. Which is why his results were closer to our single-point AF-C results -- many in or close to focus with a few significantly out, which indicates that the system still does play catch-up. With our results being a bit worse because we had constant Z-axis movement with less lateral movement, and perhaps due to the lens (they were different). The point being, we're actually trying to work out the reasons for the disparities, not immediately discrediting one another. Which is exactly how it should work.
Rather than painting us the enemy, perhaps you'd find more success were you to civilly engage with us - as if we were human beings. As if we were a human being you were speaking to in person. What's sad and disappointing to us, after the thoroughness with which we try to do our work, the passion with which we approach this task of providing responsible information to our audience, is to see knee-jerk responses that treat us as an 'enemy' and frankly dehumanize us simply because of results that may disagree with your experience, or bring into question the purchase decision you yourself made. In science, if that's how disagreements played out, nothing would really move forward. Far more productive would be to try & understand the reasons for the discrepancies, try and understand (as some other audience members have) that our frames of reference may be different - given that we test many, many cameras, and therefore have a different viewpoint and basis/standard for comparison.
One thing I will concede - it's difficult for someone to trust us without knowing exactly what we did. I understand that. We're taking this reaction as impetus to write some articles on how exactly we test AF, and why. Many of the suggestions people are making for what we should and shouldn't do when it comes to testing AF - while appreciated - have already been considered. In fact, we've put far more thought into this than most give us credit for. The reality is that we, like all such media outlets, are incredibly resource restrained, and thoroughly testing AF is such a difficult task that even the manufacturers making these very devices struggle in developing relevant tests to iterate their own systems. Realizing this, and taking into account our constraints, we've tried to come up with a set of checks and balances and tests that generally allow us to "predict" camera AF performance in variety of scenarios. We're constantly checking and re-checking our understanding, and whether or not our tests correlate with real-world performance. Furthermore, we do every test at least in triplicate, changing different AF settings to try and get the best out of the system, typically handing off the camera to multiple people on the team, testing with multiple lenses, etc. - after which we look for trends (since any one AF test in isolation can be misleading - yes, we know that). It's these trends that we end up publishing, with representative bike/mannequin/face-detect/soccer, etc. rollovers that demonstrate what we've experienced in our extensive time with the camera. In other words, real-world experience and the bike/mannequin tests are used in a sort of feedback loop - where we look for repeating patterns that allow us to glean things about the AF system.
We wouldn't have to do this if we had a set set of lab tests we knew correlated with real-world performance. But it turns out that's more difficult than you might initially think. For example, a simple low light AF test that tests the lowest light level at which focus works might over-estimate the performance of, say, a Sony a7S, which can focus down to like -5 EV in our tests. But because it's CDAF, it's slow, so if your subject is moving at all, even just rocking back and forth a little bit at a bar (not even talking about sports here), the a7S may go into a perpetual hunt. So is that -5 EV rating relevant in the real world, when in a more reasonable light level, like -1 EV, a PDAF system will perform far better by just making a measurement and jumping to the right point and taking a shot? Yes, there are ways to get around this, by perhaps measuring total time to focus for 10 attempts at various light levels, but what about subject contrast now? The reality is there are many variables, and we think about all these things, and look for trends. Another example is subject tracking (auto-shifting the AF point to stay on a subject) - where a Canon can do quite well at telephoto distances, but falls apart for tracking an eye at 35mm, e.g. Whereas a Nikon can do both. One test might have missed this nuance, which is why we now evaluate at both long distance (telephoto) and short distance (wider fast prime) - also because these simulate two different use-cases: telephoto sports vs. tracking a moving toddler, or bride at a wedding, etc. I bring these things up as examples of how much thought we try to put into our tests, their limitations, and what we can extrapolate from them. We have to try and extrapolate a considerable amount from a limited number of tests, so we put a lot of thought into everything.
I suppose I could go on forever, so I'll stop now. I hope this helps clarify some misconceptions. I think all the discussion has detracted from one overarching thing: that the K-1 is really a lovely camera, with "outstanding image quality and a number of fantastic features that simply can't be matched at this price point." (last sentence of our review)
Rishi Sanyal, Ph.D
Technical Editor, DPReview.com
|
