You get what you measure

Nesster · 06-07-2013, 07:00 AM - **1 Like**

There is a well known effect of metrics and measurements: if these become publicised (i.e. by management at your company, by enthusiast magazines, by the government, etc) they become behavior drivers.

But often enough, the resulting behaviors have perverse consequences. The measurements are very often selected because a) they are easy to make b) they appear related to some area of performance and c) they seem objective. This is metonymy.

In hi-fi the magazines measured THD and watts. The manufacturers embarked on a THD and Watt war. We didn't get better sound; we got amplifiers designed to do well on the standard tests.

At one point in my company we measured on time project delivery. There was fear of punishment if a project was late. The result? 98% on time rate. Industry norm? maybe 1/3 of projects finish when originally planned. We didn't get better projects; we got project managers who learned to manage target dates.

So what does this have to do with photography? I'm grumbling about auto focus.

What are the measurements measuring? Speed of acquiring focus, and low light focus ability, and a count of focus points. All else being equal, these are fine things to pay attention to. But we are starting to get designs that seek to maximize performance in these tests, since these numbers sell cameras.

What happens is the center spot becomes larger in order to become more sensitive, so the camera can focus on a test target in near total darkness. Great, but the trade off is uncertain focus placement in portraiture when using the center spot - it is too large to pinpoint the eye, or the photographer has to learn how to place the spot just so...

We also get systems where actual photographic usability suffers - I want to be able to have a mode where I can select the point of focus easily and quickly, have positive feedback where the focus point is, and have the camera focus exactly at that point. This sort of test can be made and standardized, but it is not easy and quick to perform and present as a graph or number.

ElJamoquio · 06-07-2013, 08:00 AM

Originally posted by Nesster

So what does this have to do with photography? I'm grumbling about auto focus.

What are the measurements measuring? Speed of acquiring focus, and low light focus ability, and a count of focus points. All else being equal, these are fine things to pay attention to. But we are starting to get designs that seek to maximize performance in these tests, since these numbers sell cameras.

What happens is the center spot becomes larger in order to become more sensitive, so the camera can focus on a test target in near total darkness. Great, but the trade off is uncertain focus placement in portraiture when using the center spot - it is too large to pinpoint the eye, or the photographer has to learn how to place the spot just so...

We also get systems where actual photographic usability suffers - I want to be able to have a mode where I can select the point of focus easily and quickly, have positive feedback where the focus point is, and have the camera focus exactly at that point. This sort of test can be made and standardized, but it is not easy and quick to perform and present as a graph or number.

I have the same pet peeve. I sent a manifesto to Pentax about it. I don't think they care, but I tried.

Kobayashi.K · 06-07-2013, 08:23 AM - **1 Like**

... "I want to be able to have a mode where I can select the point of focus easily and quickly, have positive feedback where the focus point is, and have the camera focus exactly at that point."

That's manual focus. Try it.

Na Horuk · 06-07-2013, 10:37 AM

Well written, good points. Websites like snapsort especially encourage this. But its hard to compare without measurements, hard to critique without concrete numbers. I agree with your overall point, though. It would be great if Pentax could focus on photography, instead of "winning the review battles"

JinDesu · 06-07-2013, 10:40 AM

"What happens is the center spot becomes larger in order to become more sensitive, so the camera can focus on a test target in near total darkness."

Is that true?

ElJamoquio · 06-07-2013, 11:08 AM

The Pentax's center point is huge compared to the competition, and does very well in the dark.

Don't know that it proves anything, but a large (AF) sensor is much better in low light than a small sensor,for the same reason large imaging sensors are better than smaller imaging sensors.

wildman · 06-07-2013, 11:13 AM

Originally posted by Nesster

we are starting to get designs that seek to maximize performance in these tests, since these numbers sell cameras.

The very definition of "success" in a capitalistic market system. Rational market behavior does not necessarily = good.

DeadJohn · 06-07-2013, 11:19 AM

My job involves tracking performance metrics for a government agency. Ever hear the story of a fire department pushed to improve 911 response time every year? Eventually, aggressive driving led to more firefighters being hurt on the way to fires than in fires.

It happens with helpdesks too. Techs get evaluated on how many tickets they close. So, a dead PC generates dozens of tickets, one to replace the hardware and a separate ticket for each piece of software needed on the new PC. Word, Excel, Outlook, Visio, Project, Powerpoint equals 6 tickets.

JinDesu · 06-07-2013, 11:19 AM

Originally posted by ElJamoquio

The Pentax's center point is huge compared to the competition, and does very well in the dark.

Don't know that it proves anything, but a large (AF) sensor is much better in low light than a small sensor,for the same reason large imaging sensors are better than smaller imaging sensors.

I don't know about "does very well in the dark".

The current k-5II does very well in the dark. The previous k-5 didn't do as well in the dark. I don't think there is a AF point size difference between the two.

normhead · 06-07-2013, 11:26 AM - **1 Like**

Originally posted by Kobayashi.K

... "I want to be able to have a mode where I can select the point of focus easily and quickly, have positive feedback where the focus point is, and have the camera focus exactly at that point."

That's manual focus. Try it.

People used to say this and I'd kind of hear "wah wah wah", then I got my A-400. Now, I'm starting to look for lenses that work well MF. With MF I can select a spot in front of a bird where there is nothing to focus on. You have to ask, is there an AF system that will focus on nothing? But you can do that easily manually, and then snap back to an eye focus, not at 51 stinking focus points, but anywhere in the frame, without selecting anything but a slight movement of the focus ring. If there was an A* 200 ƒ4 I'd snap up the first one I saw. The combination of FL and weight would be awesome.

I'm having a bit of trouble understanding why everyone wants to do auto-maticly what is so easy to do manually.

Nesster · 06-07-2013, 11:37 AM

Originally posted by JinDesu

"What happens is the center spot becomes larger in order to become more sensitive, so the camera can focus on a test target in near total darkness."

Is that true?

I don't know for sure! but trying to find out.

https://www.pentaxforums.com/forums/pentax-k-5/177572-all-fuss-about-auto-focus-point-size.html

http://www.luminous-landscape.com/tutorials/nikon-focus.shtml

Nesster · 06-07-2013, 11:39 AM

Originally posted by normhead

I'm having a bit of trouble understanding why everyone wants to do auto-maticly what is so easy to do manually.

LOL, because if I'm going to auto focus, I'd like it to work for me, instead of the other way around. I'm lazy that way.

I run into a similar situation with auto exposure vs manual -- I have to approach the thing in two very different manners. And with auto exposure I typically forget to apply brains to exposure compensation.

Nesster · 06-07-2013, 12:02 PM

LensRentals.com - Autofocus Reality Part 3B: Canon Cameras

this article has some very interesting info

normhead · 06-07-2013, 12:17 PM

You get the same thing with standardized testing in Education. You get whole schools focused on preparing for Standardized tests so they look good on province wide results that are published. They would actually walk into my photography class and give me booklets, so once or twice a week we could work on preparing for standardized tests in math, english and reading comprehension. The smart kids were totally bored and totally disengaged. Every other subject suffered and each passing year I could actually teach less and less of my assigned subjects. But my school was rising star on the test scores. In the 5 years before standardized testing was brought in, we put one student every year into either Sheridan or Ryerson to continue on in photography, some years two. After the standardized testing craze started, not one in 10 years. But the school was a "better" school, at least in the newspapers.

reeftool · 06-10-2013, 03:58 PM

Since I started out in photography with a manual camera which had a split prism viewfinder, to this day I focus in the center and reframe my shot if I need to. After taking pictures for over 50 years, I'm not going to change now. What do I want in auto focus? Get it right as quickly as possible. Most specifications don't really mean much to me. Is 50 auto focus points really more accurate than 10 or does it mean the camera has 40 more points to get it wrong? I'm waiting for a "bird in flight" spec. and the camera/lens that will nail it 98% of the time.

06-07-2013, 07:00 AM - 1 Like	#1
Nesster Veteran Member Join Date: Oct 2006 Location: NJ USA Photos: Gallery Posts: 13,072	You get what you measure There is a well known effect of metrics and measurements: if these become publicised (i.e. by management at your company, by enthusiast magazines, by the government, etc) they become behavior drivers. But often enough, the resulting behaviors have perverse consequences. The measurements are very often selected because a) they are easy to make b) they appear related to some area of performance and c) they seem objective. This is metonymy. In hi-fi the magazines measured THD and watts. The manufacturers embarked on a THD and Watt war. We didn't get better sound; we got amplifiers designed to do well on the standard tests. At one point in my company we measured on time project delivery. There was fear of punishment if a project was late. The result? 98% on time rate. Industry norm? maybe 1/3 of projects finish when originally planned. We didn't get better projects; we got project managers who learned to manage target dates. So what does this have to do with photography? I'm grumbling about auto focus. What are the measurements measuring? Speed of acquiring focus, and low light focus ability, and a count of focus points. All else being equal, these are fine things to pay attention to. But we are starting to get designs that seek to maximize performance in these tests, since these numbers sell cameras. What happens is the center spot becomes larger in order to become more sensitive, so the camera can focus on a test target in near total darkness. Great, but the trade off is uncertain focus placement in portraiture when using the center spot - it is too large to pinpoint the eye, or the photographer has to learn how to place the spot just so... We also get systems where actual photographic usability suffers - I want to be able to have a mode where I can select the point of focus easily and quickly, have positive feedback where the focus point is, and have the camera focus exactly at that point. This sort of test can be made and standardized, but it is not easy and quick to perform and present as a graph or number.

06-07-2013, 11:37 AM	#11
Nesster Veteran Member Join Date: Oct 2006 Location: NJ USA Photos: Gallery Posts: 13,072 Original Poster	Originally posted by JinDesu "What happens is the center spot becomes larger in order to become more sensitive, so the camera can focus on a test target in near total darkness." Is that true? I don't know for sure! but trying to find out. https://www.pentaxforums.com/forums/pentax-k-5/177572-all-fuss-about-auto-focus-point-size.html http://www.luminous-landscape.com/tutorials/nikon-focus.shtml Last edited by Nesster; 06-07-2013 at 11:44 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
If you had the choice, what color kx would you get?	dancinonwater	Pentax DSLR Discussion	25	11-19-2010 09:05 PM
You may get what you deserved	jeffkrol	General Talk	13	11-06-2010 04:49 PM
If you were just starting, what PP program would you get?	coachteeter	Digital Processing, Software, and Printing	27	05-03-2010 03:14 PM
Is what you see what you get?	janstew	Troubleshooting and Beginner Help	9	01-11-2010 09:57 AM
What you see is not what you get	sebberry	Digital Processing, Software, and Printing	14	08-31-2009 08:17 AM

06-07-2013, 08:00 AM	#2
ElJamoquio Veteran Member Join Date: Feb 2011 Posts: 4,873	Originally posted by Nesster So what does this have to do with photography? I'm grumbling about auto focus. What are the measurements measuring? Speed of acquiring focus, and low light focus ability, and a count of focus points. All else being equal, these are fine things to pay attention to. But we are starting to get designs that seek to maximize performance in these tests, since these numbers sell cameras. What happens is the center spot becomes larger in order to become more sensitive, so the camera can focus on a test target in near total darkness. Great, but the trade off is uncertain focus placement in portraiture when using the center spot - it is too large to pinpoint the eye, or the photographer has to learn how to place the spot just so... We also get systems where actual photographic usability suffers - I want to be able to have a mode where I can select the point of focus easily and quickly, have positive feedback where the focus point is, and have the camera focus exactly at that point. This sort of test can be made and standardized, but it is not easy and quick to perform and present as a graph or number. I have the same pet peeve. I sent a manifesto to Pentax about it. I don't think they care, but I tried.

06-07-2013, 08:23 AM - 1 Like	#3
Kobayashi.K Veteran Member Join Date: Mar 2013 Photos: Albums Posts: 716	... "I want to be able to have a mode where I can select the point of focus easily and quickly, have positive feedback where the focus point is, and have the camera focus exactly at that point." That's manual focus. Try it.

06-07-2013, 10:37 AM	#4
Na Horuk Veteran Member Join Date: Feb 2012 Location: Slovenia, probably Photos: Gallery \| Albums Posts: 11,186	Well written, good points. Websites like snapsort especially encourage this. But its hard to compare without measurements, hard to critique without concrete numbers. I agree with your overall point, though. It would be great if Pentax could focus on photography, instead of "winning the review battles"

06-07-2013, 10:40 AM	#5
JinDesu Veteran Member Join Date: Jun 2011 Location: New York City Photos: Gallery Posts: 5,638	"What happens is the center spot becomes larger in order to become more sensitive, so the camera can focus on a test target in near total darkness." Is that true?

06-07-2013, 11:08 AM	#6
ElJamoquio Veteran Member Join Date: Feb 2011 Posts: 4,873	The Pentax's center point is huge compared to the competition, and does very well in the dark. Don't know that it proves anything, but a large (AF) sensor is much better in low light than a small sensor,for the same reason large imaging sensors are better than smaller imaging sensors.

06-07-2013, 11:13 AM	#7
wildman Veteran Member Join Date: Jun 2008 Location: Flyover America Posts: 4,469	Originally posted by Nesster we are starting to get designs that seek to maximize performance in these tests, since these numbers sell cameras. The very definition of "success" in a capitalistic market system. Rational market behavior does not necessarily = good.

06-07-2013, 11:19 AM	#8
DeadJohn Pentaxian Join Date: Jan 2011 Location: New York Posts: 4,834	My job involves tracking performance metrics for a government agency. Ever hear the story of a fire department pushed to improve 911 response time every year? Eventually, aggressive driving led to more firefighters being hurt on the way to fires than in fires. It happens with helpdesks too. Techs get evaluated on how many tickets they close. So, a dead PC generates dozens of tickets, one to replace the hardware and a separate ticket for each piece of software needed on the new PC. Word, Excel, Outlook, Visio, Project, Powerpoint equals 6 tickets.

06-07-2013, 11:19 AM	#9
JinDesu Veteran Member Join Date: Jun 2011 Location: New York City Photos: Gallery Posts: 5,638	Originally posted by ElJamoquio The Pentax's center point is huge compared to the competition, and does very well in the dark. Don't know that it proves anything, but a large (AF) sensor is much better in low light than a small sensor,for the same reason large imaging sensors are better than smaller imaging sensors. I don't know about "does very well in the dark". The current k-5II does very well in the dark. The previous k-5 didn't do as well in the dark. I don't think there is a AF point size difference between the two.

06-07-2013, 11:26 AM - 1 Like	#10
normhead Pentaxian Join Date: Jun 2007 Location: Near Algonquin Park Photos: Gallery \| Albums Posts: 40,451	Originally posted by Kobayashi.K ... "I want to be able to have a mode where I can select the point of focus easily and quickly, have positive feedback where the focus point is, and have the camera focus exactly at that point." That's manual focus. Try it. People used to say this and I'd kind of hear "wah wah wah", then I got my A-400. Now, I'm starting to look for lenses that work well MF. With MF I can select a spot in front of a bird where there is nothing to focus on. You have to ask, is there an AF system that will focus on nothing? But you can do that easily manually, and then snap back to an eye focus, not at 51 stinking focus points, but anywhere in the frame, without selecting anything but a slight movement of the focus ring. If there was an A* 200 ƒ4 I'd snap up the first one I saw. The combination of FL and weight would be awesome. I'm having a bit of trouble understanding why everyone wants to do auto-maticly what is so easy to do manually.

06-07-2013, 11:39 AM	#12
Nesster Veteran Member Join Date: Oct 2006 Location: NJ USA Photos: Gallery Posts: 13,072 Original Poster	Originally posted by normhead I'm having a bit of trouble understanding why everyone wants to do auto-maticly what is so easy to do manually. LOL, because if I'm going to auto focus, I'd like it to work for me, instead of the other way around. I'm lazy that way. I run into a similar situation with auto exposure vs manual -- I have to approach the thing in two very different manners. And with auto exposure I typically forget to apply brains to exposure compensation.

06-07-2013, 12:02 PM	#13
Nesster Veteran Member Join Date: Oct 2006 Location: NJ USA Photos: Gallery Posts: 13,072 Original Poster	LensRentals.com - Autofocus Reality Part 3B: Canon Cameras this article has some very interesting info

06-07-2013, 12:17 PM	#14
normhead Pentaxian Join Date: Jun 2007 Location: Near Algonquin Park Photos: Gallery \| Albums Posts: 40,451	You get the same thing with standardized testing in Education. You get whole schools focused on preparing for Standardized tests so they look good on province wide results that are published. They would actually walk into my photography class and give me booklets, so once or twice a week we could work on preparing for standardized tests in math, english and reading comprehension. The smart kids were totally bored and totally disengaged. Every other subject suffered and each passing year I could actually teach less and less of my assigned subjects. But my school was rising star on the test scores. In the 5 years before standardized testing was brought in, we put one student every year into either Sheridan or Ryerson to continue on in photography, some years two. After the standardized testing craze started, not one in 10 years. But the school was a "better" school, at least in the newspapers.

06-10-2013, 03:58 PM	#15
reeftool Pentaxian Join Date: Dec 2007 Location: Upstate New York Photos: Gallery \| Albums Posts: 9,554	Since I started out in photography with a manual camera which had a split prism viewfinder, to this day I focus in the center and reframe my shot if I need to. After taking pictures for over 50 years, I'm not going to change now. What do I want in auto focus? Get it right as quickly as possible. Most specifications don't really mean much to me. Is 50 auto focus points really more accurate than 10 or does it mean the camera has 40 more points to get it wrong? I'm waiting for a "bird in flight" spec. and the camera/lens that will nail it 98% of the time.