Larger buffer might actually be HW?

Gimbal · 10-31-2010, 02:37 PM

I'm beginning to think that the increased buffer size isn't any neat software trick but in fact is a bigger hardware memory, changed in the last minute before production and now enabled through a firmware update.

And why do I believe this?
Mainly because even if we compress the raw file on the fly before putting it in the buffer, it should still be a lot larger then a jpeg. And when shooting jpegs we obviously already compress before putting it in the buffer, or otherwise there wouldn't be any difference from shooting raw. And now with the new firmware we are getting as many raws as we where getting jpegs with the old firmware, that shouldn't be possible unless a jpeg are as big as a compressed raw, or unless they messed up the jpegs mode as well.

So my guess is that we now have a larger hardware buffer then was originally specified, and that is probably thanks to d7000 price bomb which also gave us iso 51200.

But then I haven't done the math as I don't have the numbers, so maybe this theory doesn't add up. (How large is a k5 jpeg?)

Parallax · 10-31-2010, 02:44 PM

Not news or rumor. Sigh.................
Off to the DSLR discussion forum.

Raylon · 10-31-2010, 03:13 PM

I doubt they just activated memory, memory is memory, it's either there or it's not...seems like a silly idea to me. I think they just figured out a way in software to allow the buffer to hold more. It would only have been in Pentax's advantage to advertise a bigger buffer. They would have known if they had a bigger one. They wouldn't have just waited and been like 'Oh by the way we tricked you into thinking it wasn't going to shoot a lot of photos in continous mode...'

JackBak · 10-31-2010, 03:21 PM

Originally posted by Raylon

I doubt they just activated memory, memory is memory, it's either there or it's not...seems like a silly idea to me. I think they just figured out a way in software to allow the buffer to hold more. It would only have been in Pentax's advantage to advertise a bigger buffer. They would have known if they had a bigger one. They wouldn't have just waited and been like 'Oh by the way we tricked you into thinking it wasn't going to shoot a lot of photos in continous mode...'

Yes, memory is memory BUT they could have designed the board for more memory and just populated it with say 1/2 of what the board could support. This would give them a higher profit margin if no one got upset or noticed (don't ask me how no one would have noticed). This is the kind of things us sneaky engineers always do - we design in some wiggle room. Now they keep the high price and do a quick HW upgrade by adding a chip, and if their software was smart and did an auto detect of memory, Bob's your uncle as our UK friends would say.

rawr · 10-31-2010, 03:34 PM

Some of the gurus on other forums have deduced all sorts of ways that the fps rate could have been improved via software changes alone, including improving the PEF algorithms, changing compression arrangements, putting in place an option for 12/14 bit RAW, etc etc etc. All pure speculation of course, but a lot of it sounds plausible. However at the end of the day only Pentax really knows what is going on inside their hardware.

Gimbal · 10-31-2010, 03:51 PM

Yes that was my first thoughts as well, but then it hit me that the key might be in the jpeg/raw ratio. But then I don't have the sizes of a normal jpeg nor a normal raw so I can't really tell. But I figured that an 8 bit lossy jpeg should be a lot smaller than a 12/14 bit lossless raw. And thus if we can fit 20 jpegs then we shouldn't be able to fit 20 raws.

But then again, if the jpegs really are as big as a compressed raw or they have messed up the continuous jpeg mode as well then indeed a software fix alone might do the trick.

TiminyCricket · 10-31-2010, 04:36 PM

Originally posted by Gimbal

I'm beginning to think that the increased buffer size isn't any neat software trick but in fact is a bigger hardware memory, changed in the last minute before production and now enabled through a firmware update.

It really has nothing to do with software, or firmware. Memory is memory. When you tell your camera to format a card it formats all available physical memory and any and all sectors not reported back as "bad" during the format should be utilized. Now, the camera software may tell the camera to limit the camera to let's say 7 shots whether the memory is actually filled, or not... I suppose that is possible... but I have no reason to see why they would purposely make the camera use substantially less buffer space than is available within the embedded flash circuitry.

Having said that (and as I have mentioned before on this forum) it boggles my mind why the RAW shooting buffer capacity is so low. Memory density and speed have increased dramatically since my own camera, the K10D, came out. We used to have RAMBUS, SDRAM, then DDR, then DDR2, and now DDR3. Each standard has brought with it higher memory density per square inch and increased throughput. If a typical RAW file from say a K-5 is (maybe) 30MB and you can squeeze off about 7 shots before the buffer is apparently filled that means a little over 200MB of internal flash is being utilized, plus the additional overhead space required for the file-system used by the flash memory for organization. So, it's likely they are using a very inexpensive 256MB flash chip on the main board and that is a paltry amount of memory these days to use in embedded circuitry. Even cell phones these days have up to 8GB or more of built-in flash memory integrated directly onto the board.

There are other things which could be limiting the buffer aside from the built-in flash, such as the bus (pipeline) between the built-in flash and the actual memory card slot. Either way, I think the burst rate is most certainly a hardware limitation and would not likely be affected by any software or firmware updates, unless such updates used a newer, more efficient compression algorithm to make the file sizes smaller and allow more shots into the buffer at a time. Of course, RAW doesn't use compression in the first place, so there's really no way to do that.

Smeggypants · 10-31-2010, 05:28 PM

Originally posted by JackBak

Yes, memory is memory BUT they could have designed the board for more memory and just populated it with say 1/2 of what the board could support. This would give them a higher profit margin if no one got upset or noticed (don't ask me how no one would have noticed). This is the kind of things us sneaky engineers always do - we design in some wiggle room. Now they keep the high price and do a quick HW upgrade by adding a chip, and if their software was smart and did an auto detect of memory, Bob's your uncle as our UK friends would say.

This makes a lot of sense. I've seen lots of boards with pads for extra chips.

TiminyCricket · 10-31-2010, 05:38 PM

Originally posted by Smeggypants

This makes a lot of sense. I've seen lots of boards with pads for extra chips.

In either case built-in flash memory is almost always soldered directly to the mainboard to reduce latency. There could be additional spaces for memory on the board, but it wouldn't be as simple as just plugging another chip into the mainboard to increase performance, the memory would have to be soldered in and the firmware would have to be told to look for memory in that location.

johnmflores · 10-31-2010, 05:48 PM

Memory is a commodity, so it's not out of the realm of possibility that when they started building the cameras that they got a deal on larger memory chips that they could not say no to.

It's also not outside the realm of possibility that the firmware was written with the originally-specced memory in mind and the programmers had put a hard stop on the upper limit to avoid buffer-overrun errors.

So it's quite possible that they started building the cameras with a larger buffer than originally planned and then had to update firmware to match the new buffer size. I'd believe this before any 11th hour theories.

TiminyCricket · 10-31-2010, 06:00 PM

Originally posted by johnmflores

Memory is a commodity, so it's not out of the realm of possibility that when they started building the cameras that they got a deal on larger memory chips that they could not say no to.

It's also not outside the realm of possibility that the firmware was written with the originally-specced memory in mind and the programmers had put a hard stop on the upper limit to avoid buffer-overrun errors.

So it's quite possible that they started building the cameras with a larger buffer than originally planned and then had to update firmware to match the new buffer size. I'd believe this before any 11th hour theories.

It's also not uncommon for manufacturers to switch memory suppliers whilst in the middle of production (like say switching from Hynix memory to Samsung memory). Each manufacturer's chips may follow the same memory standard and contain the same rated capacity, but one vendor's memory chips can be faster than the others (gold-plated leads instead of silver, smaller nanometer manufacturing process, increased heat dissipation, etc.)

noblepa · 10-31-2010, 06:35 PM

Originally posted by Raylon

I doubt they just activated memory, memory is memory, it's either there or it's not...seems like a silly idea to me. I think they just figured out a way in software to allow the buffer to hold more. It would only have been in Pentax's advantage to advertise a bigger buffer. They would have known if they had a bigger one. They wouldn't have just waited and been like 'Oh by the way we tricked you into thinking it wasn't going to shoot a lot of photos in continous mode...'

True. Memory is memory, but the software must know how much is there and be able to use it.

For example, if you have a computer that is running the 32 bit version of Windows (any version, XP, Vista or 7) on a motherboard with 8GB of RAM installed, Windows will only "see" and be able to use 4GB of it. The other 4GB will go unused, because a 32 bit address can only address 4GB of memory. That is the major difference between the 32 bit and 64 bit versions of the OS. Upgrade to a 64 bit OS and, Voila, the other 4 GB magically appears.

On a hard drive, you can create a partition that is less than the full capacity of the drive. For example, if I have a 500GB drive, I can create a partition that is only 100GB and leave the other 400GB unpartitioned, and unuseable. I just popped a 2GB SD card into my PC (32bit, Windows XP Pro) and opened Disk Management. I could not delete the partition, so I could not create a new partition. I suspect that this is because Windows does not write a partition table on flash memory cards, the way it does on a HDD.

I have done a fair amount of Assembler and machine-language programming in my day. Even on a sophisticated, general purpose operating system, such as Windows, Linux or z/OS (IBM mainframe), the programmer must program the amount of memory to be allocated for a given storage area. The OS in a camera is not all that sophisticated or general-purpose, and memory hardware usually doesn't change, so a programmer would not waste his/her time writing the program in such a way that it would examine its surroundings and allocate as much memory as is available. It could be done, but since you control both the software and the hardware environment in which it will run, and since the memory size usually doesn't change, you would be wasting your time and valuable memory for the instructions to do this.

So, as someone who has done this type of programming, I can very easily believe that it is at least possible that the software was developed with a target memory size, only to have that physical memory size changed in the hardware at the last minute. Making such a change to the software would not be difficult, but you would want to be very, very careful and do all sorts of regression testing before releasing the new software. Again, as someone who has been there, I can tell you that even for simple changes, the law of unintended consequences is in full force. A seemingly trivial change can cause unbelieveable havoc with a program, in totally unexpected ways, so nothing is ever taken for granted. You test, test again and then test some more.

I doubt if Pentax will ever release the real story behind this buffer upgrade, so we're left to speculations like this.

Jodokast96 · 10-31-2010, 07:03 PM

So they have enough time to add memory at the last minute, but not enough time to make a simple change to the firmware to utilize it? I find that very unlikely. Do you really think the production line was only started up after the release of the D7000 to be able to get that memory in there?

Smeggypants · 10-31-2010, 07:14 PM

Originally posted by TiminyCricket

In either case built-in flash memory is almost always soldered directly to the mainboard to reduce latency. There could be additional spaces for memory on the board, but it wouldn't be as simple as just plugging another chip into the mainboard to increase performance, the memory would have to be soldered in and the firmware would have to be told to look for memory in that location.

That's right you wouldn't get sockets, especailly in a camera, they are just unpopulated pads ready to add another chip to the automated production line process. Scokets not only cost, they would add an unreliablity factor. There's definitely new firmware, so as said it does make sense that extra memory was added late in the day.

Although there's also the theory that the memory wasalready there.

Anyway, it's all good.

secateurs · 10-31-2010, 07:17 PM

Originally posted by johnmflores

Memory is a commodity, so it's not out of the realm of possibility that when they started building the cameras that they got a deal on larger memory chips that they could not say no to.

I very much doubt the memory size was updated mid-production. If they did this, they could only offer the firmware upgrade and buffer increase to the cameras with the different memory chip.

This sort of HW change is usually reserved for MkII products.

However, there is still a possibility that a higher memory chip was swapped into the design before the K-5 went to production.

I still like Falk's theory though.

10-31-2010, 02:37 PM	#1
Gimbal Pentaxian Join Date: Apr 2007 Location: Sweden Posts: 2,106	Larger buffer might actually be HW? I'm beginning to think that the increased buffer size isn't any neat software trick but in fact is a bigger hardware memory, changed in the last minute before production and now enabled through a firmware update. And why do I believe this? Mainly because even if we compress the raw file on the fly before putting it in the buffer, it should still be a lot larger then a jpeg. And when shooting jpegs we obviously already compress before putting it in the buffer, or otherwise there wouldn't be any difference from shooting raw. And now with the new firmware we are getting as many raws as we where getting jpegs with the old firmware, that shouldn't be possible unless a jpeg are as big as a compressed raw, or unless they messed up the jpegs mode as well. So my guess is that we now have a larger hardware buffer then was originally specified, and that is probably thanks to d7000 price bomb which also gave us iso 51200. But then I haven't done the math as I don't have the numbers, so maybe this theory doesn't add up. (How large is a k5 jpeg?)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
K5 Buffer	BrendanPK	Pentax K-5 & K-5 II	133	10-30-2010 01:09 AM
K-5 / K-7 : eight raw buffer limit exists	gubak	Pentax K-5 & K-5 II	54	10-19-2010 10:52 AM
K7 Buffer Issue?	samholton	Pentax DSLR Discussion	4	01-08-2010 07:19 AM
K-7 40 jpeg buffer enough?	Pentaxor	Pentax DSLR Discussion	5	06-28-2009 01:28 PM
K100D Super Buffer?	res3567	Pentax DSLR Discussion	7	05-10-2009 05:13 AM

10-31-2010, 02:44 PM	#2
Parallax Senior Moderator Loyal Site Supporter Join Date: Apr 2008 Location: South Dakota Photos: Gallery Posts: 19,332	Not news or rumor. Sigh................. Off to the DSLR discussion forum.

10-31-2010, 03:13 PM	#3
Raylon Guest	I doubt they just activated memory, memory is memory, it's either there or it's not...seems like a silly idea to me. I think they just figured out a way in software to allow the buffer to hold more. It would only have been in Pentax's advantage to advertise a bigger buffer. They would have known if they had a bigger one. They wouldn't have just waited and been like 'Oh by the way we tricked you into thinking it wasn't going to shoot a lot of photos in continous mode...'

10-31-2010, 03:21 PM	#4
JackBak Forum Member Join Date: Jul 2008 Location: Boulder, Colorado Posts: 88	Originally posted by Raylon I doubt they just activated memory, memory is memory, it's either there or it's not...seems like a silly idea to me. I think they just figured out a way in software to allow the buffer to hold more. It would only have been in Pentax's advantage to advertise a bigger buffer. They would have known if they had a bigger one. They wouldn't have just waited and been like 'Oh by the way we tricked you into thinking it wasn't going to shoot a lot of photos in continous mode...' Yes, memory is memory BUT they could have designed the board for more memory and just populated it with say 1/2 of what the board could support. This would give them a higher profit margin if no one got upset or noticed (don't ask me how no one would have noticed). This is the kind of things us sneaky engineers always do - we design in some wiggle room. Now they keep the high price and do a quick HW upgrade by adding a chip, and if their software was smart and did an auto detect of memory, Bob's your uncle as our UK friends would say.

10-31-2010, 03:34 PM	#5
rawr Veteran Member Join Date: Jun 2009 Posts: 11,913	Some of the gurus on other forums have deduced all sorts of ways that the fps rate could have been improved via software changes alone, including improving the PEF algorithms, changing compression arrangements, putting in place an option for 12/14 bit RAW, etc etc etc. All pure speculation of course, but a lot of it sounds plausible. However at the end of the day only Pentax really knows what is going on inside their hardware.

10-31-2010, 03:51 PM	#6
Gimbal Pentaxian Join Date: Apr 2007 Location: Sweden Posts: 2,106 Original Poster	Yes that was my first thoughts as well, but then it hit me that the key might be in the jpeg/raw ratio. But then I don't have the sizes of a normal jpeg nor a normal raw so I can't really tell. But I figured that an 8 bit lossy jpeg should be a lot smaller than a 12/14 bit lossless raw. And thus if we can fit 20 jpegs then we shouldn't be able to fit 20 raws. But then again, if the jpegs really are as big as a compressed raw or they have messed up the continuous jpeg mode as well then indeed a software fix alone might do the trick.

10-31-2010, 04:36 PM	#7
TiminyCricket Forum Member Join Date: Aug 2008 Location: New Jersey Posts: 76	Originally posted by Gimbal I'm beginning to think that the increased buffer size isn't any neat software trick but in fact is a bigger hardware memory, changed in the last minute before production and now enabled through a firmware update. It really has nothing to do with software, or firmware. Memory is memory. When you tell your camera to format a card it formats all available physical memory and any and all sectors not reported back as "bad" during the format should be utilized. Now, the camera software may tell the camera to limit the camera to let's say 7 shots whether the memory is actually filled, or not... I suppose that is possible... but I have no reason to see why they would purposely make the camera use substantially less buffer space than is available within the embedded flash circuitry. Having said that (and as I have mentioned before on this forum) it boggles my mind why the RAW shooting buffer capacity is so low. Memory density and speed have increased dramatically since my own camera, the K10D, came out. We used to have RAMBUS, SDRAM, then DDR, then DDR2, and now DDR3. Each standard has brought with it higher memory density per square inch and increased throughput. If a typical RAW file from say a K-5 is (maybe) 30MB and you can squeeze off about 7 shots before the buffer is apparently filled that means a little over 200MB of internal flash is being utilized, plus the additional overhead space required for the file-system used by the flash memory for organization. So, it's likely they are using a very inexpensive 256MB flash chip on the main board and that is a paltry amount of memory these days to use in embedded circuitry. Even cell phones these days have up to 8GB or more of built-in flash memory integrated directly onto the board. There are other things which could be limiting the buffer aside from the built-in flash, such as the bus (pipeline) between the built-in flash and the actual memory card slot. Either way, I think the burst rate is most certainly a hardware limitation and would not likely be affected by any software or firmware updates, unless such updates used a newer, more efficient compression algorithm to make the file sizes smaller and allow more shots into the buffer at a time. Of course, RAW doesn't use compression in the first place, so there's really no way to do that.

10-31-2010, 05:28 PM	#8
Smeggypants Veteran Member Join Date: Sep 2010 Posts: 1,536	Originally posted by JackBak Yes, memory is memory BUT they could have designed the board for more memory and just populated it with say 1/2 of what the board could support. This would give them a higher profit margin if no one got upset or noticed (don't ask me how no one would have noticed). This is the kind of things us sneaky engineers always do - we design in some wiggle room. Now they keep the high price and do a quick HW upgrade by adding a chip, and if their software was smart and did an auto detect of memory, Bob's your uncle as our UK friends would say. This makes a lot of sense. I've seen lots of boards with pads for extra chips.

10-31-2010, 05:38 PM	#9
TiminyCricket Forum Member Join Date: Aug 2008 Location: New Jersey Posts: 76	Originally posted by Smeggypants This makes a lot of sense. I've seen lots of boards with pads for extra chips. In either case built-in flash memory is almost always soldered directly to the mainboard to reduce latency. There could be additional spaces for memory on the board, but it wouldn't be as simple as just plugging another chip into the mainboard to increase performance, the memory would have to be soldered in and the firmware would have to be told to look for memory in that location.

10-31-2010, 05:48 PM	#10
johnmflores Veteran Member Join Date: Apr 2009 Location: Somerville, NJ Photos: Gallery \| Albums Posts: 5,361	Memory is a commodity, so it's not out of the realm of possibility that when they started building the cameras that they got a deal on larger memory chips that they could not say no to. It's also not outside the realm of possibility that the firmware was written with the originally-specced memory in mind and the programmers had put a hard stop on the upper limit to avoid buffer-overrun errors. So it's quite possible that they started building the cameras with a larger buffer than originally planned and then had to update firmware to match the new buffer size. I'd believe this before any 11th hour theories.

10-31-2010, 06:00 PM	#11
TiminyCricket Forum Member Join Date: Aug 2008 Location: New Jersey Posts: 76	Originally posted by johnmflores Memory is a commodity, so it's not out of the realm of possibility that when they started building the cameras that they got a deal on larger memory chips that they could not say no to. It's also not outside the realm of possibility that the firmware was written with the originally-specced memory in mind and the programmers had put a hard stop on the upper limit to avoid buffer-overrun errors. So it's quite possible that they started building the cameras with a larger buffer than originally planned and then had to update firmware to match the new buffer size. I'd believe this before any 11th hour theories. It's also not uncommon for manufacturers to switch memory suppliers whilst in the middle of production (like say switching from Hynix memory to Samsung memory). Each manufacturer's chips may follow the same memory standard and contain the same rated capacity, but one vendor's memory chips can be faster than the others (gold-plated leads instead of silver, smaller nanometer manufacturing process, increased heat dissipation, etc.)

Thread Tools	Search this Thread
	Search this Thread: Advanced Search

10-31-2010, 06:35 PM	#12
noblepa Veteran Member Join Date: Sep 2007 Location: Bay Village, Ohio USA Posts: 1,142	Originally posted by Raylon I doubt they just activated memory, memory is memory, it's either there or it's not...seems like a silly idea to me. I think they just figured out a way in software to allow the buffer to hold more. It would only have been in Pentax's advantage to advertise a bigger buffer. They would have known if they had a bigger one. They wouldn't have just waited and been like 'Oh by the way we tricked you into thinking it wasn't going to shoot a lot of photos in continous mode...' True. Memory is memory, but the software must know how much is there and be able to use it. For example, if you have a computer that is running the 32 bit version of Windows (any version, XP, Vista or 7) on a motherboard with 8GB of RAM installed, Windows will only "see" and be able to use 4GB of it. The other 4GB will go unused, because a 32 bit address can only address 4GB of memory. That is the major difference between the 32 bit and 64 bit versions of the OS. Upgrade to a 64 bit OS and, Voila, the other 4 GB magically appears. On a hard drive, you can create a partition that is less than the full capacity of the drive. For example, if I have a 500GB drive, I can create a partition that is only 100GB and leave the other 400GB unpartitioned, and unuseable. I just popped a 2GB SD card into my PC (32bit, Windows XP Pro) and opened Disk Management. I could not delete the partition, so I could not create a new partition. I suspect that this is because Windows does not write a partition table on flash memory cards, the way it does on a HDD. I have done a fair amount of Assembler and machine-language programming in my day. Even on a sophisticated, general purpose operating system, such as Windows, Linux or z/OS (IBM mainframe), the programmer must program the amount of memory to be allocated for a given storage area. The OS in a camera is not all that sophisticated or general-purpose, and memory hardware usually doesn't change, so a programmer would not waste his/her time writing the program in such a way that it would examine its surroundings and allocate as much memory as is available. It could be done, but since you control both the software and the hardware environment in which it will run, and since the memory size usually doesn't change, you would be wasting your time and valuable memory for the instructions to do this. So, as someone who has done this type of programming, I can very easily believe that it is at least possible that the software was developed with a target memory size, only to have that physical memory size changed in the hardware at the last minute. Making such a change to the software would not be difficult, but you would want to be very, very careful and do all sorts of regression testing before releasing the new software. Again, as someone who has been there, I can tell you that even for simple changes, the law of unintended consequences is in full force. A seemingly trivial change can cause unbelieveable havoc with a program, in totally unexpected ways, so nothing is ever taken for granted. You test, test again and then test some more. I doubt if Pentax will ever release the real story behind this buffer upgrade, so we're left to speculations like this.

10-31-2010, 07:03 PM	#13
Jodokast96 Veteran Member Join Date: Oct 2007 Location: Erial, NJ USA Photos: Gallery \| Albums Posts: 2,134	So they have enough time to add memory at the last minute, but not enough time to make a simple change to the firmware to utilize it? I find that very unlikely. Do you really think the production line was only started up after the release of the D7000 to be able to get that memory in there?

10-31-2010, 07:14 PM	#14
Smeggypants Veteran Member Join Date: Sep 2010 Posts: 1,536	Originally posted by TiminyCricket In either case built-in flash memory is almost always soldered directly to the mainboard to reduce latency. There could be additional spaces for memory on the board, but it wouldn't be as simple as just plugging another chip into the mainboard to increase performance, the memory would have to be soldered in and the firmware would have to be told to look for memory in that location. That's right you wouldn't get sockets, especailly in a camera, they are just unpopulated pads ready to add another chip to the automated production line process. Scokets not only cost, they would add an unreliablity factor. There's definitely new firmware, so as said it does make sense that extra memory was added late in the day. Although there's also the theory that the memory wasalready there. Anyway, it's all good.

10-31-2010, 07:17 PM	#15
secateurs Senior Member Join Date: Oct 2010 Location: Adelaide, Australia Photos: Albums Posts: 250	Originally posted by johnmflores Memory is a commodity, so it's not out of the realm of possibility that when they started building the cameras that they got a deal on larger memory chips that they could not say no to. I very much doubt the memory size was updated mid-production. If they did this, they could only offer the firmware upgrade and buffer increase to the cameras with the different memory chip. This sort of HW change is usually reserved for MkII products. However, there is still a possibility that a higher memory chip was swapped into the design before the K-5 went to production. I still like Falk's theory though.