damaged files? what is wrong with my images?

pete_pf · 02-13-2011, 05:21 PM

Originally posted by eddie1960

that is odd, as when i look at the preview on importing it's fine but once i have it in lightroom it's the same issue. So it's not your software, it could still be a corrupted hard drive partition

try a clean import of a file and work from the memory card and see if it is a problem

can't already formatted the card lol boo. can you redownload that file? i added the pef file i took from my laptop, and it seems to be loading ok (for now lol)

wondering if anyone can find why the one pef loads fine and the other doesn't? i know the names are different, but even when i rename the file back to the original on my desktop it goes corrupt. this one in the zip is renamed just like i did on my desktop. also want to remind people that i got the corruption on my laptop as well, so don't say it's my desktop :P

Last edited by pete_pf; 02-13-2011 at 05:50 PM.

eddie1960 · 02-13-2011, 07:37 PM

Ill tty agsin when im back online
May not be till tomorrow though

pete_pf · 02-13-2011, 08:31 PM

i know this will bite me in the ass for posting. ok, so i went on my laptop to see the image to see if it loads there (i originally saved it there from my card). it did, so i saved it in my dropbox and then checked it on my computer (where i'm currently having the issues) i load up the image, everything seems ok. i bring in my xmp file, all my editing is there, no issues so far. i read the other comment about renaming and exif issues, so i didn't rename it like i usually do this time. but so far the image loads and edits fine, so now i'm even more stumped about this lol

my original process from my memory card was 'add metadata > rename-move to laptop > move to external hard drive' - why all the moving? originally put on my laptop to view and edit a little, then realized editing on laptop sucks, so moved to desktop. i'm wondering, possible there's some read/write issue when i move from laptop to external hard drive? my external was formatted under xp, maybe 7 isn't reading them probably when i move them over? next time i'll move straight from camera to external to test that theory out.

sterretje · 02-13-2011, 10:17 PM

Originally posted by twitch

I'm not sure if this has been mentioned but I tihnk it's an important point.

What is seen on the camera LCD, and on the computer screen during and soon after import is not the RAW file at all (or even the computer converted RAW file), it is the JPEG embedded within the RAW file. Once the computer processes the RAW file it then generates it's own JPEG preview which it uses to display the photo on the screen rather than the JPEG embedded in the RAW file. This can take from seconds to many minutes depending on how many images were imported and how fast the computer is.

You can never see a RAW file displayed, only the JPEG/TIFF interpretation of it.

If the file "goes crazy" it could be that the conversion of the RAW file to JPEG on the computer has screwed up, so as it replaces the embedded JPEG for it's preview with the RAW converted one it's at that point that it may be screwing up.

It is my understanding that when pete_pf starts editing the image is OK. So that should not be the embedded jpeg but the converted raw.

pete_pf · 02-13-2011, 10:21 PM

Originally posted by sterretje

It is my understanding that when pete_pf starts editing the image is OK. So that should not be the embedded jpeg but the converted raw.

yes. the raw file is usually ok for about a day lol i did literally nothing to the image but basic raw editing, i click save, then ctrl+r at a later time, and it goes funky.

has anyone had their images goes nuts like this, or am i the only one? lol please let there be others except me :P

sterretje · 02-13-2011, 10:25 PM

So to get the overview, this is your process

Download files from camera to laptop
Edit on laptop
Move to external harddisk (make this a copy in future till you're sure the copied image is OK)
Connect external HD to desktop
Edit on external HD and the problem shows at occasion

Is this a correct summary?

pete_pf · 02-13-2011, 10:33 PM

Originally posted by sterretje

So to get the overview, this is your process

Download files from camera to laptop
Edit on laptop
Move to external harddisk (make this a copy in future till you're sure the copied image is OK)
Connect external HD to desktop
Edit on external HD and the problem shows at occasion

Is this a correct summary?

yuuup

from laptop to external it's always copy because from folder to folder windows copies by default, but from camera to where ever i usually move by renaming straight from sd card inside bridge. "problem shows at occasion" exactly why this is so annoying. if it happened to every image every time, i could understand. but that it happens so randomly ... aggh.

jolepp · 02-14-2011, 03:46 AM

My guess currently would be that you are seeing the effects of a marginal data block somewhere in the chain of medias (of which the memory card is least suspect as it has passed the polygraf alredy

). Actually, the problematic block might even fail quite consistently but then the effect of that could vary depending on what exact location a image file it ends up.

If you have the space you might let the copies that are created at various stages to to stick around. Then, when/if this happens next time you could go back in the chain of copies to locate the first copy without / with the problem and then take it from there: e.g. if the copy on the laptop is good, but the one on the external HD is not, then the HD would seem to be the one to blame.

I suppose H2testw could be applied to the other disks in the copy chain as well; it simply writes out a file with generated data on the disk in a location you can give, reads the file back and checks it gets the expected data back (and finally gets rid of the test file). The check part failing should mean having identified the culprit. Another thing you might do (for all the memory cards and hard disks involved would be "MyComputer - <disk X> - Properties - Tools - Check for errors" with the option "scan for bad sectors" checked in addition to "automatically fix ..." (takes a fair bit of time, as it goes through the entire disk). In case Windows finds "bad sectors" its marking them as such should mean that you image files do not end up using these (it is worth noting that the list of bad sectors is cleared with format).

borno · 02-14-2011, 03:57 AM

I wouldn't rename the raws if that is what you are doing. Put them in a folder with a name of your choosing and rename after converting to .jpg . I would look at the simplest solution first.

0rangepeel · 02-14-2011, 10:22 PM

I had this problem when I used an external SD card reader. It was a really cheap 5$ one, but it was blazing fast.

Try transferring from your camera instead of using an SD card reader. Or alternatively, try using another sd card reader.

My theory is that the cable you use for USB may not be well shielded enough, so corrupted data is flying by.

jolepp · 02-15-2011, 08:56 AM

Originally posted by 0rangepeel

I had this problem when I used an external SD card reader...

Good point, a flaky reader / cable could cause occasional corruption like this. In general the really annoying part of this sort of thing would seem to be infrequent-but-still-sometimes-happens -ness. On a related note I wonder whether Windows still retries a failed i/o operation behind the scenes which tends to result in things getting flaky going unnoticed until the trouble gets so bad that a couple of retries do not help any more.

pete_pf · 02-15-2011, 09:35 AM

Originally posted by 0rangepeel

I had this problem when I used an external SD card reader. It was a really cheap 5$ one, but it was blazing fast.

Try transferring from your camera instead of using an SD card reader. Or alternatively, try using another sd card reader.

My theory is that the cable you use for USB may not be well shielded enough, so corrupted data is flying by.

ohh man wouldn't that be heaven is this was the problem? instead of memory or hard drive issues that it could be as simple as getting a new usb cable or reader?! i wonder if that's the issue though, because when i moved from camera to laptop the image is fine. but when i connected my laptop to my external hard drive, that is when the image went nuts. also when i moved the images from this post from desktop to laptop i used a card reader. i would just love is this was the problem

with the most recent image that went nuts, i recently put the image into my dropbox and downloaded to my desktop, and it's working fine so far on there. so really has me thinking hmm

sterretje · 02-15-2011, 11:45 PM

You have to do some trouble shooting (with sensible thinking) and draw the conclusions. You have a system now where files don't get corrupted and you're now working on images that are on the HD of your desktop (if I understand it correctly).

What is the difference between the current setup (dropbox) and the previous one (external HD)? That you don't use your external HD anymore.

So what can be wrong? External HD, cable, usb ports on both laptop and desktop.

The sensible sequence of tests will start at the source (laptop/external HD). I do have a suspicion as to what is going wrong so I will start at the other side. Where do you connect your external HD to the desktop? Front or back? Assuming this is an external HD that gets its power from the USB ports and you connect to the front, you can eliminate the front USB as a source by using the USB at the back.

1)
1a) Copy images from your laptop to the external HD.
1b) Connect external HD to one of the USB ports on the back of the desktop.
1c) Edit a couple of them on the external HD.
1d) Does the problem occur?

If not, the problem is the USB ports on the front of the desktop. And the (more than likely) reason is that those ports can’t deliver enough power to the external HD because they are wired with thin wires and when too much power is required, the voltage will drop. And I think that that is currently your problem.
If yes, no conclusions can be drawn except that my suspicion was wrong and I would start at the source (laptop / external HD)

2)
2a) Copy images from your laptop to the external HD.
2b) Edit a couple of them on the external HD using the laptop.
2c) Does the problem occur?

If not, I would conclude that it's the combination desktop / external HD that causes the issue; go to step 3
If yes, test the external HD while connected to the laptop; maybe jolepp's tools that you've already used can do that (not sure). Also test laptop's memory and internal HD.

3)
I’m not sure how to continue. The dropbox solution proves that your desktop is basically OK (possibly except for USB). I would test the external HD while connected to the desktop using jolepp’s tools (on the front USB as well as on the back USB).

Good luck.

PS Don’t ask me why this problem shows up in Win7 and not in WinXP. Only God knows

dvest · 02-15-2011, 11:59 PM

Sounds to me like it's the external drive.

DeadJohn · 02-16-2011, 02:32 PM

I just caught up on this thread. twitch made a very good point mentioning that the RAW file contains an embedded JPG for preview purposes ... another complexity to consider while trying to find the source/fix for this frustrating corruption problem.

A question to everyone: wouldn't an intermittent problem with a memory card, cable, or hard drive likely cause additional problems? It seems like the photo corruption is always happening in a similar manner. I would expect a memory problem to sometimes corrupt files badly enough to prevent opening them.

pete_pf, the 3 sample images you provided all start with a good band at the top. Have you had any photos that were 100% garbled? That started out bad at the top but then were good at the bottom? That were so bad you got an error message trying to open them?

02-14-2011, 03:46 AM	#38
jolepp Veteran Member Join Date: Oct 2010 Location: Finland Photos: Albums Posts: 3,196	My guess currently would be that you are seeing the effects of a marginal data block somewhere in the chain of medias (of which the memory card is least suspect as it has passed the polygraf alredy ). Actually, the problematic block might even fail quite consistently but then the effect of that could vary depending on what exact location a image file it ends up. If you have the space you might let the copies that are created at various stages to to stick around. Then, when/if this happens next time you could go back in the chain of copies to locate the first copy without / with the problem and then take it from there: e.g. if the copy on the laptop is good, but the one on the external HD is not, then the HD would seem to be the one to blame. I suppose H2testw could be applied to the other disks in the copy chain as well; it simply writes out a file with generated data on the disk in a location you can give, reads the file back and checks it gets the expected data back (and finally gets rid of the test file). The check part failing should mean having identified the culprit. Another thing you might do (for all the memory cards and hard disks involved would be "MyComputer - <disk X> - Properties - Tools - Check for errors" with the option "scan for bad sectors" checked in addition to "automatically fix ..." (takes a fair bit of time, as it goes through the entire disk). In case Windows finds "bad sectors" its marking them as such should mean that you image files do not end up using these (it is worth noting that the list of bad sectors is cleared with format). Last edited by jolepp; 02-14-2011 at 05:23 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
RAW duplicate images - saving as RAW files	jpzk	Digital Processing, Software, and Printing	17	02-15-2010 07:01 PM
DxO doesn't correct images/ Am I doing sth. wrong?	Egg Salad	Digital Processing, Software, and Printing	9	01-28-2010 05:35 PM
Damaged Files - How to recover	sabarrett	Digital Processing, Software, and Printing	9	10-30-2008 04:24 PM
yesterday's images... (warning - larger files x3)	Marc Langille	Post Your Photos!	24	11-14-2007 05:09 PM
New hummingbird images... (warning- some large files)	Marc Langille	Post Your Photos!	24	09-29-2007 09:57 AM

02-13-2011, 05:21 PM	#31
pete_pf Veteran Member Join Date: Nov 2007 Location: Toronto, Canada Posts: 599 Original Poster	Originally posted by eddie1960 that is odd, as when i look at the preview on importing it's fine but once i have it in lightroom it's the same issue. So it's not your software, it could still be a corrupted hard drive partition try a clean import of a file and work from the memory card and see if it is a problem can't already formatted the card lol boo. can you redownload that file? i added the pef file i took from my laptop, and it seems to be loading ok (for now lol) wondering if anyone can find why the one pef loads fine and the other doesn't? i know the names are different, but even when i rename the file back to the original on my desktop it goes corrupt. this one in the zip is renamed just like i did on my desktop. also want to remind people that i got the corruption on my laptop as well, so don't say it's my desktop :P Last edited by pete_pf; 02-13-2011 at 05:50 PM.

02-13-2011, 07:37 PM	#32
eddie1960 Veteran Member Join Date: Aug 2010 Location: Toronto Photos: Gallery \| Albums Posts: 13,667	Ill tty agsin when im back online May not be till tomorrow though

02-13-2011, 08:31 PM	#33
pete_pf Veteran Member Join Date: Nov 2007 Location: Toronto, Canada Posts: 599 Original Poster	i know this will bite me in the ass for posting. ok, so i went on my laptop to see the image to see if it loads there (i originally saved it there from my card). it did, so i saved it in my dropbox and then checked it on my computer (where i'm currently having the issues) i load up the image, everything seems ok. i bring in my xmp file, all my editing is there, no issues so far. i read the other comment about renaming and exif issues, so i didn't rename it like i usually do this time. but so far the image loads and edits fine, so now i'm even more stumped about this lol my original process from my memory card was 'add metadata > rename-move to laptop > move to external hard drive' - why all the moving? originally put on my laptop to view and edit a little, then realized editing on laptop sucks, so moved to desktop. i'm wondering, possible there's some read/write issue when i move from laptop to external hard drive? my external was formatted under xp, maybe 7 isn't reading them probably when i move them over? next time i'll move straight from camera to external to test that theory out.

02-13-2011, 10:17 PM	#34
sterretje Veteran Member Join Date: Dec 2008 Location: Roodepoort, South Africa Photos: Gallery \| Albums Posts: 3,561	Originally posted by twitch I'm not sure if this has been mentioned but I tihnk it's an important point. What is seen on the camera LCD, and on the computer screen during and soon after import is not the RAW file at all (or even the computer converted RAW file), it is the JPEG embedded within the RAW file. Once the computer processes the RAW file it then generates it's own JPEG preview which it uses to display the photo on the screen rather than the JPEG embedded in the RAW file. This can take from seconds to many minutes depending on how many images were imported and how fast the computer is. You can never see a RAW file displayed, only the JPEG/TIFF interpretation of it. If the file "goes crazy" it could be that the conversion of the RAW file to JPEG on the computer has screwed up, so as it replaces the embedded JPEG for it's preview with the RAW converted one it's at that point that it may be screwing up. It is my understanding that when pete_pf starts editing the image is OK. So that should not be the embedded jpeg but the converted raw.

02-13-2011, 10:21 PM	#35
pete_pf Veteran Member Join Date: Nov 2007 Location: Toronto, Canada Posts: 599 Original Poster	Originally posted by sterretje It is my understanding that when pete_pf starts editing the image is OK. So that should not be the embedded jpeg but the converted raw. yes. the raw file is usually ok for about a day lol i did literally nothing to the image but basic raw editing, i click save, then ctrl+r at a later time, and it goes funky. has anyone had their images goes nuts like this, or am i the only one? lol please let there be others except me :P

02-13-2011, 10:25 PM	#36
sterretje Veteran Member Join Date: Dec 2008 Location: Roodepoort, South Africa Photos: Gallery \| Albums Posts: 3,561	So to get the overview, this is your process Download files from camera to laptop Edit on laptop Move to external harddisk (make this a copy in future till you're sure the copied image is OK) Connect external HD to desktop Edit on external HD and the problem shows at occasion Is this a correct summary?

02-13-2011, 10:33 PM	#37
pete_pf Veteran Member Join Date: Nov 2007 Location: Toronto, Canada Posts: 599 Original Poster	Originally posted by sterretje So to get the overview, this is your process Download files from camera to laptop Edit on laptop Move to external harddisk (make this a copy in future till you're sure the copied image is OK) Connect external HD to desktop Edit on external HD and the problem shows at occasion Is this a correct summary? yuuup from laptop to external it's always copy because from folder to folder windows copies by default, but from camera to where ever i usually move by renaming straight from sd card inside bridge. "problem shows at occasion" exactly why this is so annoying. if it happened to every image every time, i could understand. but that it happens so randomly ... aggh.

02-14-2011, 03:57 AM	#39
borno Veteran Member Join Date: Nov 2006 Location: md-usa Photos: Gallery \| Albums Posts: 2,580	I wouldn't rename the raws if that is what you are doing. Put them in a folder with a name of your choosing and rename after converting to .jpg . I would look at the simplest solution first.

02-14-2011, 10:22 PM	#40
0rangepeel Senior Member Join Date: Nov 2009 Location: Vancouver Photos: Gallery Posts: 111	I had this problem when I used an external SD card reader. It was a really cheap 5$ one, but it was blazing fast. Try transferring from your camera instead of using an SD card reader. Or alternatively, try using another sd card reader. My theory is that the cable you use for USB may not be well shielded enough, so corrupted data is flying by.

02-15-2011, 08:56 AM	#41
jolepp Veteran Member Join Date: Oct 2010 Location: Finland Photos: Albums Posts: 3,196	Originally posted by 0rangepeel I had this problem when I used an external SD card reader... Good point, a flaky reader / cable could cause occasional corruption like this. In general the really annoying part of this sort of thing would seem to be infrequent-but-still-sometimes-happens -ness. On a related note I wonder whether Windows still retries a failed i/o operation behind the scenes which tends to result in things getting flaky going unnoticed until the trouble gets so bad that a couple of retries do not help any more.

02-15-2011, 09:35 AM	#42
pete_pf Veteran Member Join Date: Nov 2007 Location: Toronto, Canada Posts: 599 Original Poster	Originally posted by 0rangepeel I had this problem when I used an external SD card reader. It was a really cheap 5$ one, but it was blazing fast. Try transferring from your camera instead of using an SD card reader. Or alternatively, try using another sd card reader. My theory is that the cable you use for USB may not be well shielded enough, so corrupted data is flying by. ohh man wouldn't that be heaven is this was the problem? instead of memory or hard drive issues that it could be as simple as getting a new usb cable or reader?! i wonder if that's the issue though, because when i moved from camera to laptop the image is fine. but when i connected my laptop to my external hard drive, that is when the image went nuts. also when i moved the images from this post from desktop to laptop i used a card reader. i would just love is this was the problem with the most recent image that went nuts, i recently put the image into my dropbox and downloaded to my desktop, and it's working fine so far on there. so really has me thinking hmm

02-15-2011, 11:45 PM	#43
sterretje Veteran Member Join Date: Dec 2008 Location: Roodepoort, South Africa Photos: Gallery \| Albums Posts: 3,561	You have to do some trouble shooting (with sensible thinking) and draw the conclusions. You have a system now where files don't get corrupted and you're now working on images that are on the HD of your desktop (if I understand it correctly). What is the difference between the current setup (dropbox) and the previous one (external HD)? That you don't use your external HD anymore. So what can be wrong? External HD, cable, usb ports on both laptop and desktop. The sensible sequence of tests will start at the source (laptop/external HD). I do have a suspicion as to what is going wrong so I will start at the other side. Where do you connect your external HD to the desktop? Front or back? Assuming this is an external HD that gets its power from the USB ports and you connect to the front, you can eliminate the front USB as a source by using the USB at the back. 1) 1a) Copy images from your laptop to the external HD. 1b) Connect external HD to one of the USB ports on the back of the desktop. 1c) Edit a couple of them on the external HD. 1d) Does the problem occur? If not, the problem is the USB ports on the front of the desktop. And the (more than likely) reason is that those ports can’t deliver enough power to the external HD because they are wired with thin wires and when too much power is required, the voltage will drop. And I think that that is currently your problem. If yes, no conclusions can be drawn except that my suspicion was wrong and I would start at the source (laptop / external HD) 2) 2a) Copy images from your laptop to the external HD. 2b) Edit a couple of them on the external HD using the laptop. 2c) Does the problem occur? If not, I would conclude that it's the combination desktop / external HD that causes the issue; go to step 3 If yes, test the external HD while connected to the laptop; maybe jolepp's tools that you've already used can do that (not sure). Also test laptop's memory and internal HD. 3) I’m not sure how to continue. The dropbox solution proves that your desktop is basically OK (possibly except for USB). I would test the external HD while connected to the desktop using jolepp’s tools (on the front USB as well as on the back USB). Good luck. PS Don’t ask me why this problem shows up in Win7 and not in WinXP. Only God knows

02-15-2011, 11:59 PM	#44
dvest Forum Member Join Date: Dec 2010 Location: Southeast USA Photos: Gallery Posts: 91	Sounds to me like it's the external drive.

02-16-2011, 02:32 PM	#45
DeadJohn Pentaxian Join Date: Jan 2011 Location: New York Posts: 4,834	I just caught up on this thread. twitch made a very good point mentioning that the RAW file contains an embedded JPG for preview purposes ... another complexity to consider while trying to find the source/fix for this frustrating corruption problem. A question to everyone: wouldn't an intermittent problem with a memory card, cable, or hard drive likely cause additional problems? It seems like the photo corruption is always happening in a similar manner. I would expect a memory problem to sometimes corrupt files badly enough to prevent opening them. pete_pf, the 3 sample images you provided all start with a good band at the top. Have you had any photos that were 100% garbled? That started out bad at the top but then were good at the bottom? That were so bad you got an error message trying to open them?