K20D + RAW + Linux = Very Frustrated

*Venturi* · 07-23-2008, 09:08 AM

I took the plunge and picked up a K20D from Amazon and it came UPS Monday. YAY!
I'm absolutely loving all the added versatility. WOO HOO!
Downloaded first round of pics and previewing them in DigiKam. Wow them are big!

I then attempt to convert the from RAW to JPEG and things go horribly wrong. PEF shots white balance is completely mucked up, and both DNG and PEF end up with a black stripe on the "right hand side" of the resultant JPEG. BOO!

I have tried LightZone, UFRaw, RawTherapee, RawStudio and F-Stop. All produce the same results. Checked to make sure I had the latest versions of all the above as well. However ... I did conversions of both DNG and PEF from the K20D using purely CLI tools: dcraw & NetPbm (pnmtojpeg) and the JPEG outputted is fullly intact. I even used the version of dcraw (8.77) packaged with LightZone 3.5 in the CLI and got good results.

So... I don't believe it is an issue with dcraw but I have zero clue what is the next place to look. I googled around a bit and haven't come up with any fixes that actually work. The only K20D fix I have seen thus far is for firmware 1.01 to change the "Make" exif data to "PENTAX Corporation" rather than merely "PENTAX" - but I am still on v1.00 of the firmware so that isn't even an issue. K10D DNG/PEF files convert just fine and dandy and I've got about 5K images to prove that.

Any gurus (or just lucky individuals) that have found a solution?

FYI I'm using openSuSE 10.3 32-bit on a 2yo AMD 3800 machine w/ 4GB of RAM.

Gooshin · 07-23-2008, 09:14 AM

why are you on Linux in the first place

also can you please clarifiy if you think this is a camera fault or just the OS missing a few files?

have you tried this on someone else's computer with windows/mac running ACR or similar?

davemdsn · 07-23-2008, 09:24 AM

I am not a Linux guy, nor do I use any of the apps you mentioned, but the first thing that comes to mind is to ask if those apps fully support the K20. The WB issue you mention sounds consistent with how apps like Lightroom and Aperture displayed the K20 files until full support was added. Pentax stores WB info differently from the other companies so without specific camera support the colors come out way off.

RBellavance · 07-23-2008, 09:35 AM

I don't know about the other apps, but for UFRaw you must download & compile the very latest version from CVS for it to support the K20D correctly.

Instructions are on UFRaw's site: SourceForge.net: CVS

*troyz* · 07-23-2008, 12:56 PM

Originally Posted by Venturi

I took the plunge and picked up a K20D from Amazon and it came UPS Monday. YAY!
I'm absolutely loving all the added versatility. WOO HOO!
Downloaded first round of pics and previewing them in DigiKam. Wow them are big!

I then attempt to convert the from RAW to JPEG and things go horribly wrong. PEF shots white balance is completely mucked up, and both DNG and PEF end up with a black stripe on the "right hand side" of the resultant JPEG. BOO!

I have tried LightZone, UFRaw, RawTherapee, RawStudio and F-Stop. All produce the same results. Checked to make sure I had the latest versions of all the above as well. However ... I did conversions of both DNG and PEF from the K20D using purely CLI tools: dcraw & NetPbm (pnmtojpeg) and the JPEG outputted is fullly intact. I even used the version of dcraw (8.77) packaged with LightZone 3.5 in the CLI and got good results.

So... I don't believe it is an issue with dcraw but I have zero clue what is the next place to look. I googled around a bit and haven't come up with any fixes that actually work. The only K20D fix I have seen thus far is for firmware 1.01 to change the "Make" exif data to "PENTAX Corporation" rather than merely "PENTAX" - but I am still on v1.00 of the firmware so that isn't even an issue. K10D DNG/PEF files convert just fine and dandy and I've got about 5K images to prove that.
Any gurus (or just lucky individuals) that have found a solution?

FYI I'm using openSuSE 10.3 32-bit on a 2yo AMD 3800 machine w/ 4GB of RAM.

You can try using dcraw version 8.86 or higher to generate an 8-bit TIFF file directly;

dcraw -v -T -w rawfile

davek · 07-23-2008, 01:40 PM

No idea if this helps, but Bibble has a converter which runs on Unix... (i'm using XP).

*Venturi* · 07-23-2008, 05:09 PM

Originally Posted by RBellavance

I don't know about the other apps, but for UFRaw you must download & compile the very latest version from CVS for it to support the K20D correctly.

Instructions are on UFRaw's site: SourceForge.net: CVS

Thanks, I had seen mention of this (perhaps here by you in an earlier thread?) but the instructions seem incomplete. I'm fine with compiling but doing CVS pulls not so much.

cvs -z3 -d:pserver:anonymous@ufraw.cvs.sourceforge.net:/cvsroot/ufraw co -P modulename

what modulename am I supposed to use? I tried ufraw and that obviously did not work.

Thanks again!

*Venturi* · 07-23-2008, 05:14 PM

Originally Posted by troyz

You can try using dcraw version 8.86 or higher to generate an 8-bit TIFF file directly;

dcraw -v -T -w rawfile

Yep, I can convert the DNG/PEF files just fine using the dcraw and netpbm CLI as I mentioned above to either TIFF or JPEG, which is why I'm a little puzzled on why the other tools don't work as a good many of them use dcraw and all the versions of dcraw from 8.77 and up I have tried work just fine.

If I can get the ufraw CVS version downloaded and compiled that may clear up things for DigiKam, f-stop and rawtherapee as I believe they all use the ufraw backend. LightZone is my tool of choice but I'm not going to hold my breath on them getting out a patch any time this century *grrrr*.

Thanks to all that have posted replies. If I find a magic bullet for this I'll be sure to pass it along.

RBellavance · 07-24-2008, 04:35 AM

Originally Posted by Venturi

Thanks, I had seen mention of this (perhaps here by you in an earlier thread?) but the instructions seem incomplete. I'm fine with compiling but doing CVS pulls not so much.

cvs -z3 -d:pserver:anonymous@ufraw.cvs.sourceforge.net:/cvsroot/ufraw co -P modulename

what modulename am I supposed to use? I tried ufraw and that obviously did not work.

Thanks again!

This is what I use:

cvs -d:pserver:anonymous@ufraw.cvs.sourceforge.net:/cvsroot/ufraw login
followed by:
cvs -z3 -d:pserver:anonymous@ufraw.cvs.sourceforge.net:/cvsroot/ufraw co -P ufraw

I just tested this and it worked. Remember to run ./autogen.sh in the pulled directory before the usual "./configure".

*sajmmiller* · 07-24-2008, 06:48 AM

Originally Posted by Venturi

If I can get the ufraw CVS version downloaded and compiled that may clear up things for DigiKam, f-stop and rawtherapee as I believe they all use the ufraw backend.

I believe the apps you mention, except RawTherapee use code from dcraw, not ufraw, so getting ufraw working will have no effect on the others. As mentioned by RBellavance, use the CVS version of UFRaw until the next release of UFRaw. As for Digikam, 0.94 final lists the K20D as a supported camera, so you might see if your Linux distribution has that version packaged before going to the trouble of compiling it manually. It became available on my Gentoo system yesterday.

I don't know about f-spot, but would guess that it uses code from dcraw also...probably older dcraw code compiled in statically, not dynamically, so the version of dcraw you have installed on your system won't affect it. I don't believe RawTherapee uses dcraw at all, so you will need to go to the RawTherapee forums for your answer. The author of RawTherapee is generally pretty responsive on the forums.

Hope that helps.

*Venturi* · 07-24-2008, 07:46 AM

I must have been holding my mouth wrong, or the universe just hates me... Tried again today and low and behold it worked perfectly. Thank you again.

So one down and four to go. I'm not a huge fan of the ufraw UI nor that its default settings "underexpose" every pic but at least it works.

Entropy · 07-24-2008, 08:13 AM

Originally Posted by sajmmiller

I believe the apps you mention, except RawTherapee use code from dcraw, not ufraw, so getting ufraw working will have no effect on the others. As mentioned by RBellavance, use the CVS version of UFRaw until the next release of UFRaw. As for Digikam, 0.94 final lists the K20D as a supported camera, so you might see if your Linux distribution has that version packaged before going to the trouble of compiling it manually. It became available on my Gentoo system yesterday.

I don't know about f-spot, but would guess that it uses code from dcraw also...probably older dcraw code compiled in statically, not dynamically, so the version of dcraw you have installed on your system won't affect it. I don't believe RawTherapee uses dcraw at all, so you will need to go to the RawTherapee forums for your answer. The author of RawTherapee is generally pretty responsive on the forums.

Hope that helps.

I can confirm that the digikam 0.94 series works with the K20D - I use it for culling out really bad shots before going through stuff with ufraw.

*Venturi* · 07-24-2008, 08:53 AM

Originally Posted by sajmmiller

I believe the apps you mention, except RawTherapee use code from dcraw, not ufraw, so getting ufraw working will have no effect on the others. As mentioned by RBellavance, use the CVS version of UFRaw until the next release of UFRaw.

UFRaw working now with the 0.14 CVS version. Ugly but functional.

As for Digikam, 0.94 final lists the K20D as a supported camera, so you might see if your Linux distribution has that version packaged before going to the trouble of compiling it manually. It became available on my Gentoo system yesterday.

Just scanned the changelog for digikam and did not see mention of the K20D anywhere but I may do a local compile of 0.94 just to see how it goes. SuSE is still at v0.92 for their repositories and Packman is currently at v0.93.

I don't know about f-spot, but would guess that it uses code from dcraw also...probably older dcraw code compiled in statically, not dynamically, so the version of dcraw you have installed on your system won't affect it.

I had read 'somewhere' that f-spot and/or showfoto basically were frontends for ufraw; but that very well could have been heresay. It would make more sense that they have compiled dcraw (old version) into their executable, as you said. LightZone packages a standalone dcraw executable that is v8.79 and it (dcraw 8.79) does read the K20D raw images just fine but something else they are using in their code mucks things up the same as the other apps.

I don't believe RawTherapee uses dcraw at all, so you will need to go to the RawTherapee forums for your answer. The author of RawTherapee is generally pretty responsive on the forums.

Hope that helps.

After I attempt to get 0.94 of digikam going if that doesn't work out I am probably going to either batch convert raw->tiff with dcraw, or shoot jpeg for a while, and do whatever PP necessary in LZ - and patiently await the vendors to catch up. I've got enough stress in my life already and don't need to compoud it by fighting software.

Thanks!

*Venturi* · 07-24-2008, 08:55 AM

Originally Posted by Entropy

I can confirm that the digikam 0.94 series works with the K20D - I use it for culling out really bad shots before going through stuff with ufraw.

Cool beans.

I'm off to download and compile - wheee!

*sajmmiller* · 07-24-2008, 09:11 AM

Originally Posted by Venturi

So one down and four to go. I'm not a huge fan of the ufraw UI nor that its default settings "underexpose" every pic but at least it works.

I really like UFRaw and use it for all my raw conversions. It does take a bit of setup to get it working well though. Once setup, I find the user interface to be fast and precise.

The first thing you need to do is set up a base curve to compensate for the linear raw file. Others apps seem to have defaults for that, UFRaw does not. PM me and I'll send you my base curve for the K10D. It should be a good starting point; you can adjust it as necessary for the K20D. Shoot some Raw+Jpeg and compare the camera jpeg to the raw file opened in ufraw. Use the camera jpeg as a reference to help you adjust the base curve in ufraw.

The second thing you need is a Pentax raw camera profile. I use the Pentax3 profile from the free version of Bibble. I don't know if that will be the best profile for the K20D....you'll have to experiment. Bibble also has a Pentax1 profile. Anyway select your chosen profile as the input (camera) profile in ufraw on the color management tab.

Once you have things setup, processing becomes very easy in UFRaw. I rarely need to adjust the base curve. I do use the luminosity curve to do the usual stuff like setting white point, black point, adjusting contrast, shadows, highlights, etc. 1.10 seems to be a good default setting for saturation, at least for the Pentax3 profile on the K10D.

Hopefully, that gives you enough information to get started. UFRaw really is a great program, but it requires some effort to get it set up. Like you, I was frustrated with UFRaw at first, but now I much prefer it over the other raw converters. I love that I can store my original raw file plus a 2K ufraw-id file and open up the raw file at a later date using the id file with all my raw post-processed settings intact. In other words, by having ufraw store a ufraw-id file for each raw file I process, I don't have to store a huge 16-bit tiff or png file to retain my post-processed raw file. And once you have the UFRaw defaults setup properly, ufraw-batch will work great if you don't have the time to process each raw file individually.

Hope that helps. Anyone who wants my UFRaw K10D base curve can PM me and I'll send it to you. Or, use the Raw+jpeg method outlined above and create your own base curve.