[uffe]

Hi,

I am trying to learn batchconversion on kubuntu and try to automate everything from copying the files from the memorycards, creating jpg:s in several resolutions from jpeg and making backup.

I have solved everything except creating the jpeg files with good quality, I have the camera set on jpeg + dng to have out of camera jpeg as reference. I have problem to sharpen the photos with imagemagick in a good way, jpegsfrom camera have much better details.

Does anyone use a setup as this with good results I would be happy to see what commands you use with what input you use, currently I have just copied stuff from different scripts but do not manage to get good output.

Below is the current jpeg conversion, I have tried to minimize the jpeg conversion also but got worse result so even if I am not sure how the exposure compensation work I kept it. The curve is from a k20D-file I found on this forum in an older thread. Did not find any good for k10D (only a really bad one).

I tried to use GIMP and digikam to get good start values but found that I did not get the same results in ufraw-batch so I guess some other default parameters of values might trick me.

Any input is really appreciated for other pentax cameras as well but please note which camera type you use.

Best regards
Uffe


for i in *.DNG; do

#Get ISO
ufraw-batch --exposure=auto --clip=film --black-point=auto --wb=camera --create-id=only $i 2> /dev/null
ISO=`cat ${i%\.*}.ufraw | grep "<ISOSpeed>" | sed 's/<[^<>]*>//g'`

#Set ratio between ISO and noise reduction
let "NOISE=ISO/7"


#Get exposure information
OUTPUT=`cat ${i%\.*}.ufraw | grep "<Exposure>" | sed 's/<[^<>]*>//g'`

#Adjust automatic exposure compensation
EXPOSURE=`echo $(echo "scale=6; (-0.025*$OUTPUT^3)+(0.6*$OUTPUT)+0.1" | bc) | sed 's/^\./0,/g' |sed 's/\./,/g' | sed 's/-\,/-0,/g'`

#Remove temp file
rm ${i%\.*}.ufraw


#Behandla för utskrift original
echo "Skapar filen $HOME/Temp/BildUt/Print/${i%\.*}_print.jpg"

ufraw-batch --wb=camera --gamma=0.35 --exposure=$EXPOSURE --black-point=auto --clip=film --wavelet-denoising-threshold=$NOISE --base-curve-file="$HOME/Temp/Pentax_K20D_natural_curve.txt" --saturation=1.0 $i --output=- | convert - -sigmoidal-contrast 1.4,20 -unsharp 10x10+0.7+0.06 -quality 95 $HOME/Temp/BildUt/Print/${i%\.*}_print.jpg

exiftool -TagsFromFile "$i" -overwrite_original -all:all --orientation -copyright='Uffe' "$HOME/Temp/BildUt/Print/${i%\.*}_print.jpg"

done

[nanok]

hi uffe. welcome to the club . i came across this post while searching for my own settings (hoped i posted them somewhere here, some time.. ) -- mentioning just for anecdotical value

here we go, see my comments online

Hi,

I am trying to learn batchconversion on kubuntu and try to automate everything from copying the files from the memorycards, creating jpg:s in several resolutions from jpeg and making backup.

I have solved everything except creating the jpeg files with good quality, I have the camera set on jpeg + dng to have out of camera jpeg as reference. I have problem to sharpen the photos with imagemagick in a good way, jpegsfrom camera have much better details.

Does anyone use a setup as this with good results I would be happy to see what commands you use with what input you use, currently I have just copied stuff from different scripts but do not manage to get good output.

Below is the current jpeg conversion, I have tried to minimize the jpeg conversion also but got worse result so even if I am not sure how the exposure compensation work I kept it. The curve is from a k20D-file I found on this forum in an older thread. Did not find any good for k10D (only a really bad one).

I tried to use GIMP and digikam to get good start values but found that I did not get the same results in ufraw-batch so I guess some other default parameters of values might trick me.

play with ufraw (gui) more, gimp is quite different, they are very different beasts, handling very different jobs with very different input .

Any input is really appreciated for other pentax cameras as well but please note which camera type you use.

Best regards
Uffe


for i in *.DNG; do

#Get ISO
ufraw-batch --exposure=auto --clip=film --black-point=auto --wb=camera --create-id=only $i 2> /dev/null
ISO=`cat ${i%\.*}.ufraw | grep "<ISOSpeed>" | sed 's/<[^<>]*>//g'`

this is commonly called "useless use of cat" . nothing wrong with it as far as image wuality is concerned, though . "grep ISOSpeed ${i%\.*}.ufraw" should do the same job. however, the real wtf up here seems to me running ufraw to create an id file to get the iso setting. i'd just use exiftool instead.

exiftool pics/aug30/104_3008/k20a5554.pef | awk '/^ISO /{print $3}'
400

(not the most enlightened regexp, but it's an example)

#Set ratio between ISO and noise reduction
let "NOISE=ISO/7"

interesting, never thought of devising such a formula. does that work well? (i'll have to try it )

#Get exposure information
OUTPUT=`cat ${i%\.*}.ufraw | grep "<Exposure>" | sed 's/<[^<>]*>//g'`

#Adjust automatic exposure compensation
EXPOSURE=`echo $(echo "scale=6; (-0.025*$OUTPUT^3)+(0.6*$OUTPUT)+0.1" | bc) | sed 's/^\./0,/g' |sed 's/\./,/g' | sed 's/-\,/-0,/g'`

#Remove temp file
rm ${i%\.*}.ufraw


#Behandla för utskrift original
echo "Skapar filen $HOME/Temp/BildUt/Print/${i%\.*}_print.jpg"

ufraw-batch --wb=camera --gamma=0.35 --exposure=$EXPOSURE --black-point=auto --clip=film --wavelet-denoising-threshold=$NOISE --base-curve-file="$HOME/Temp/Pentax_K20D_natural_curve.txt" --saturation=1.0 $i --output=-

if you have so many options you want to set, i'd use a more elegant way (see man ufraw, something like id file but more generic, can't remember, i never used it, as i mostly let ufraw do it's job, as it does it well in most cases)

| convert - -sigmoidal-contrast 1.4,20 -unsharp 10x10+0.7+0.06 -quality 95 $HOME/Temp/BildUt/Print/${i%\.*}_print.jpg

sigmoidal-contrast with 1.4,20 (isnt that "1.4,20\%"?) sounds strange to me, it means you are setting the mid grey to 20% (which is quite far from 50%).

the usm settings are very unballanced to my eye. i can't remember mine though . i think it was radius of 3 (first number, yours is 10, must take a long time to do, btw ), 3x0.5+1.5+0.02 (1.5 is the amount, that's usually what i change, usually to bring it down). i reached these settings through a lot of trial and error, and after finding an excellent explanation on how it works for imagemagick (different from gimp and most others, notice they tend to have 3params, you have 4 here). a very interesting read, if you are inclined:

Sharpening using Image Magick Red Skies at Night

(not sure this is exactly what i read back then, but it looks good, and i can't find the other one )

exiftool -TagsFromFile "$i" -overwrite_original -all:all --orientation -copyright='Uffe' "$HOME/Temp/BildUt/Print/${i%\.*}_print.jpg"

done

hope this helps. feel free to ask if there's anything more we can help with.

ps: oh, btw, i'd be interested in that curve file (especially as i have a k20d)

ps2: to get an idea of what i get:
http://nanok.is-a-geek.com/gallery2/main.php?g2_itemId=4741

almost all pictures on my gallery letely are processed that way (most importantly, with those sharpening settings)

[nanok]

a few other quick notes: i only do mass conversion for previews, if i need to tweak a picture i really like, i will do it in ufraw directly (that's why i bother shooting raw, right?). i don't even do that anymore, a little program called "geeqie" (fork of gqview, which was my viewer of choice) freed me from that wasted time (and cpu time) and hdd space: this little viewer knows how to extract the embedded jpeg in the raw file (low quailty, yes, but perfect for a preview, and extremely fast too). make sure you really do want to go through all this trouble, before you find out you actually want to convert and tweak each file that matters manually anyway, and you don't really care that much for most others (and can let ufraw on "auto", as it will rarely do a bad job)

for transferring pcitures around (from card or to backup) i use rsync. i didn't automate that though (but i might soon, especially the backup part )

another thing worth noting, if you do the "mass conversion" thing: ufraw from the debian package is for some reason sllow, it seems (i just reported the bug), if i compile the same version from source, it's almsot twice as fast. it might matter to you or not (dependign how fast your machine, how many pcitures you process in one go, and so on). another thing, which still buggs me, is that ufraw doesn't know multi-cpus; what this means is that, effectively, if you have a dual core cpu, and an smp-enabled kernel, running two ufraw-batch processes, converting different pictures, at the same time, in parallel, will not slow them down one bit (so you are getting twice the speed, or almost).

that's about it for now (will be back with more details on my workflow tomorrow, if you're curious). have fun

[uffe]

Thanks for really great input, I will do some updates of my script. I know there are really a lot of settings in my script, I copied in the long version to show a bit what I have been playing with.

rsync is exactly what I will use for backup as well.

It would be very interesting to hear more about your workflow, and batch-setting. Just started with Raw and use it to learn scripting in linux and raw conversion at the same time.

The attached files are the files I found for k20D, found them on this forum in some older thread I think.

Attached Files

	Pentax K20D bright.curve.txt (326 Bytes, 498 views)
	Pentax K20D natural.curve.txt (287 Bytes, 505 views)
	Pentax K20D vibrant.curve.txt (367 Bytes, 495 views)

[nanok]

i suggest for testing that you break the whole process into pieces, see how each part works, and tweak what you like/don't like (for instance take the sharpening aside, play with a few of the pictures which are typical for your shooting style, and tweak untill you are happy, and so on)

glad to be of help, thanks for the curves, i'll give them a spin soon

see you around

[Quicksand]

Very interesting. My script for K20D raw processing (posted about a week ago) is over here.

My script is a little more complicated, in that it does some different calculations for ISO-dependent noise reduction and sharpening, but other than that the two scripts are really more similar than they are different.

Like you, I adjust sigmoidal contrast and sharpening in imagemagick. I also do local contrast enhancement and boost saturation a bit. Even in comparable operations, the parameters are different, but -- honestly -- I'm not all that satisfied with my parameters yet either.

It's a work in progress.

Here's what I do, by default:

-modulate 100,105,100 # the 105 is saturation
-sigmoidal-contrast 5.0x35%
-unsharp 0x40.0+0.2+0.0 # local contrast boost
-unsharp 0x1.2x___x0.03 # 0.00 to 0.75 sharpening, depending on ISO

And you:

-sigmoidal-contrast 1.4,20
-unsharp 10x10+0.7+0.06

I haven't had much time to play with the parameters recently, but I try to do so from time to time.

EDIT: Here are some examples of output from my K20D script (cropped, but otherwise exactly as turned out by the script).

[jbinpg]

I think one reason why I gave up on batch processing using dcraw is that I do a lot of cropping. And if you do a lot of cropping, you have to cut down on the USM - the more you crop, the smaller the radius should be. It is hard to know how much you will be cropping beforehand which is why it is hard to deal with the USM amount in a batch file.

Jack

[nanok]

usm should be one of the last things you apply. i apply it only at the point of resizing (i don't store my processed pictures full size, sharpened). usm depends on the final size, and really needs to be aplied when resizing (that's why i use batch, and have a resizing script for web-output )

[uffe]

Thanks for all your input, I do get much better results now, still not completely happy and primary tested with the latest pictures which are all quite similar but will tweak some more the coming weeks.
Specially exposure I need to change improve and NR I just made sure I get decent result.

I failed to used the exiftool and AWK to get ISO but I think the new solution below looks quite good.

But it is much better then before, so thanks very much all of you who have answered.


for i in *.DNG; do

ISO=`exiftool -s -s -s -ISO $i`

let "NOISE=ISO/4"

#Behandla för utskrift original
echo "Create file $HOME/Temp/BildUt/Print/${i%\.*}_print.jpg"

ufraw-batch --exposure=0.6 --saturation=1.05 --wavelet-denoising-threshold=$NOISE $i --output=- | convert - -unsharp 3x0.5+1.5+0.02 -quality 95 $HOME/Temp/BildUt/Print/${i%\.*}_print.jpg

exiftool -TagsFromFile "$i" -overwrite_original -all:all --orientation -copyright='Uffe' "$HOME/Temp/BildUt/Print/${i%\.*}_print.jpg"

done

[nanok]

Thanks for all your input, I do get much better results now, still not completely happy and primary tested with the latest pictures which are all quite similar but will tweak some more the coming weeks.
Specially exposure I need to change improve and NR I just made sure I get decent result.

i would just leave out the exposure compensation stuff for now. imho, unless the pictures are badly exposed, you should let ufraw deal with them in the default way (or at least treat them all the same), if they are badly exposed you will either discard those or, if you really want to, open the raw file in ufraw, and tweak it (if the picture matters so much you want to save it).

I failed to used the exiftool and AWK to get ISO but I think the new solution below looks quite good.

But it is much better then before, so thanks very much all of you who have answered.


for i in *.DNG; do

ISO=`exiftool -s -s -s -ISO $i`

actually, this is much better than my example. i overlooked the fact that exiftool is designed to do that kind of stuff itself (i am just so used to piping stuff around and parsing it, professional hazard i guess ). well done!

let "NOISE=ISO/4"

#Behandla för utskrift original
echo "Create file $HOME/Temp/BildUt/Print/${i%\.*}_print.jpg"

ufraw-batch --exposure=0.6 --saturation=1.05 --wavelet-denoising-threshold=$NOISE $i --output=- | convert - -unsharp 3x0.5+1.5+0.02 -quality 95 $HOME/Temp/BildUt/Print/${i%\.*}_print.jpg

exiftool -TagsFromFile "$i" -overwrite_original -all:all --orientation -copyright='Uffe' "$HOME/Temp/BildUt/Print/${i%\.*}_print.jpg"

done

glad to be of help, do post back here when you make progress, we might find your finds useful.

i played with the curves a little bit, the results are indeed closer to the camera jpegs (but not identical), they are lighter and less contrasty than the default ufraw output, i am not sure i like it better that way, though (but definetly a good start)