Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
08-24-2009, 10:01 PM - 1 Like   #1
Pentaxian




Join Date: Apr 2008
Location: Central Ohio (formerly SF Bay Area)
Posts: 1,519
Linux: batch raw conversion script w/ ISO-dependent processing

A few weeks ago I threatened to post this -- it's about time I made good on it!

I shoot in raw to preserve my post-processing options (in particular, it's nice to be able to correct exposure and white balance). But in general, I don't want to spend a lot of time post-processing my crummy pictures when only some relatively small percentage of them are keepers.

In using some other batch-enabled raw conversion tools for Linux (e.g. UFRaw, RawTherapee), I found that I couldn't settle on a single conversion profile to apply to all of my shots. Some of them needed more noise reduction, and some of them needed sharpening. But excess noise reduction and sharpening can really mess things up. What I found is that I can usually match my desired level of noise reduction and sharpening to the ISO value of the photo. But as far as I know, the Linux-compatible raw converters don't really allow for that, at least not easily. Maybe the fancy-pants commercial conversion programs like Lightroom (for Windows/Mac) and Bibble can do this, or maybe not. I haven't tried.

So that's why I wrote this script. It looks at the ISO value for each photo, then applies the appropriate level of noise reduction and sharpening. I can run it once in a directory full of images and walk away.

This script is intended to do "good enough" RAW to JPG conversion in the majority of cases. It's more than adequate as a tool to figure out which ones to subsequently tune by hand, and I think it gets the job done well enough that most of the resulting photos are usable as-is (on the web, at least).

Here's how it works:

First, for each PEF or DNG, it uses dcraw to convert the raw file to 16-bit-per-channel TIF. Noise reduction is applied at this stage.

Then it uses imagemagick to boost saturation a bit, apply a sigmoidal contrast enhancement curve (to emphasize the midtones) and a relatively conservative amount of local contrast enhancement (that's unsharp mask with a large radius). These adjustments are all applied to the 16-bit TIF.

Then it sharpens the image (via another unsharp mask pass, with a small radius) and converts it to 8-bit JPG.

Finally, the script copies all of the EXIF data from the original raw file to the new JPG.

It does ISO-dependent noise reduction ("NR") using dcraw's wavelet denoise function, and ISO-dependent sharpening using imagemagick's unsharp mask. In general, more noise reduction and less sharpening is desired at higher ISO values, and this script automates that.

You can choose your own NR and sharpening values at ISO values of 100, 200, 400, etc. up through 6400, and for actual ISO values in between those power-of-two selections, this script will interpolate between the adjacent NR and sharpening values. It's just linear interpolation, nothing fancy.

I find that this script works pretty well with photos from my Pentax K20D (I have dynamic range enhancement turned OFF). For other cameras and other camera settings, adjustments may be necessary -- particularly to the ISO-dependent noise reduction and sharpening profiles, but also to the (fixed) local contrast enhancement value, the saturation boost, and the gamma. All of these variables are set at the top of the script, so hack away!

Requirements:
dcraw (raw converter)
imagemagick (command-line image processing Swiss Army Knife)
exiv2 (exif data manipulation tool)
bash, of course, plus the customary GNU command-line utilities

I'm sure there are a million ways to improve this script, but for me, it has reached a point of diminishing returns where I don't see much benefit to adding more. I'll hand-process those cases. But if you want to add something, please feel free -- and post your improvements here!

I'll start with some ideas:
- Better noise reduction (e.g. more emphasis on reducing chroma-channel noise) using something like G'MIC
- Better handling of gamma and color profiles (I know next to nothing about this subject)
- Better error/exception handling
- Processing command-line options to override defaults
- Automatic exposure compensation when needed
- Windows and Mac versions
- etc. etc. etc.

Usage: run without any command-line options, and it will process all of the PEF and DNG files in the current directory. Alternatively, you can specify the files you want processed by giving their filenames. That's it!

Here ya go:

Code:
#!/bin/bash
#
# convertraw
# Convert from PEF/DNG to JPG
#
# Copyright 2009 Clarke A. Wixon
#
# This work is licensed under the Creative Commons
# Attribution-Noncommercial-Share Alike 3.0 Unported License.
# To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/
# or send a letter to Creative Commons, 171 Second Street, Suite 300,
# San Francisco, California, 94105, USA.
#

comment="Autoconversion script rev. 2009-06-26"
gamma="2.2"
contrast="5.0x35%"
lc="0x40.0+0.2+0.0"
saturation=105
jpgqual=85
isolevel=( 0 100 200 400 800 1600 3200 6400 )
nrlevel=( 25 25 50 100 200 400 800 1600 ) # change this to alter dcraw noise-reduction levels
sharplevel=( 0.7500 0.7500 0.5000 0.2500 0.0000 0.0000 0.0000 0.0000 ) # change this to alter imagemagick sharpening amounts

shopt -s nullglob # if there are no matching files, don't try to process the glob as an actual filename

if [ -z "$1" ]
then
infiles="*.PEF *.DNG"
else
infiles="$@"
fi

echo "Converting $infiles . . ."

for rawname in $infiles
do

# set filenames

if [[ "$rawname" =~ ".PEF" ]]
then
basename="${rawname%.PEF}"
else
basename="${rawname%.DNG}"
fi
tifname="${basename}.tif"
pefexif="${basename}.exv"
jpegname="${basename}-batch.jpg"
jpgexif="${jpegname%.jpg}.exv"

if [ -f $rawname ]
then

isoval=$( exiv2 print "$rawname" | grep "ISO speed" | awk '{print $4}' )

if (( $isoval >= 6400 ))
then
denoise=${nrlevel[7]} # max value
unsharp=${sharplevel[7]} # max value
else
denoise=${nrlevel[0]} # minimum default value; shouldn't be needed
unsharp=${sharplevel[7]} # minimum default value; shouldn't be needed
for index in 0 1 2 3 4 5 6
do
if (( ( $isoval >= ${isolevel[$index]} ) && ( $isoval < ${isolevel[$index + 1]} ) ))
then
interval=$(( ${isolevel[$index + 1]} - ${isolevel[$index]} ))
denoise=$(echo "scale=2; ${nrlevel[$index]} + ( ( ${nrlevel[$index + 1]} - ${nrlevel[$index]} ) * ( ( $isoval - ${isolevel[$index]} ) / $interval ) )" | bc | cut -d. -f1)
unsharp=$(echo "scale=4; ${sharplevel[$index]} + ( ( ${sharplevel[$index + 1]} - ${sharplevel[$index]} ) * ( ( $isoval - ${isolevel[$index]} ) / $interval ) )" | bc)
fi
done
fi

echo
echo "Converting $rawname from PEF to JPG (ISO "$isoval": NR "$denoise" sharpness "$unsharp")..."
dcraw -w -q 3 -n $denoise -4 -T -c -t 0 -v -H 0 -S 4095 "$rawname" > "$tifname"

echo "Processing..."
convert "$tifname" -verbose -depth 16 -gamma $gamma -modulate 100,"$saturation",100 -sigmoidal-contrast $contrast -unsharp $lc -unsharp 0x1.2x"$unsharp"x0.03 -depth 8 -strip -quality $jpgqual "$jpegname"

echo "Transferring EXIF and cleaning up..."
exiv2 extract -f "$rawname"
mv "$pefexif" "$jpgexif"
exiv2 insert -f -M"set Exif.Photo.UserComment charset=Ascii $comment (NR=$denoise gamma=$gamma sat=$saturation contrast=$contrast lc=$lc unsharp=$unsharp qual=$jpgqual)" "$jpegname"
rm "$jpgexif"
rm "$tifname"

echo "Done."
fi
done


08-25-2009, 12:39 AM   #2
Site Supporter
Site Supporter




Join Date: Mar 2008
Location: Prince George, BC
Photos: Gallery | Albums
Posts: 3,543
Nice script. However, I do something similar in RawTherapee. I have a whole mess of profiles set up based on ISO. Then I select all those with the same ISO and send to the processing queue. Rinse and repeat for the next ISO if there are more than one. Works fine. I can get away with this because I do minimal sharpening.
08-25-2009, 06:12 AM   #3
Veteran Member




Join Date: Nov 2006
Location: Near Montréal, Canada
Photos: Gallery
Posts: 1,716
Nice, thanks for posting this !
08-25-2009, 12:22 PM   #4
Inactive Account




Join Date: Jul 2009
Posts: 115
I have to test this.

08-25-2009, 01:32 PM   #5
Veteran Member
DanLoc78's Avatar

Join Date: Apr 2008
Location: Philadelphia
Photos: Gallery
Posts: 915
is there a way to do batch RAW conversions using PS Elements?
08-09-2010, 04:34 PM   #6
Forum Member
Tim_R's Avatar

Join Date: Jan 2010
Location: Denver, CO, USA
Photos: Gallery | Albums
Posts: 53
Very nice script. Useful, as I like small jpegs for review before I tackle PP on the "keepers." However, I ran into a slight snag:

QuoteQuote:
Converting IMGP0019.DNG from PEF to JPG (ISO 200: NR 50 sharpness .5000)...
Loading PENTAX K20D image from IMGP0019.DNG ...
Wavelet denoising...
Scaling with darkness 0, saturation 4095, and
multipliers 1.617188 1.000000 1.421875 1.000000
AHD interpolation...
Converting to sRGB colorspace...
Writing data to standard output ...
Processing...
IMGP0019.tif=>IMGP0019-batch.jpg TIFF 4688x3124 4688x3124+0+0 8-bit DirectClass 1.901MB 16.590u 0:09.989
Transferring EXIF and cleaning up...
./IMGP0019.exv: Could not write metadata to file: Size of Exif JPEG segment is larger than 65535 bytes
Done.
Anyone else run into this? And is there a fix?

Tim
08-10-2010, 11:49 AM   #7
Site Supporter
Site Supporter




Join Date: Mar 2008
Location: Prince George, BC
Photos: Gallery | Albums
Posts: 3,543
2nd thoughts on this one: if all you are doing is a first run through to get rid of the tossers, then I can't see why you just don't use a program that loads up the jpeg embedded in every PEF to do it. Geeqie is what I use and it is very fast.

08-11-2010, 07:05 PM   #8
Pentaxian




Join Date: Apr 2007
Location: Oklahoma USA
Posts: 2,192
Thanks very much for that script. It might be helpful to add a comment regarding the versions of the various utilities that you've tested this with, for example if someone gets an error from imagemagick, they can compare their version of that to what's known to work.

Paul
07-14-2016, 11:53 AM   #9
Banned




Join Date: Jul 2015
Posts: 3
Thank you so much for sharing that script, Clarke! I am using Shotwell on Linux for managing single PEF files, but sometimes I just want to batch convert a couple of files to JPEG, your script is perfect for that. When running this, it always reported errors about the operators in lines 57 and 66 for me, though. I removed the $ in front of $isoval to make those errors disappear, as has been suggested to me when I ran your script through shellcheck.net - maybe you should run it through there, too, to see if there are any compatibility problems. That site reports 5 more troublesome lines, but your script works for me as is now.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
conversion, echo, fi, iso, jpg, noise, photography, photoshop, reduction, script, value, values
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
For Sale - Sold: DxO Batch/single RAW conversion Software for D-7 Ver6 (CONUS) Djedi Sold Items 1 10-15-2010 05:39 PM
Processing RAW files in Linux (Ubuntu) krishna Troubleshooting and Beginner Help 9 04-23-2010 10:58 PM
K-7 Raw processing on linux; dull colors K7er Digital Processing, Software, and Printing 10 03-08-2010 07:45 PM
Linux -- Ubuntu Karmic 9.10, libraw 0.8.5, Raw file processing K7er Digital Processing, Software, and Printing 4 01-13-2010 08:35 PM
RAW Processing in Linux soprano Digital Processing, Software, and Printing 21 07-20-2009 03:43 PM



All times are GMT -7. The time now is 06:55 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top