Linux Workflow (Ubuntu)

disasm · 07-06-2009, 11:08 AM

Hey all you pentax owners,

I'm a linux users and saw some other threads with linux users, so I thought I'd share my workflow, and hope to get some better suggestions from everyone else does for a workflow. I'm completely new to SLR's but spent my honeymoon in Germany learning tricks of the trade. I shoot in manual, PEF raw, and in the last month I've had the camera, I've shot almost 50 gigs of pictures.

I have a directory ~/photos I keep all my photos in to separate them from other Pictures and what not.
I rsync my camera card to ~/photos, and then clear the card. When prices come down a bit more I might consider buying new cards for every shoot, but for now I delete them.
Then I create subdirectories for the event and move the entire day directories into that subdirectory i.e. ~/photos/honeymoon/101_0605
and then look at 101_0605, and decide what to rename that folder, i.e. heidelburg
Then any pictures I want to "develop" I open in ufraw, and save to ~/photos/developed/category/etc...

Now I'm obviously missing some steps, of most importance metadata. Is there anyway to edit metadata for the pef files, or should I add metadata to the finished jpegs, or would you suggest I take my photos in DNG format?

I'm open to any suggestions, and am curious what all of you are doing for your workflow.

Sam

**jbinpg** · 07-06-2009, 03:45 PM

You should get lots of feedback on this. Ubuntu 9.04 user here. My workflow:

- take card out of camera, stick it in PC, transfer to Photos folder.
- once transferred to PC successfully, delete all files from card, put card back in K20D and format in camera.
- files saved under folders by date; this works for me but might not for others; those who need a better memory jog than date may use tags, etc., to help.
- once through on all files done in Geeqie (speedy fork of GQView); tossers deleted without mercy to save disk space.
- RAW conversion done via RawTherapee using neutral profile for those images that need further dissemination; I use PEF format myself but DNGs are fine, too. Only other manipulation I do in RT is cropping.
- any further manipulation such as noise reduction, refocus, etc. done in the Gimp. Only exception are those rare cases where I need 16-bit output. In that case, I do most of the editing in RT and output to 16-bit from there.

Note that a sound backup strategy should be of paramount importance for anyone who cares about their work. You can do a search for a recent backup thread on the forums here. You should be thinking of your RAW files the same way film photogs regard their negatives and positives. Without them, its all gone.

pasipasi · 07-06-2009, 03:51 PM

Mine's similar to jbinpg's.

- take the card out and plug it in to the card reader
- create a directory under my photo dir with date, 200x-xx-xx-location/whatever and transfer the photos
- conversion using rawtherapee. I usually add some curve with rt, but any further pp using gimp. Sometimes I can create a profile for a set of pictures in rawtherapee and save time in processing.

I've never needed 16 bit for editing, but I'd like to welcome that option in gimp as well.

heliphoto · 07-06-2009, 05:27 PM

Originally posted by disasm

...
Now I'm obviously missing some steps, of most importance metadata. Is there anyway to edit metadata for the pef files, or should I add metadata to the finished jpegs, or would you suggest I take my photos in DNG format?

Sam

I've found exiftool can read and write metadata to PEF files.

**rparmar** · 07-06-2009, 06:57 PM

Originally posted by jbinpg

My workflow:

- take card out of camera, stick it in PC, transfer to Photos folder.
- once transferred to PC successfully, delete all files from card, put card back in K20D and format in camera.

I never do this. I backup the photos from the local drive to two different network drives, so I have the files in three places total. Then, and only then, I format the memory card.

However, before backing up I go through the images and purge the obviously bad ones. I have no interest in keeping every half-baked shot I take.

Mr. The Guy · 07-06-2009, 07:05 PM

Originally posted by heliphoto

I've found exiftool can read and write metadata to PEF files.

exiftool is awesome. With bash scripts, you can automatically file and back-up too.

simico · 07-07-2009, 04:41 AM

Originally posted by jbinpg

- any further manipulation such as noise reduction, refocus, etc. done in the Gimp.

Do you use some plugin for noise reduction? I haven't found any built-in tool for that.

Quicksand · 07-07-2009, 11:02 AM

I have a bash batch-conversion script I've been working on for a couple of weeks, in my spare time, to do basic raw conversions.

Basically, I was pretty unhappy with the Linux-based options for batch conversion, in that whatever profile selections I made (e.g. sharpening, noise reduction, etc.) would be applied to *all* files, in wildly disparate situations, sometimes very inappropriately. For example, with no noise reduction, ISO3200 files would look terrible. And with enough NR to make those look decent, the ISO100 pictures would get smeared out. It was too labor-intensive to go and convert all my raw captures by hand.

So I wrote this script that pulls the ISO value out of every file it converts, and uses that to select how much noise reduction and sharpening is applied. That's much better! It's still no substitute for hand-editing the really good "keepers," of course (I use RawTherapee for that), but it's better than blind defaults for everything else.

Anyway, I'll probably work on this script a little more over the coming days, and since I posted this tease I guess I'll have to post what I come up with.

It uses dcraw for raw conversion (and noise reduction) and imagemagick for some post-conversion processing (gamma, contrast, saturation, sharpening), and everything is done in 16-bits-per-channel until the final conversion to JPG. It uses exiv2 to copy exif information from the raw file to the converted JPG.

Pretty handy, if I do say so myself. But I'll have to keep you in suspense a little longer, since it's not yet ready for prime time.

bobD · 07-07-2009, 01:57 PM

Phatch batch processor .

photobatch.stani.be

Quicksand · 07-07-2009, 02:15 PM

Originally posted by bobD

Phatch batch processor .

photobatch.stani.be

Hey, that looks pretty cool for watermarks and such. But it looks like it doesn't handle raw files yet:

https://blueprints.launchpad.net/phatch/+spec/raw-support

Although it's probably possible to hack it in easily enough, since it's in Python.

jamonation · 07-09-2009, 09:22 PM

I tend to just do a ufraw-batch conversion at 2048x1024 resolution for every pef file. Then delete the jpgs I don't want. At which point I'll do a for i in *.jpg; bash script to rm any pef without an associated jpg file. Then open the file I want to work on with ufraw and tweak it before exporting to the gimp.

Seen me through 10k+ photos on my K10D and another 2k+ on my new K20D, so so far so good. Probably I spend too much time, but hey, it's a hobby for me

RBellavance · 07-10-2009, 05:58 AM

Originally posted by simico

Do you use some plugin for noise reduction? I haven't found any built-in tool for that.

Wavelet denoise plugin (based on the denoising algorithm of dcraw & UFRaw): Wavelet denoise | GIMP Plugin Registry

Duck Dodgers · 07-10-2009, 06:07 AM

Originally posted by Quicksand

[...] Anyway, I'll probably work on this script a little more over the coming days, and since I posted this tease I guess I'll have to post what I come up with.

Yes. Yes, you will.

RBellavance · 07-10-2009, 07:16 AM

I wrote my own transfer script to bring the files from the SD card to the computer (I configured it as the default action to take when a new card is detected in my card reader).

It copies the files from the card in a directory named "~/photos/yyyy/yyyymmddnn/", where nn is a sequential number starting at 01, useful in case of multiple transfers on the same day.

The files are renamed as they are transfered, to yyyymmddnn-xxx where yyyymmddnn is the same as the directory name, and xxx is a sequential number. I shoot mostly RAW+JPG these days, and both files are kept with the same name, but the relevant extension (i.e. .pef, .dng, or .jpg).

The files are not automatically removed from the card. I manually format each card after I have confirmed the transfer is complete.

The whole hierarchy is hosted on a mirrored filesystem, and I also rsync the whole thing to two other machines.

I browse the images with Gthumb (who cheerfully ignores the RAW files), deleting as I go along. This only deletes the JPG, however, so I have written a "cleanup" script to delete "orphan" RAW files (i.e. those with no corresponding JPG file) periodically.

I have linked the "0" key on the keypad in Gthumb with a script that calls Gimp with either the JPG being viewed or its corresponding RAW version if the latter exists. RAWs are handled by the UFRaw Gimp plugin.

I use Gimp for corrections (rotation, shadows&highlight, b&w conversions, etc), sharpening (Eg-SmartSharpen), and resizing. I have written a Gimp script to automate the resize-for-the-web-sharpen-and-add-a-border operation.

I usually try to keep an "original" JPG version under the initial filename. I use "original" in quotes because this will often be the RAW-converted file, sometimes with cropping or other non-trivial modification. Other files will get suffixes appended to their filename, ex: "2009071001-005-crop-bw-web.jpg" would be a cropped, black & white, "web" version of the image 20090710-005.jpg.

Well, it makes sense to me anyway

pontusadefjord · 07-10-2009, 11:51 AM

I use Digikam for most stuff. Importing, organizing and some editing (curves, wb etc). I still use Gimp for more advanced editing, but Digikam is a great app. It made me start using KDE!

I'm not so sure about the RAW stuff, I mostly shoot JPEGs. I've just gotten started with "real" photography and not ventured into the RAW world yet.

07-06-2009, 11:08 AM	#1
disasm New Member Join Date: Jul 2009 Location: State College, PA Posts: 2	Linux Workflow (Ubuntu) Hey all you pentax owners, I'm a linux users and saw some other threads with linux users, so I thought I'd share my workflow, and hope to get some better suggestions from everyone else does for a workflow. I'm completely new to SLR's but spent my honeymoon in Germany learning tricks of the trade. I shoot in manual, PEF raw, and in the last month I've had the camera, I've shot almost 50 gigs of pictures. I have a directory ~/photos I keep all my photos in to separate them from other Pictures and what not. I rsync my camera card to ~/photos, and then clear the card. When prices come down a bit more I might consider buying new cards for every shoot, but for now I delete them. Then I create subdirectories for the event and move the entire day directories into that subdirectory i.e. ~/photos/honeymoon/101_0605 and then look at 101_0605, and decide what to rename that folder, i.e. heidelburg Then any pictures I want to "develop" I open in ufraw, and save to ~/photos/developed/category/etc... Now I'm obviously missing some steps, of most importance metadata. Is there anyway to edit metadata for the pef files, or should I add metadata to the finished jpegs, or would you suggest I take my photos in DNG format? I'm open to any suggestions, and am curious what all of you are doing for your workflow. Sam

07-06-2009, 07:05 PM	#6
Mr. The Guy Inactive Account Join Date: Jan 2007 Location: Vilcabamba, Loja Posts: 216	Bash scripts Originally posted by heliphoto I've found exiftool can read and write metadata to PEF files. exiftool is awesome. With bash scripts, you can automatically file and back-up too.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Processing RAW files in Linux (Ubuntu)	krishna	Troubleshooting and Beginner Help	9	04-23-2010 10:58 PM
Linux -- Ubuntu Karmic 9.10, libraw 0.8.5, Raw file processing	K7er	Digital Processing, Software, and Printing	4	01-13-2010 08:35 PM
Linux guru, DSLR n00b: how do I formulate a good workflow?	rpriedhorsky	Digital Processing, Software, and Printing	3	12-26-2008 03:23 AM
Complete color management and workflow under Linux -- from a Pentax user	zaurus	Photography Articles	14	05-04-2008 04:17 PM
A More Precise Workflow & Sharpening Workflow	benjikan	Digital Processing, Software, and Printing	2	06-01-2007 06:07 AM

07-06-2009, 03:45 PM	#2
jbinpg Site Supporter Join Date: Mar 2008 Location: Prince George, BC Photos: Gallery \| Albums Posts: 3,546	You should get lots of feedback on this. Ubuntu 9.04 user here. My workflow: - take card out of camera, stick it in PC, transfer to Photos folder. - once transferred to PC successfully, delete all files from card, put card back in K20D and format in camera. - files saved under folders by date; this works for me but might not for others; those who need a better memory jog than date may use tags, etc., to help. - once through on all files done in Geeqie (speedy fork of GQView); tossers deleted without mercy to save disk space. - RAW conversion done via RawTherapee using neutral profile for those images that need further dissemination; I use PEF format myself but DNGs are fine, too. Only other manipulation I do in RT is cropping. - any further manipulation such as noise reduction, refocus, etc. done in the Gimp. Only exception are those rare cases where I need 16-bit output. In that case, I do most of the editing in RT and output to 16-bit from there. Note that a sound backup strategy should be of paramount importance for anyone who cares about their work. You can do a search for a recent backup thread on the forums here. You should be thinking of your RAW files the same way film photogs regard their negatives and positives. Without them, its all gone.

07-06-2009, 03:51 PM	#3
pasipasi Veteran Member Join Date: Feb 2008 Location: Oulu Photos: Gallery Posts: 532	Mine's similar to jbinpg's. - take the card out and plug it in to the card reader - create a directory under my photo dir with date, 200x-xx-xx-location/whatever and transfer the photos - conversion using rawtherapee. I usually add some curve with rt, but any further pp using gimp. Sometimes I can create a profile for a set of pictures in rawtherapee and save time in processing. I've never needed 16 bit for editing, but I'd like to welcome that option in gimp as well.

07-06-2009, 05:27 PM	#4
heliphoto Veteran Member Join Date: May 2008 Location: Region 5 Photos: Gallery Posts: 2,539	Originally posted by disasm ... Now I'm obviously missing some steps, of most importance metadata. Is there anyway to edit metadata for the pef files, or should I add metadata to the finished jpegs, or would you suggest I take my photos in DNG format? Sam I've found exiftool can read and write metadata to PEF files.

07-06-2009, 06:57 PM	#5
rparmar Site Supporter Join Date: Jan 2008 Photos: Gallery Posts: 8,823	Originally posted by jbinpg My workflow: - take card out of camera, stick it in PC, transfer to Photos folder. - once transferred to PC successfully, delete all files from card, put card back in K20D and format in camera. I never do this. I backup the photos from the local drive to two different network drives, so I have the files in three places total. Then, and only then, I format the memory card. However, before backing up I go through the images and purge the obviously bad ones. I have no interest in keeping every half-baked shot I take.

07-07-2009, 04:41 AM	#7
simico Veteran Member Join Date: Jan 2009 Location: Budapest Posts: 821	Originally posted by jbinpg - any further manipulation such as noise reduction, refocus, etc. done in the Gimp. Do you use some plugin for noise reduction? I haven't found any built-in tool for that.

07-07-2009, 11:02 AM	#8
Quicksand Pentaxian Join Date: Apr 2008 Location: Central Ohio (formerly SF Bay Area) Posts: 1,519	I have a bash batch-conversion script I've been working on for a couple of weeks, in my spare time, to do basic raw conversions. Basically, I was pretty unhappy with the Linux-based options for batch conversion, in that whatever profile selections I made (e.g. sharpening, noise reduction, etc.) would be applied to all files, in wildly disparate situations, sometimes very inappropriately. For example, with no noise reduction, ISO3200 files would look terrible. And with enough NR to make those look decent, the ISO100 pictures would get smeared out. It was too labor-intensive to go and convert all my raw captures by hand. So I wrote this script that pulls the ISO value out of every file it converts, and uses that to select how much noise reduction and sharpening is applied. That's much better! It's still no substitute for hand-editing the really good "keepers," of course (I use RawTherapee for that), but it's better than blind defaults for everything else. Anyway, I'll probably work on this script a little more over the coming days, and since I posted this tease I guess I'll have to post what I come up with. It uses dcraw for raw conversion (and noise reduction) and imagemagick for some post-conversion processing (gamma, contrast, saturation, sharpening), and everything is done in 16-bits-per-channel until the final conversion to JPG. It uses exiv2 to copy exif information from the raw file to the converted JPG. Pretty handy, if I do say so myself. But I'll have to keep you in suspense a little longer, since it's not yet ready for prime time.

07-07-2009, 01:57 PM	#9
bobD Pentaxian Join Date: Nov 2008 Location: Waikikamukau Photos: Albums Posts: 1,248	Phatch batch processor . photobatch.stani.be

07-07-2009, 02:15 PM	#10
Quicksand Pentaxian Join Date: Apr 2008 Location: Central Ohio (formerly SF Bay Area) Posts: 1,519	Originally posted by bobD Phatch batch processor . photobatch.stani.be Hey, that looks pretty cool for watermarks and such. But it looks like it doesn't handle raw files yet: https://blueprints.launchpad.net/phatch/+spec/raw-support Although it's probably possible to hack it in easily enough, since it's in Python.

07-09-2009, 09:22 PM	#11
jamonation Veteran Member Join Date: Jun 2008 Location: Toronto Photos: Gallery Posts: 364	I tend to just do a ufraw-batch conversion at 2048x1024 resolution for every pef file. Then delete the jpgs I don't want. At which point I'll do a for i in *.jpg; bash script to rm any pef without an associated jpg file. Then open the file I want to work on with ufraw and tweak it before exporting to the gimp. Seen me through 10k+ photos on my K10D and another 2k+ on my new K20D, so so far so good. Probably I spend too much time, but hey, it's a hobby for me

07-10-2009, 05:58 AM	#12
RBellavance Veteran Member Join Date: Nov 2006 Location: Near Montréal, Canada Photos: Gallery Posts: 1,716	Originally posted by simico Do you use some plugin for noise reduction? I haven't found any built-in tool for that. Wavelet denoise plugin (based on the denoising algorithm of dcraw & UFRaw): Wavelet denoise \| GIMP Plugin Registry

07-10-2009, 06:07 AM	#13
Duck Dodgers Veteran Member Join Date: Dec 2007 Location: in the 24½th Century! Photos: Gallery Posts: 439	Originally posted by Quicksand [...] Anyway, I'll probably work on this script a little more over the coming days, and since I posted this tease I guess I'll have to post what I come up with. Yes. Yes, you will.

07-10-2009, 07:16 AM	#14
RBellavance Veteran Member Join Date: Nov 2006 Location: Near Montréal, Canada Photos: Gallery Posts: 1,716	I wrote my own transfer script to bring the files from the SD card to the computer (I configured it as the default action to take when a new card is detected in my card reader). It copies the files from the card in a directory named "~/photos/yyyy/yyyymmddnn/", where nn is a sequential number starting at 01, useful in case of multiple transfers on the same day. The files are renamed as they are transfered, to yyyymmddnn-xxx where yyyymmddnn is the same as the directory name, and xxx is a sequential number. I shoot mostly RAW+JPG these days, and both files are kept with the same name, but the relevant extension (i.e. .pef, .dng, or .jpg). The files are not automatically removed from the card. I manually format each card after I have confirmed the transfer is complete. The whole hierarchy is hosted on a mirrored filesystem, and I also rsync the whole thing to two other machines. I browse the images with Gthumb (who cheerfully ignores the RAW files), deleting as I go along. This only deletes the JPG, however, so I have written a "cleanup" script to delete "orphan" RAW files (i.e. those with no corresponding JPG file) periodically. I have linked the "0" key on the keypad in Gthumb with a script that calls Gimp with either the JPG being viewed or its corresponding RAW version if the latter exists. RAWs are handled by the UFRaw Gimp plugin. I use Gimp for corrections (rotation, shadows&highlight, b&w conversions, etc), sharpening (Eg-SmartSharpen), and resizing. I have written a Gimp script to automate the resize-for-the-web-sharpen-and-add-a-border operation. I usually try to keep an "original" JPG version under the initial filename. I use "original" in quotes because this will often be the RAW-converted file, sometimes with cropping or other non-trivial modification. Other files will get suffixes appended to their filename, ex: "2009071001-005-crop-bw-web.jpg" would be a cropped, black & white, "web" version of the image 20090710-005.jpg. Well, it makes sense to me anyway

07-10-2009, 11:51 AM	#15
pontusadefjord Senior Member Join Date: Apr 2009 Location: Uppsala Photos: Gallery Posts: 111	I use Digikam for most stuff. Importing, organizing and some editing (curves, wb etc). I still use Gimp for more advanced editing, but Digikam is a great app. It made me start using KDE! I'm not so sure about the RAW stuff, I mostly shoot JPEGs. I've just gotten started with "real" photography and not ventured into the RAW world yet.