Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
05-16-2018, 03:52 PM   #1
Digitiser of Film
Loyal Site Supporter
BigMackCam's Avatar

Join Date: Mar 2010
Location: North East of England
Posts: 20,571
Thoughts on my library and workflow directory structure?

As part of my recent move to Linux, I'm now working with different tools for my photography.

I'll start by mentioning that I have a variety of cameras. Where available, I shoot RAW. For Pentax, I use DNG format, but I have other cameras that only produce their native manufacturer RAW format (e.g. ARW for Sony). Then, I have a couple of older cameras that output TIFF, and others that only produce JPEG.

Darktable reads numerous RAW formats, but the one it's almost-guaranteed to read is DNG, so for RAW developing, my preference is to use DNG files - either straight from the camera, or converted from the native format using either digiKam or, when necessary, Adobe DNG Converter (which I'm running under WINE).

My workflow is as follows:
  • Import, rename, tag, cull and manage photos with digiKam 5.x
  • Where required, convert to DNG with either digiKam or Adobe DNG Converter
  • Develop images with Darktable 2.4.3, exporting to TIFF
  • Edit developed TIFFs (or original TIFFs & JPEGs) with GIMP 2.8.x, saving sessions as .XCF
  • Export full- and re-sized JPEGs for screen and print use with GIMP

As part of the move, I'm taking the opportunity to completely re-design my file-naming conventions and directory structure for my future work to support this (and, hopefully, any future tool migration - however unlikely), and I'd appreciate your thoughts on what I've come up with - whether you think it's over- or under-engineered, what I might be missing, ways to streamline etc.

Here's my proposed directory structure:


Firstly, albums will be organised into year directories (2016, 2017, 2018... etc.). Beneath this will be an album for each day, named "yyyy-mm-dd" as a minimum, but when possible, with a suffix relating to the event (e.g. "2018-06-05 Durham Cathedral". Within each dated album, the following structure will be used:




The purpose of the directories is as follows:
  • 1_Development - contains DNG, TIFF or JPEG originals for development in Darktable (so sidecar files would reside here too) or possibly in GIMP (JPEG and TIFF)
  • 2_Developed_Exports - fully developed images, exported from Darktable as TIFF
  • 3_Editor_Files - when files from 2_DevelopedExports are edited in GIMP (or other), the XCF or equivalent files containing the layers and edits are stored here
  • 4_Edited_Exports - final edits from GIMP, exported as TIFF
  • 5_Outputs_Full_Size - full size output files from GIMP (or, potentially, direct from Darktable if no edits were needed), ready for printing or display - typically in JPEG but possibly in other formats too
  • 6_Outputs_Resized - resized output files, mostly for web use or sharing with family / friends
  • 7_Native_RAW_non_DNG - for my cameras that only produce manufacturer's native RAW, those files are stored here (converted-to-DNG files are stored in "1_Development")
  • 8_Camera_Imports - this is where I will import files to from camera or card; it's a staging area, from which I'll convert to DNG and move files to either "1_Development" or "7_Native_RAW_non_DNG" as required. When I've completed my conversions and file moves, this directory should be empty

I'm still thinking about my file-naming conventions, but this is what I have so far:
  • Files will be imported into "8_Camera_Import", renamed in the format "yyyymmdd_hhmmss.ext"
  • Developed exports will be named as "yyyymmdd_hhmmss_D.ext" ("_D" to signify they are from the Development stage)
  • XCF files created from JPEG or TIFF originals will have the original name ("yyyymmdd_hhmmss") while those created from developed DNG will retain the "yyyymmdd_hhmmss_D" naming
  • Edited exports will be named "yyyymmdd_hhmmss_E.TIF" ("_E" for edited)
  • Full-sized outputs will be named "yyyymmdd_hhmmss_O_FS.xxx" (where "O" is for output and "xxx" is the required format, usually "JPG")
  • Resized outputs will be named "yyyymmdd_hhmmss_O_[width]x[height]_[intent]" (where "width" and "height" are the pixel dimensions, and "intent" is "screen" or "print", but optionally blank for general use)

So... what do you good folks think? Am I missing anything? Have I gone too far? The intent here is to have original files, converted files, edits and exports properly organised and named so that I can find and identify them easily.

Any input - advice, feedback, critique - would be much appreciated!


Last edited by BigMackCam; 05-16-2018 at 04:08 PM.
05-16-2018, 04:21 PM - 1 Like   #2
Senior Member




Join Date: Feb 2017
Posts: 146
Hi BigMack, I read most of your post, no so much the file naming details and how you would organise the files but I get the general meaning.<br />
<br />
I photograph for fun so my workflow is likely different from that of a pro.<br />
I used to keep my photos in computer folders and it was painful... File naming was a pain, as can be concluded form your post. It was also hard to find and filter certain photos (eg all photos of a specific person), hard to select and share them, hard to organise them in albums - forget about having the same photo in more than one album, way too complicated - etc etc etc....<br />
<br />
Now I use apple photos. I edit all my photos there, I keep them all there, I show them there, I organise them by albums there, I can find photos by tag or by person or by day. The app can even find photos if you search with keywords (eg: “dogs”). It’s smart enough to know which photos have dogs...You can also search photos by date taken and create smart albuns with filters that self-populate them. If you tell the app the same person is in multiple photos, it will start detecting them in newer photos.. You can export to any format as well.<br />
<br />
For more complex edits, I use affinity photo, save as jpeg or tiff and import back to apple photos but generally it’s a one stop solution.
05-16-2018, 04:25 PM - 1 Like   #3
Senior Member




Join Date: Jan 2016
Location: Phoenixville, PA
Posts: 136
The workflow is nice, well designed.. however will work only on your staging area.
What have you thought about archiving the photos eg., as per location/month/topic etc., else these folders will quickly become huge and unmanageable.

I personally use Pentax SilkyPix and this allows me to save the photo info like lens, aperture, speed, iso etc right in the file name.
I simply create folders with YYYYMMDD_<Topic name> so that I can easily retrieve them back. I save the DNGs in a separate folder and eventually delete them after my jpegs are processed
05-16-2018, 04:43 PM - 1 Like   #4
Site Supporter
Site Supporter




Join Date: Mar 2008
Location: Prince George, BC
Photos: Gallery | Albums
Posts: 3,543
Looks good, Mike. The query about backup strategy is on point. You need to develop one if you don't have one already. Note that with the original raw and the matching xmp for the edits, you can recreate anything you like up to the point of sending to GIMP. - Jack

05-16-2018, 05:03 PM - 1 Like   #5
Loyal Site Supporter
Loyal Site Supporter




Join Date: Dec 2014
Location: Marietta, GA
Posts: 51
QuoteQuote:
Files will be imported into "8_Camera_Import", renamed in the format "yyyymmdd_hhmmss.ext"
Having gone through a similar exercise, I don't believe that the file naming convention is guaranteed to be unique even for a single body. For example, a burst with multiple frames within the same second.

With multiple bodies, the potential for conflicts is even worse especially if there is any error in the clock settings.

The scheme I have adopted is "yyyymmdd-hhmmssNN-CCC-BBBB.ext" where NN is a sequence number withing the same second, CCC is an identifier for the camera body, and BBBB is a batch number across multiple bodies.

The details of the naming scheme are listed in: Consolidating Pics From Two Cameras - PentaxForums.com

The batch number at the end of the name is particularly useful for uniquely identifying files in the date folder. This lesson was learned the hard way and was added after the initial renaming pass. If gets old fast finding files by time.
05-16-2018, 06:12 PM - 2 Likes   #6
Loyal Site Supporter
Loyal Site Supporter




Join Date: Apr 2015
Location: Bay Area California
Photos: Gallery | Albums
Posts: 796
I dislike file and directory naming as a way to impart info to images. I think a much more robust, universal, scalable and flexible system is to use keywords and other IPTC info instead. That's what it's designed for. You don't have to make some of the hard choices about what directory something has to be in, since one file can only be in one directory (yeah, there's symlinks but c'mon). All the info you list above could be accommodated in keywords, I believe. Some might fit into "instructions" or other IPTC categories as well. Filenames I leave along unless there's a reason to change them for sequencing, but that's usually just exports.

I'd not aware of the tools that are easily used in finding files with IPTC data in Linux, but digiKam is probably quite good at that. And Xnview MP would work, and even does hierarchical keywords quite well.
05-16-2018, 06:35 PM - 1 Like   #7
Site Supporter
Site Supporter
jatrax's Avatar

Join Date: May 2010
Location: Washington Cascades
Photos: Gallery | Albums
Posts: 12,991
QuoteOriginally posted by Oakland Rob Quote
I dislike file and directory naming as a way to impart info to images.
Agree. @Bigmackcam I think you are over thinking the whole directory thing. I've never tried Darktable but I'll make the assumption it has similar sort and find functions as Lightroom. So embed your search criteria within the IPTC as keywords and other metadata. That is future proofing your images.

A directory structure is important but maybe not as extensive as you envision. As an example mine is a folder for each year and under that a folder for each day. So even without any keywords I can find at the least the images from a particular day. But with embedded keywords and metadata I can do searches both from Lightroom and from the operating system or most any other photo viewing application.

I started out many years ago naming a folder for each 'event' and found it quickly became very awkward. I strongly recommend against that. YMMV of course.

05-16-2018, 10:50 PM - 1 Like   #8
Pentaxian




Join Date: May 2016
Photos: Albums
Posts: 1,990
I suggest, if you are renaming files, to call them yyyymmdd_hhmmss_XXXX.ext, where XXXX is the number in the original filename (as in _IMG2354.RAW).

I keep my image files in a folder structure for permanent storage and ease of backup and verification, named similar to yours, with the date and then a description. But I keep them separate from the edited files. I also agree, however, that there is a lot of benefit to be had by also using a cataloging program, like Apple's Photos, or Aperture, or Lightroom. I'm not aware of what the Linux options are, however.
05-16-2018, 10:53 PM - 1 Like   #9
Veteran Member




Join Date: Sep 2017
Location: Bangalore, India
Posts: 581
I did not see any mention of naming of HDR and Stitched files. Coming to the EXIF for final stitched file, I used to include the source file names in description field, I did that manually. I have not done any stitching in many years; may be current tools do that by default.

I generate UUID for each file and suffix it to original file(Ex: _xxxxx_K1_e5932793e764493fb4ef8c28fc2102de.*). I have written a tool to do this. UUID is auto generated, not MD5 hash of the file content. Having UUID in the name opens up many possibilities. May sound like over-engineering but may come handy sometime in future. Simplest application is you can merge several folders in to one folder without worrying about name clash.


I prefer date based folder hierarchy instead of storing everything in one single folder. NTFS gets slower if number of files in a folder crosses some threshold. Microsoft has documented this already. Having folders helps in backup management also. Say something like 2017 goes HD1 and 2018 files goes to Hd2.

Last edited by pentaxfall; 05-17-2018 at 12:46 AM.
05-17-2018, 02:31 AM   #10
Digitiser of Film
Loyal Site Supporter
BigMackCam's Avatar

Join Date: Mar 2010
Location: North East of England
Posts: 20,571
Original Poster
Thanks for all the helpful responses thus far, folks. Lots of food for thought

I perhaps didn't make clear that my intention is to use quite extensive key-word tagging (from within digiKam) to support image searches. I've used tagging in the past, but not as prolifically as I should. As a result, quite a lot of my back-catalogue isn't as well tagged as I'd like.

The idea behind my directory structure and naming conventions wasn't to replace, but rather complement the key-word tagging approach. I figured it would provide an easy and consistent means to locate files without being totally reliant on my current library management tool and metadata searches. I'm not sure if I'll even need that capability, but I suppose I'm trying to ensure I'm not tied into a specific tool-set.

As for unique photo naming (beyond date and time), that's something I hadn't thought of. Whilst I tend to use only one camera at a time, I do often shoot bracketed exposures for scenes with high dynamic range - so it's definitely conceivable that I could have a naming conflict. I like the idea of using the original file name as a suffix, but the other options are interesting too.

Regarding my backup strategy, I don't yet have one specifically for photos, but for my PC (running Linux Mint 18.3) I take RSYNC snapshops of my system files after each system update using Timeshift, and store these on an external hard drive. For my data (which includes photos), I currently use the supplied Backup Tool to create a compressed archive on an external disk - however, I'd like to move to something like Back In Time, as Backup Tool is rather inefficient in taking full backups rather than incremental snapshots.


Thanks again for the feedback thus far. It's much appreciated
05-17-2018, 10:37 AM - 1 Like   #11
Loyal Site Supporter
Loyal Site Supporter
Kevin B123's Avatar

Join Date: Jul 2016
Location: Hampshire
Photos: Gallery | Albums
Posts: 3,162
This is an interesting question and one I have faced in the last year. I use LR, and have no knowledge of the software you are using, however the structures are independant of that to some degree.

I think your structure is sound, if a little complicated.
Would it be important to know the camera?, for malleability of the files for example.

Jatrax states that naming folder after events became complicated, it might be worth exploring why that is. For my workflow I have done just that.

For what is is worth, here is how I have run it in LR6, if any of this is useful, then great.


Lightroom will import a folder if there is a picture in it.
So, in windows explorer create a new folder and then numbered folders within the new folder:

New Folder1
1. RAW K-3ii
1. RAW K-50
2. NIK-Other
3. Export
4. SOOC
5. Moto G4 Phone

In each sub folder, paste a small gash photo, the same one in each as it will be deleted after (this enables LR to see the folders).

Then, go further and create a folder template for the year 20XX for example.

Copy and paste the New Folder 1 into it n times.

Import a renamed copy of the folder template for the year e.g 2019 into LR.
(when I do this in LR, I delete the gash photos as soon as the structure is imported)

My camera files are set to create folders as Number-date (e.g. 100_DDMM).

My workflow then is this:
Suppose I went to town in January (01Jan18) and shot some raw files with the K-3ii.
I rename a sub folder within my new Photos 2018 main folder as ‘Town’
I rename the photos on import to the ‘Town’ - ‘1-RAW K-3ii’ folder as ‘Town 01Jan18_**’ where ** is a sequential number created by LR, and 01Jan is given by camera folder.

Of course, I will visit town often this year and each time I will use the same format, with the photo date amended to that given by the camera folder (e.g. 100_DDMM).

That way, I can do it later, I don’t have to remember anything, and I can just look into the folder to see they are town photos.

If there were an event in town, I may choose to rename one of my n folders after the event name, and go from there.

Now, when I process any raw files, the JPEG output is always saved to the 3. Export folder.

I might process a SOOC image and again the JPEG output is always saved to the 3. Export folder.

I may choose to edit a raw file in Topaz, or NIK or DCU5 if it is PS image and I need that functionality. Then I save an output from those applications as a 14bit TIFF file to the ‘2.NIK-Other’ folder and work them from there until a JPEG is then saved to the 3. Export folder.
The original photo name is retained, and picks up an extension as ‘EDIT’ (e.g. ‘Town 01Jan18_1-EDIT’.JPEG’ OR ‘Town 01Jan18_1-EDIT’.TIFF’)

The beauty of this is that I can see at a glance which files have been processed, they will always be in the ‘3. Export’ folder for any event or location or year. There is no mystery or memory needed to see where I am.

Of course I would love to take credit for this system but I can’t. I got it from Ed Gregory of photos in color, his video here is well worth watching.


It is LR specific, but the folder structure should be valid anyway, and the basic workflow should read across to other software as it fundamentally has to do the same things.

I hope this is of some small use to you, or maybe others venturing into LR for example.
Attached Images
 
05-18-2018, 11:38 AM - 1 Like   #12
Veteran Member
emalvick's Avatar

Join Date: Feb 2008
Location: Davis, CA
Photos: Gallery
Posts: 1,642
What you are doing seems like it will work. It's probably a bit more than I would do, but I do think that the ideal organization scheme is one that utilizes both metadata (keywords, Exif, XMP) and file naming. File and Folder naming are great for portability if you keep a naming scheme that you can utilize. Keywords and other metadata are useful so that you aren't trapped to a file name and not having to make file names too complicated.

My own scheme isn't too different from the OP, although I forego the sub-folders. My folders are always something like:

2018\2018-05-18_Subject\

File names are then usually

20180518-0930_CIDx_1234.RAW (originals)
20180518-0930_CIDx_1234-x#.ext (versions and subversions, ext is usually PSD, JPG, or TIFF)

CIDx is a 4 character identifier for the camera (body)
1234 is the original image number (or last 4 digits of)... this eliminates the issue with multiple images in the same minute (or second even). I'm sure it isn't 100% solid, but I've never had a conflict with 15 years of photos, but I don't shoot a lot either, and I am not a pro (1k to 3k photos per year).

The last portion (after the last - ) is a way of tracking versions.
I usually keep all the files together in the same folder because I can quickly see what photos have been processed as they will also sort together as sets. the x is usually 1 to 4 letters that might help me know what the version is for (if important) or the software it came from or something else. Is that important? only to keep file names unique. That information gets cataloged within my DAM software, but the file name clarifies the chain of versions from the original RAW file.

My DAM software includes a robust version control that I set based on file name. It links the files to their original RAW based on the naming scheme and then automatically keeps keywords in-sync. I then use categories (my DAM software uses categories as either a separate metadata field or internal to its DB), which are for my own tracking. Thus I can give parameters that I don't want people who get the photos to know that are specific to my workflow.

So when a file has a name

20180515-0930_PXK3_1234-v1-b1.jpg

the DAM knows that the original file was 20180515-0930_PXK3_1234.RAW and I know that it was actually a 2nd generation black and white (because the -v1 is the first derivative and -b1 is the 2nd).

-Erik
05-18-2018, 02:23 PM - 1 Like   #13
Loyal Site Supporter
Loyal Site Supporter




Join Date: Dec 2014
Location: Marietta, GA
Posts: 51
QuoteOriginally posted by emalvick Quote
...
My DAM software includes a robust version control that I set based on file name.
...
Out of curiosity, which DAM software are you using?
05-18-2018, 03:29 PM   #14
Veteran Member
emalvick's Avatar

Join Date: Feb 2008
Location: Davis, CA
Photos: Gallery
Posts: 1,642
I am using IMatch.

It provides multiple methods of version control (i.e. it can automatically identify versions and propagate information from master to versions per user preferences).

I set things up based on file name, but other methods of version identification can be used (i.e. date and time, manual, etc).
05-24-2018, 04:27 AM - 2 Likes   #15
sbh
Site Supporter
Site Supporter
sbh's Avatar

Join Date: Oct 2012
Location: Black Forest, Germany
Photos: Gallery
Posts: 847
Now I haven't read every single post in detail, tldr. However, I am on the rather keep-it-simple side.

Why are you keywording? I tried it for a while but it was too cumbersome for too little benefit.

My current workflow is to basically neither use a large catalog nor categories like years. I tried it all. I noticed that the way that I photograph, everything can be seen as a session. A shooting, a trip, an event, outing etc... So I use a session based workflow. Sessions are just folders on the hard drive and have a certain naming scheme – that's all. Easy to handle.

YYYY-MM-DD Session name that makes sense

Every session is made of three sub folders:

1. Capture (import raw/tif/jpeg files as they are shot – culling happens here)
2. Edits (a.k.a. "Selects" – here are the tiff files that I'm working on)
3. Otuput (jpeg exports etc... easy to find and access at any time)

I find this very slick and tidy.

As software I use Capture One. It has very good support for this workflow. You can either just drag the raw files into the capture folder or use the import function to rename (incl. tokens), adjust etc. the files on import.

But basically any software can be used with this workflow.

Last edited by sbh; 05-24-2018 at 04:34 AM.
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
cameras, darktable, digikam, directory, dng, files, format, gimp, jpeg, photography, photoshop, tiff
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
K-1 side grip first clicking, now developing into structure having loose feeling lesmore49 Troubleshooting and Beginner Help 9 01-30-2018 03:17 PM
Thoughts on hybrid workflow--"appropriate" level of PP? CreationBear Film Processing, Scanning, and Darkroom 9 03-30-2017 02:24 PM
Structure-Architecture:complete and details bbluesman Mini-Challenges, Games, and Photo Stories 725 03-27-2013 08:33 PM
The structure of HOYA Corporation and myth about Tokina ogl Pentax News and Rumors 61 09-12-2009 09:12 AM
A More Precise Workflow & Sharpening Workflow benjikan Digital Processing, Software, and Printing 2 06-01-2007 06:07 AM



All times are GMT -7. The time now is 09:33 AM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top