[Lowell Goudge]

This has been beaten into the ground on every photo forum known to man. Regardless, let's do it.

I've been 'serious' about photography since 2003. Right now, I have about 25,000 photos. F-spot for Linux automatically puts images into folders by date: Photos/2008/04/30/IMGP6025.JPG. Before I was using F-spot, I would manually do things like Photos/2006/02/Feb-15-Molly's_birthday_party/P1010568.JPG.

The latter option seems much better, especially since I have almost no photos tagged within F-spot, and it won't write EXIF data to RAW files (only JPEG). The problem I have with naming folders by event is, what do you do when you have ONE photo of something? Like I'll have 2008/04/Apr-29-Kids_at_the_park, but in the middle, there's a nice bee macro. I don't want to leave it in that folder, because it's not a photo of a kid. But on the other hand, I don't want to create a folder for one picture. Similarly, what if I love taking flower pictures? I might end up with:

2008/04/Apr-15-Flowers
2008/04/Apr-16-Flowers
2008/04/Apr-16-Photos_at_the_pool
2008/04/Apr-17-Flowers

In a case like this, it would seem that creating one big "April Flowers" folder would make sense. Maybe you could subdivide it by daisies, roses, irises, etc. But that seems to go against the date organization thing. In fact, why limit it to April flowers? Why not Spring Flowers? Or 2008 Flowers? Or, hell, just "Flowers"?

Another major nagging issue is that of quantity and manageability. There is no reason I need 26000 photos. None. On the other hand, I can't bear to throw any away. That screams ARCHIVE! But how? If I archive EVERYTHING and keep only a couple hundred really good pics a year, the folder thing seems to only be necessary for the archives. Then how do I organize the keepers?

I know everyone has this same conundrum. I'd love to hear what everyone else does.

Joe

Joe

what I do, is to make directories by year, then month.

As a result, I have 12 directories in each year,

In addition, if there is a special event, such as a trip, I make subdirectories off the month for that trip, or off the year if the trip spans months. For these special events I divide things further by loction, etc.

All my origonals get filed this way.

I have separate directories, for specific items of interest, for me, mainly birds and wild life where I put copies of origonal images divided by species. I do any post processing on these images, so that the origonals files by date taken are unchanged

any time I do post processing, I save under a modified file name, again preserving the origonal image

[sabarrett]

I forgot to list my archive/backup stuff too.

On a weekly basis I copy all my files to a freenas server in my basement. Its just an old PII system with a couple of drives in it. It is a share on my home network used solely for backing stuff up on a live basis. My main system is set for the BIOS to boot the computer at 6:30 am and I use Syncback to run this automatically. I have a profile set in that software to backup a portion of my data directory each day, so it doesn't take too long.

Also, I make DVD copies (2) of the current year, roughly each quarter and one copy of this in our safe deposit box(SDB). I figure they have fire protection, a big concrete room, steel doors, climate control, etc. If that building goes, I probably have bigger worries than my pictures. The other copy stays at home.

At the end of each year, I run a full backup of that year and put that in the SDB too. It is about time to refresh the old years DVDs, as I don't want to let them get too old. Also, I buy only prograde DVD disks too. For the little extra they cost, I don't want to risk my images. Most of the shots we have are family things and its all digital - I don't want to lose my records of my two daughters lives.

I am thinking of adding a pair of large USB disks to the SDB mix, as DVD management is getting tough. The 10mp file size is getting the byte size out of hand. I would just keep one at home, constantly updated, and at the end of the quarter, swap that one with the one in the SDB and update the one I just brought home.


How does anyone else manage their data integrity?

Scott

[Lowell Goudge]

How does anyone else manage their data integrity?

Scott

I have a 2 level approach to back ups.

first approach, weekly or when ever I shoot a lot of photos, I back up all my data on a USB hard disk, which is kept in a secure location,

second approach, when there is enough data, I back up onto dvd.

I do the same with my work computer.

I use Lap-Link to do the back up, and have it set to overwrite older files, so that it only changes files that I have worked on, or added.

[Rick]

I'm overdue at sorting this out... and the files are piling up fast of course.

I'm only partly tardy; relational databases are part of my profession and that seems a natural for the subject of filing/retrieving images. I've been off and on wondering just how I should go about doing this, but haven't yet found the time to get serious. I suspect some of the current options are relational databases, but I just haven't looked yet. I expect I'll start looking at something with the image name being the primary key, and then figuring out how to quickly and efficiently assign attributes to images/groups of images that will allow them to be quickly filtered and retrieved using a select statement. Microsoft Access appeals to me simply because I use it so much, but a more specific program might well be easier to use at least as far as image specific database software goes.

While awaiting the day that I actually do something about this, the file organization is pretty simple. As files come out of the camera, they go into folders with the naming convention <100 or 101>_<month><date> e.g something from today would be 101_0502. "100" and "101" simply because I now have digital files from 2007 and 2008, so when the new year rolled in I incremented by one until I could get this increasingly important job done. Once I'm on top of it, really won't matter what the prefix is. The files within the folders have the folder name as the first part of the file name, then increment from there i.e. 101_0502_001.dng.

Did I mention it would be nice if the camera software allowed me to set up the folder/file naming schema instead of having to do it after the fact?

Once I have processed the folder, the prefix in all instances is changed to the year i.e. "100" becomes "2007", "101" becomes "2008". As I scan all my slides in from the past decades, they will be integrated into this schema as best possible. I tended to write dates on my slides whenever possible, and of course the processing companies dated slides beginning at some point, so those archives should be reasonably close chronologically. After processing, the "keepers" get a "!" appended at the end of the file name i.e. 2008_0502_001!.dng. If it was good exactly as shot, then there will only be one file like that, the original having the "!" added to its file name. If the "keeper" was postprocessed in any way, then I will have two files: 2008_0502_001.dng and 2008_0502_001!.dng. I try to minimize nesting folders as much as possible, but I have given thought to a nested "keepers" folder inside of the chronological folder. Advantages both ways, I guess.

That isn't very user friendly as far as being able to look at a directory listing and have some idea of what it is all about. However, it is logical for the purposes of indicating the time frame of the images and for back up purposes.

Sorting large amounts of data into meaningful groupings and making retrieval logical and easy is the purpose of the relational database, not the folder/file name. If I want to see pictures of gun dogs, I don't want to scroll back through ten years of images looking for titles with "gun dog" in them - I want to enter a simple select statement with criteria and have the RDB do all of that for me.

Again, I suspect some of the software mentioned functions pretty much like this using assigned keywords/tags/whatever - I just haven't had a chance to look at this.

It's good to hear what software others are using, but the first question in my mind is how are searchable attributes assigned to an image/group of images? Are the tags/keywords directly written into some part of the image IPTC/EXFIL? Or does the software link those tags/keywords with the image file name within its internal file management system? Personally, I would prefer the image title, tags, keywords, etc to be written directly into the images metadata, rather than it depending on management within the software being used.

And following out of THAT question, I personally need to learn a lot more about IPTC/EXFIL metadata. One of the first things I would like to know is how I can go in and edit that data. I'm told, for example, that there is a GIS sector in the EXFIL standard for storing position - however, it must be in degrees/minutes/second. As a GIS analyst, for varying reasons I really don't want to deal with DMS; I would much prefer using a Cartesian coordinate system like UTM. If I choose to put my photos up on an IMS for example, I don't want to mess around with projecting DMS to UTM or State Plane so that I can get a photo geodatabase to work properly.

So partly the reason I'm not entirely tardy in resolving my folder/file organization schema. I want to understand exactly what kind of image metadata I have to work with and what I can do to manipulate it before either choosing some available software or putting together an MS Access system of my own. A lot of front end planning at the front end of relational database design/selection can save a WHOLE LOT of grief and time further down the road.

Data integrity? Regular copyies of my graphics folder off-disk as part of my normal backup procedure. Double density DVD's copied from that, so I essentially have two backups of all my data at any given time, one of which is always in another physical location. HD's are cheap these days, and I have thought of a 500 Mb external for the time being, but unused HD's can die from sitting idle for months at a time as well.

My best guess is that storage options and capacity will keep up with our image storage requirements. Example: a Blu-Ray dual layer disk holds 50 Gb of data, compared to what you get on a dual layer DVD. Spendy now - but so were CD's not so very long ago.

I'll conclude by saying that Picasa is surprisingly easy and quick to use for manual photo retrieval even using my somewhat abstract organizational schema. It scrolls through folders extremely quickly while allowing you to obseve big enough thumbnails that you can see the subject matter and find what you're looking for. Haven't tried any of its other capabilities.

[zorobabel]

I forgot to list my archive/backup stuff too.

Also, I make DVD copies (2) of the current year, roughly each quarter and one copy of this in our safe deposit box(SDB). I figure they have fire protection, a big concrete room, steel doors, climate control, etc. If that building goes, I probably have bigger worries than my pictures. The other copy stays at home.

Scott

I think you've lost it

I gave tagging in Lightroom a try last night. It's funny how I can't just select some photos, type the tag and enter to save it. It does it for only one. Then the tag appears in common tags, and only then can I select multiple photos, press the tag link in the common tags section - and it saves the tag for all of them... Not user friendly.

[falconeye]

Point is:

• A filesystem is a hierarchical tree
• Images are organized in multi-dimensional way (by event, rating, type)

Therefore, it is widely accepted that it is best to store in an arbitrarily choosen scheme not producing too many files per directory (by day or by month or by no. of import is probably fine, I use by day now).

On the other side, it is important to tag by the stuff of interest (event, type, location). Your software must support mass tagging.

Because tagging can quickly become software-dependent, I write tags back into files (in lightroom you must do this explicitely). This way, my library remains self-contained on disk.

Two further tips:

• Use virtual galleries for interesting stuff (like landscape keepers etc.)
• Geotag your images so you can search for images taken in the neighborhood of a location.
  Geosetter.de reads GPS and writes a full text location descriptor for every point on this planet into the RAW or JPG
  In the newest version of LR, you can click onto a GPS coordinate and see it in Google maps (anybody noticed yet? ).

[reeftool]

I may be the only old coot reading this but even the worse digital sorting program out there is better than the hundred or so shoe boxes full of prints, slides and negatives I have accumulated over the years. A bookshelf taking up a wall with nothing but photo albums......I keep telling myself someday i'll scan all that stuff and put it on dvd's...someday. I find Picasa's very simple method of albums and date sorting wonderful. Sure beats an hour in the attic looking through shoe boxes.

[sabarrett]

Yeah, maybe I am a little paranoid... I'd hate to lose the shots of our daughters tho..

[falconeye]

shoe boxes

You forget that we got a 1000 times more images in digital age when we ever could put into shoe boxes -- except your wife is very busy in buying shoes

[Bob83]

Hi!

i find i rely mostly on a folder structure. I don't like the idea of relying on some database or catalogue program to keep track of the photos. I work with computers and i've found on many occasions with customers computers that either the program or windows itself screws up and the organisation is lost.

My system (and yes, even i can see the limitations) is
\where\when\event
and then other sub-folders to split up sub sections of the event or trip.

eg \Canberra\Easter2008\friends wedding\
with in this are folders such as 'formal photos', 'reception' and 'keepers'

This system doesn't allow me to group by 'flowers' or kids' etc. I do however, like to be able to look through the photos by time or event, such as my holiday oct 2007. Also, this means that any photo has a dir structure that describes where and when it was taken as well as the event / group of people.

I also have a dir for those select few photos that i think turned out really well. these however are all duplicates of photos out of the main file structure.

This probably sounds more complicated than it really is

Retrieval is aided by Irfanview. It has this nice feature (which isn't on in a default install IIRC) that allows a 'right-click' option 'browse with irfanview'. This gives you a quick thumbnail view that lets me quickly browse the folders looking for the one i'm after.

As far as backup, i regularly clone my photo directory onto an external usb hdd which gets stored away from the computer in case of theft. I also am about to buy another so i can leave it at work.

The next problem i have is the size of the directory. I've been shooting in RAW+ and even after deleting ones that are obviously bad, or just messing around, the size of the directories is climbing rapidly. DVD's aren't very practical when you have to split up your collection on to 10 or more dvd's. I guess this is only going to get worse.

Looking after a large collection is a lot of work, but worth it in the long run i think. Its also a matter of what works for you.

Mark

[falconeye]

I don't like the idea of relying on some database or catalogue program to keep track of the photos. I work with computers and i've found on many occasions with customers computers that either the program or windows itself screws up and the organisation is lost.

I agree. This is why I had pointed out that it is important to write back keywords into the EXIFs (or .xmp sidecars at least).

[Rick]

I don't like the idea of relying on some database or catalogue program to keep track of the photos. I work with computers and i've found on many occasions with customers computers that either the program or windows itself screws up and the organisation is lost.

I am not going to promote one method of archiving/retrieving images over another. However, I can't agree over the concerns regarding using a database for organization/retrieval.

I work with computers as well, briefly as a network engineer and now as a GIS analyst (more field time in the mountains, less time chasing down network issues). Small projects aside, that incredibly valuable spatial data is stored in either geodatabases or even more complex SDE's. All the imagery people enjoy using on Google Earth is coming out of a map server - backed by a database. If databases were indeed so error prone, even with a fault tolerant backup scheme in place, people would simply find a better way of storing and retrieving data because they couldn't afford the down time to be constantly restoring data to databases.

Relational databases don't just "screw up" and lose data... okay, to be more accurate, they "rarely" screw up and lose data. But hard drives die as well, taking with them whatever software and schema was stored on them if that also isn't backed up. Do you blame the database if you didn't have your software backed up and when you lost the hard drive you lost the database created within the database software on that drive? The database itself is far more reliable than the hard drive it is stored on.

I have over a Terabyte of data in my company data warehouse that has been in there for over five years now. Most days I am constantly adding to, deleting, and editing various shapefiles and raster data in that data warehouse - some of the raster data from satellite imagery is gigabytes in size. I have yet to lose the integrity of that database. Moreover, I have worked as a subcontractor to probably a dozen other companies who maintain their own spatial databases, and I have yet to hear or experience a loss of database integrity with any of those clients.

I am certainly not conversant with every single RDBMS out there. And some - like Oracle for example - are both overkill and overcomplicated for the average person. However, there are relatively simple and reliable software packages out there like Access or Firebird that will perform as advertised. If people are losing their databases, then it is overwhelmingly due to issues other than the database itself. The fact they didn't back up their hard drive and the drive was corrupted or died isn't the fault of the software, any more than if the drive where all their images is stored dies and they didn't have that backed up either. Hardware does die, and software does get corrupted, that's just a fact of life.

A relational database system of organization/retrieval, either disguised as one of the commercial photography systems out there or as a standalone system, obviously isn't for everybody, and people should use whatever system works for them and that they feel comfortable with. But it is simply factually incorrect to suggest that good database software is unreliable for image organization/retrieval.

[sabarrett]

My fear is not that the database will go bad/corrupt/drive failure, etc. Mine is that the program that has my data will no longer be supported or new features in another program will look attractive enough to me that I will jump ship to another program. Obviously in that case I don't want my data locked in a format that I can't extract somehow. Having the data as part of the image metadata is one way, and having a robust enough export or a scripting language is another.

[falconeye]

The database itself is far more reliable than the hard drive it is stored on. [...] there are relatively simple and reliable software packages out there like Access or Firebird that will perform as advertised.

I run an enterprise software consulting firm.

We had one occasion where Oracle "screwed up" where the file system didn't. This is why backups are so important with databases. A database may loose its integrity and become unusable whereas a filesystem is able to degrade file by file.

MS Access is much worse than Oracle but still quite usable.

Firebird does so easily loose its data that it will hit any individual installation sooner or later.

In practice, however and as it was pointed out, one has to rely on a particular imaging system and its database.

E.g., ThumbsPlus is able to use any SQL database and uses MS Access by default.

LightRoom stores its database in "SQLite format 3" and asks for daily backup... SQLite is open source (SQLite Home Page) and I don't know how reliable this is. I think it is fragile enough to be worried, as I found this remark from one of the SQLite sponsors:

Can its files be corrupted by failures?
We rely on sqlite for data file reliability. If the sqlite file is corrupted at startup, we remove it and repopulate it from the server.

A reliable database must never have corrupt database files on disk concerning transactions which were successfully committed. We better all backup our lightroom catalog...

[PDL]

I have a totally unorgainzed approach.
I copy each SD card to my wolverine and then copy the woverine data to my PC's hard drive. No date, name, subject or anything associated with the folder.
I have a images folder (on another drive) where I copy all the RAW images into that folder by month - the directories are labeled yyyy-mm/RAW. I work out of the month folder to create JPEG's for posting, printing via Costco (bigger than 8x10). I have a "working" folder where I that I use for storing images by theme or subject (forums, Pike Place Market, Yahoo groups, PPG etc).

I backup the images to DVD by the original wolverine folder (on my brand new RAID 1 SATA disks - for fault tollerance not backup). I copy my Working folder to DVD - mostly when I think that the drive is going to fail - about once a quarter. About once a year I have to re-format the wolverine as it gets full. I store the backup DVD's in my closet - I should get a SDB.

I do have real concerns about the viability of DVD's, PEF's, databases, USB, SATA, IDE and other technologies through the years. As storage devices (IDE, SCSI, SATA), DVD -> Blu-Ray --? OS's, CPU's, motherboards, card interfaces (ISA -> PCI -> PCI-x -->?) change - what will we have to do to keep our images around? I have inherited my fathers collection of images (which I have yet to go get), now what am I going to do with several thousand more slides and negatives? Easy - put them into the closet and selectively scan those I want to "do something with". Now as for my collection of digital images? My son has shown no interest at all in preserving my images. I bet that by the time I go to the great beyond, he will be hard pressed to find a piece of hardware that will read a CD or DVD let alone find a SATA drive that would replace the one in my array.

Digtial is fragile in terms of longevity and technological support.

I really am going to go out and buy the DAM book The DAM Book: Digital Asset Management for Photographers to try to get a clue. The only thing I can see as a possible issue is - I do not have PS - Lightroom - yes - PS no way I am spending that kind of money.

The Elitist - formerly known as PDL