Tongfamily (Richtong.com Sync)

OK, it is a bit eccentric, but when you have 10,000 or so photos a year (?!!) that need to get processed, you begin to rely on yourself more and less on software. That is I’ve had various photo programs (iPhoto, Aperture, Lightroom) lose databases and crash several times. Also I’ve had bad hard drives corrupt JPGs (why is there no error detection in JPGs I wonder), so I get paranoid.
Finally, although most folks do everything in Adobe (and indeed, I can sort of use Lightroom and Photoshop decently well now), I still like certain programs for certain things that they do. So here goes the horrible adventures of Rich with photographs:

1. Shoot in Raw. SD cards are basically free and so is disk, so I normally shoot in Raw and then process. I used to do JPG+Raw but after a while, I really just didn’t like the in camera Jpeg processing for most of my cameras (Canon 5D2 mainly). So I process everything.
2. Copy .CR2 files to at least two servers using different technologies. You never know when a bad drive will get you and I’ve got a couple photos this way, so I keep the files backed up on separate drives. They are also backed up into the cloud and also I keep the jpgs as a final backup on another hosted server.
3. Organization of .CR2. Believe it or not, the fundamental organization I used is by year, then by month folders then by day. This seems like a waste, but it is nicely unambiguous and you rather get collisions of file names this way. I don’t try to index these, but use metaindices (right now mainly iPhoto Faces). Most importantly I find that file systems are pretty stable beasts.
4. File Corruption and Error Correction [To be done]. An important to do is to find a daemon which goes through files every so often and does a compare and spits out what doesn’t match. I also at some point need to find something that adds a SHA1 hash against files so that you can see which ones have been corrupted and ideally do some error correction. I’m sure someone has done this but need to research. I could write some scripts to do this as rsync has this facility and does but I’m sure someone has figured this out.
5. For iPhone photos, I copy these out of iPhoto when I occasionally sync (there seems to be no easy way to get the high resolution photos out of iCloud that I can find), then process them as well.

After I get the raw files into locations, I have a simple method for having a queue of unprocessed files:

1. Put the unprocessed files in /Pictures/jpg/2014/2014_08_19 and then I can see by the underscores what is not processed. And when I’m done processing, the finished jpgs are in /Pictures/jpg/2014/2014-08/2014-08-19 and the raw goes into a parallel directory along with all the sidecar files needed by the various processing programs (eg /Pictures/raw/2014/2014-08/2014-08-19).
2. So at my leisure after copying, I can go to everyone of the underscored directories and start working. This also gives me a way to do things on laptops and multiple machines. I have the same directory structure and then when I’m done, I use the tool Goodsync (a mac program which is basically a visual rsync to copy files up) to the main server and the backup server.
3. So what is processing. The first step is using DxO. Although an older and slower program, I find that its distortion correction is very good and automatic. There is much discussion about its noise algorithms but most of my images don’t have too much. For the dark photos, I use their Prime noise.
4. I set the thing for unconstrained crop so that I can crop out things easily and then export the files as JPEGs are 96 reduction. This gives me 4-10MB JPEGs from 22-25MB CR2 raws. DxO automatically produces sidecar files as well so I take all those and the .CR2 and put them into the raw sub directory.
5. I have a few special cases for processing. For panoramas that need to be stitched, I’ve tried Lightroom’s integration with Photoshop but it doesn’t seem as flexible as Hugin. This really gives you some nice tuning and blending. The only major bug I’ve hit with Hugin is an ironic one. If you overlap the photos together too much then the enblend crashes. You need to first force DxO to make TIFs and then process these so the files are huge. One I have the panorama in TIF, I normally use Preview to create the JPEG at just under the maximum resolution. There is just a slider, so you have to go the max and oodge down.
6. For HDR with multiple exposures, I use paid applications called Photomagic. It seems to have the best combination of settings. I particularly like the Tonal Fusion. I’ve tried both Luminance HDR which seemed to crash alot and Lightroom’s Edit In Photoshop HDR mode but that didn’t see to give enough options.
7. For really blurry photos, I turn up DxO unmask, but that hasn’t worked super well, so I’m trying Focus Magic’s Photoshop plugin. It has gaussian deblur but I haven’t really investigated if it works better. Photoshop itself has a smart unblur but I’ve not tried it.
8. If I have shot a short 30 second video, I just put that directly into the JPEG directory and don’t process it too much mainly because iMovie and so forth a slow and hard to work with for simple things like cropping or image stabilization.

Finally now that I’m done, I then start replicating files. Here’s the process:

1. Copy from the work machine using Goodsync to the main server. I should really also have a CRC check done, but I haven’t find an easy way with Goodsync to do this.
2. Copy from the main server to the backup server. Again I should verify that, but don’t now
3. The backup server is automatically backed up with Crashplan to the cloud.
4. I then manually ftp the JPEg directory to Bluehost for the viewing gallery and as another archive.

I know this isn’t the most efficient, but it works for me and seems pretty robust. For disseminating, I use two techniques:

1. I use iPhoto for metadata which is mainly the face view, so I will import (without copying the photos) into a dedicated machine and have it grind away looking for Faces. I have had iPhoto crash many times with 40,000 photos, so every month or so I manually backup the .iphotolibrary directory to the servers.
2. I use iPhoto to then push up contents to iCloud by creating new streams for people who have iDevices to look at. For general folks, I use the Bluehost gallery with password protection to share.