| March 24, 2011 9:36 pm

For my first entry in the “Lifehacks” series, I thought that it would be nice to tackle a very common problem. Namely;

What is the best way to synchronize a photo library between two computers?

The Problem: Many Pictures and Many Computers

Many Pictures

To those who might not be aware, I moonlight as an artist. I like to photograph, paint, draw, design, and play about with type. I also avidly “collect” images that inspire me. Some of these get filed away as potential references, others are sent to my digital photo collection. Because of these interests, I have a lot of pictures.

Some of them are precious (like those of family and friends), others are less so. Regardless, I want to be able to view and edit them on any of my computers.

Windows Live SyncMany Computers

Like any geeky person, I also use a bunch of different computers. I have a work laptop (a PC), a personal laptop (a Mac), and a variety of desktops (two Linux workstations, a Mac workstation, and a collection of PCs).

I understand that the number of computers might seem a little bit excessive (and I suppose it is), but each one is used for a different purpose. The laptops are my everyday workhorses, the Linux boxes run specialized software, and the PCs came with the various offices I frequent. To be productive across all of them, though, it’s important that my files are available. (I dislike hauling around USB sticks, they tend to get lost.)

Keeping in Sync

I’ve been experimenting with ways to keep my files synchronized for a long time. I’ve even written about some of those attempts on this website. I’ve used rsync, DropBox, Version Control, and several other tools, and while each of the aforementioned has a place, I’ve been most happy with Microsoft’s Windows Live Mesh.

Windows Live Mesh gets some very important things right:

  • It works on Windows and Mac (though not on Linux, unfortunately).
  • It lets you synchronize any number of folders and any number of files.
  • It can send files across the internet and through firewalls.
  • It provides 5 GB of free space for online backup.

Note: Dropbox is also great, but lacks the ability to synchronize files between just two computers. This means that you are limited to the 2 gigabytes that they provide you with. Live Mesh, on the other hand, can be used to move hundreds of gigabytes worth of data. It handles my music collection, videos, eBooks, and other files marvelously. I use the 5 gigabytes of storage for important documents, projects that in-progress, etc.

What about Data?

Adobe LightroomBut while Mesh works for sending files, that’s only part of the problem. Like any sane person, I use a photo album program to manage my pictures. While it is possible to manage your pictures from Windows Explorer or the Mac Finder, it is miserable. Why would I want to chase around file names when there are very good programs which handle all that for you?

(Truly, a question for the ages.)

To organize my photos, I use Lightroom. It’s a great program and well worth its price. But there is one problem with it, there isn’t an easy way to synchronize your library data on one computer with that on another.

You could just synchronize the files and not worry about the data, but for a Lightroom user, that’s not really a viable option. The data is also very important. You see, one of Lightroom’s greatest strengths is that it uses a non-destructive system to edit your pictures. It stores all of the changes (adjustments to color, crops, etc.) in a side file instead of making the changes directly to the copy on the disk. This helps to prevent corruption, but, more vitally, it also helps to prevent stupidity. (I can’t tell you how many times I’ve accidentally ruined a photo by messing about with the exposure settings. Luckily, in Lightroom, you can just revert it to the original file that came out of the camera.)

All of this data is stored in a monolithic file called the “catalog.” Trying to synchronize the catalog file (which is basically a self-contained database) is a real mess. I’ve tried a variety of ways of doing it (including writing a custom script), and have simply given up. If you just let Windows Live Mesh move it between computers, it easily becomes corrupted. Unless you have a server model for syncing the data, it’s hard to resolve differences and duplicates. Writing a server solution is more work than I want to put in. See why  data synchronization is a problem?

The Solution: File Sync with Mesh, Data Sync with Lightroom

It turns out, however, that there is a rather elegant solution to this problem which lets you use Mesh to move the files and Lightroom itself to synchronize the data.Here’s how it works.

  1. Create a folder hierarchy for all of your Lightroom pictures. Use this folder hierarchy for importing new shots.
  2. Turn on the XMP side car files for your images.
  3. Add the Lightroom folder to Windows Live Mesh so that it automatically syncs between different computers.
  4. Use “Lightroom folder sync” to add new files to your Lightroom catalog on other machines in the mesh.
  5. Do not try to sync the catalog file. This will cause bad things to happen.

Step 1: Create a Folder Hierarchy for Lightroom PicturesImport your pictures into an appropriate subfolder in the hierarchy. You can do this manually or let lightroom handle it.

The first step, somewhat obviously, is to create a place for Lightroom to store your pictures. If you’ve been using Lightroom for a long time, then you probably already have this set up. If not, then you’ll want to create a folder hierarchy for your image files. A folder hierarchy is simply a top-level folder and sub-folders that are organized into a schema.

Because I’m obsessive compulsive about where my data gets stored, I do this manually. I have folders for every year of my collection, with subfolders corresponding to each month. I then organize my photos according to the event where they were taking. If you’re less of a control freak than I am, you can let Lightroom do the organization on its own. (It generally does a pretty good job, creating subfolders for each day rather than the event.)

When importing your photos, import your pictures into a folder in the schema.

Step 1: Create a folder hierarchy and import your files into an appropriate subfolder.

Step 2: Use XMP Side Car Files

Once you’ve created your folder hierarchy and imported your photos, the second step is to create XMP side-car files for them. Generally, Lightroom stores all of the metadata and printing instructions in the Lightroom catalog (a monolithic database). However, if you tell it to, it can also export that data to an XML file that is saved to the same directory as the picture.

For synchronization using Mesh, this is something that you need to enable. To do so:

  1. Go to Edit > Catalog Settings > Metadata
  2. Place a checkmark in the “Automatically write changes into XMP”
  3. Press “Ok.”Step 2: Enable XMP Side Car Files
  4. Be sure to enable XMP side car files on all the computers where you are going to use Lightroom.

Step 3: Add the Lightroom Folder to Windows Live Mesh

After you’ve enabled XMP side car files, you are then in a position where you can use Live Mesh to synchronize the folder between computers.

  1. Open Live Mesh and click on the “Sync a folder” button.
    image
  2. Navigate to the folder that you use for your Lightroom photos. Press the “Sync” button.
  3. Select the other places where you would like for it to sync to. Press “Ok.”
    image
  4. Allow for the files to synchronize. This may take several hours, depending on how many photos are in your catalog.

Step 4: Use the “Folder Sync” Option to Add New Photos to Other Catalogs

Once you’ve got Lightroom set up to use side car files and Windows Live Mesh to copy your pictures, the last step is to import the files into the catalogs on your other computers. This is done by using the “Synchronize Folder” command.

image

You can synchronize any folder on your computer, including the top level folder. When you activate the command, Lightroom will go through and add any new photos it finds to your catalog. If there is an XMP file present, it will add the metadata there as well.

The whole thing works like magic, even across a mesh that is as complicated as mine.

Comments

6 Responses to “Lifehack: Synchronize Photos with Live Mesh and Lightroom”

Jeremy wrote a comment on May 19, 2011

Great article!

I have recently noticed that using Live Mesh with Lightroom is creating multiple .lrcat files. For example on my laptop the lrcat file is called Ligthroom-Netbook.lrcat and on my pc it is called Lightroom-PC.lrcat. I have deleted these extra .lrcat files so that there is only the base Lightroom.lrcat file, but from some reason lighthroom keeps creating multiple ones. Any way to prevent this? Or am I just doing something wrong?

Rob Oakes wrote a comment on May 31, 2011

Hi Jeremy,

Thanks, I really appreciate the feedback.

Unfortunately, Windows Live Mesh doesn’t do a very good job merging different types of files. Foremost among these are database files.

.lrcat files are databases, which is why Lightroom is creating multiple copies. It is detecting differences between the catalog on one computer and on the other. Because it doesn’t know which is more current, it creates multiple copies, and allows you to delete the old ones as you notice them.

To get around this, I only use Mesh to transfer files. My Lightroom catalog is actually kept in another directory, that isn’t synchronized. (Though the backup folder is, because I want to have backups in case it becomes corrupted.)

If the extra .lrcat files bother you, you might try something similar …

Cheers,

Rob

z666zz666z wrote a comment on November 30, 2011

I need something much simpler that that and i can not find any.

On Windows WinMerge can do what i want perfectly, also it runs on Wine, but over Wine it crashes when i try to copy files/folders.

I mean:
-i want to have multiple HDDs (most are USB HDD, SD cards, microSD cards, etc) in partial sync

I explain such “partial”

For each drive i want to have a copy on another place not of all, only of some parts, and also on another place copy of other parts…

So, some relative paths are on more than one disk.

An example will illustrate it much better:
-Structure of HDD0
/dir1/subdir11/*
/dir1/subdir12/*
/dir2/subdir21/*
/dir2/subdir22/*
.etc.

-Structure of HDD1
/dir1/subdir11/*
without /dir1/subdir12/*
without /dir2/subdir21/*
/dir1/subdir22/*
.etc.

-Structure of HDD2
without /dir1/subdir11/*
/dir1/subdir12/*
/dir2/subdir21/*
without /dir1/subdir22/*
.etc.

-And more HDDs…

Imagine:
-HDD0 is an SD card on a Photo Camera
-HDD1 is an SD card on a Video Camera
-HDD2 is an USB-HDD on a SetTopBox TV
-HDD3 is an USB-HDD connected to a MediaPlayer SetTopBox
-etc

I want all Photos and Videos on both HDD2 and HDD3, but not mixed along HDD0 and HDD1, so on HDD0 there is nothing of HDD1 and viceversa, but both HDD2 and HDD3 have all, the schema is more complex but this simple example shows what problem i have.

Now the big problem: I wnat sync not only files, also empty folders!!! yes also empy folders.

And i want a tool that asks me for two paths, then reads file/folder structure and show me a list on which i can re-order by:
-Only on right
-Only on left
-On Both sides but different (only for files, different size, datetime, content, etc)
-On Botn sides and equal (only for files)

So i can order the list and do a multi-select by range and do:
-Copy from left to right
-Copy from right to left

And also for text files let me enter on them and show diffs, by lines, by words or by chars.

WinMerge on Windows do that pretty well and shows the list very clear.

Basically:
-If a dir is empy, must appear on the list
-If a dir is not empty and exists on both sides must not apear on the list (only if recurse mode is on)
-If a dir is not empy but only exist on one side, must appear on the list and its content not (last winMerge version fails on this, also show the content, so i must be care of that, previous versions did not show the content of a dir that only exist on one side)
-If a file is only on one side, must appear
-If a file is on both sides, must appear and be identified as: equal or different (for text files content and/or filesize), and also at same time, never on left, never on right, or same datetime
-etc

The list with columns like:
-Path
-Entry name (dir name for directories / file name and extension for files)
-Size (for files)
-Datetime (for both, directories and files)
-Status: Only on left, Only on Right, On both equal, On both different never left, On both different never right, On both different with equal datetime on both sides, etc

I want to be able to re-sort the list ascending / descending by any colum, including status.

I want to be able to multi-select on the list and do an action to all selected entries at once: delete, put left as right is, put right as left is; this include touching datetime of files and folders, not only copy files and create directories.

Hope i let it clear.

But in case not: I do not want an automated sync, i want a compare / sort, then do what i tell to do.

I say this because a lot of people tell me about rsync, etc… such tools do not fit at all…

Imagine there are 300 different files, but i only want some of them to be sync now… i must be able to select them from a list and then sync them… same for folders.

A workflow can be like this:
-Compare HDD0:/dir1/subdir1/ with HDD1:/another_path/other/ in non-recursive mode
-See the list and decide to only sync two subfolders of HDD0:/dir1/subdir1/ that are not present on HDD1:/another_path/other/, so i select two entries on the list and tell Left->Right
-Then select another different entries and tell Right->Left or LeftRight
-Sort for status, so only present on right side are all together on the list and can be easy selected
-Select them and telll Left<-Right
So i have only sync files that are not present on the other side
So i have also sync folders that are not present on the other side, note that such folders can also have subfolders and files, a hole sub tree

.etc.

I can not find any tool to do that on Linux that works… Wine+WinMerge crash when i tell it to do the copy, also on deletion, only alows me to see the list and sort it, so it helps, but not much since i must do manually the copy of each entry.

Most of tools i tried on Linux like Meld, etc… do different things, but i found no one that let me sort the list as i want, then select more than one list entry at same time and do an action to it… not to mention most of them tend to not put on such list the folders… most only shows files and i also need folders.

WinMerge3 is showing it will be also available for Linux… hope it arrives soon, but till then???

Just to simplify a lot imagine the next scenario when coming back from Hollidays:
-I try to have on three external disks a copy of all SD cards of my family photo-cameras and video-cameras, so i can have one USB-HDD connected to TV while other conected to a MediaPlayer and another to a PC, etc… and also have multiple backup copies of all photos and videos, and whatever i want
-When all external HDDs are on sync, the SD card can be left empty, so it can hold more future photos
-And the cicle re-start…
Hey! on the USB-HDDs i want not to lost / delete old photos / videos and also have some other kind of stuff, files and folders, and some not on all USB-HDDs.

Now imagine a paranoic: Will not use only three, will use four, five or ten USB-HDDs, each with a copy… well i also burn them on DVDs.

Rob Oakes wrote a comment on November 30, 2011

This sounds like a fantastic tool for both dissemination and backup of files. It’s far beyond what any of the “simple” synchronization clients try and set up, though. My suggestion would be that you look into rsync. rsync is a command line utility that can be used to transfer files between two unix hosts (there is a port for Windows, but it doesn’t work very well).

You would need to write some scripts to automate the process and come up with the logic beforehand, but it looks like you’ve already done that. Then, you could add those scripts to a cron job so that it happens in the background. It wouldn’t quite handle everything (such as the sequential burning of data to DVD), but it would definitely allow you to get copies of all of your files to the different destinations.

I actually use a similar technique to get copies of my pictures off my Mac (which being Unix, runs rsync) onto the server I use for backups. Hmm, I might even write a post which explains how …

Paul wrote a comment on March 25, 2012

Hi Rob, just found this now as I have exactly the same use case as your and also use live mesh. Adding the XMP sidecar data is a great tip, thanks. I did one extra step after changing this setting: right-click on catalog directory and “Save Metadata” forced LR to generate the XMP files for all the edits I had done on that machine. Rinse and repeat for the other machines and enable sync on the catalog directory. All good.

Yancy wrote a comment on October 23, 2012

Hi Rob, thanks for posting such a through review. I know this is old, for technology sake, but, I wanted to post.

While this solution works, it makes an assumption the picture folder with “files in hierarchy” can easily sync between all devices. In my case, and many photographers who shoot non-stop, I have close to 1TB of images on a Mirrored Raid array and syncing the photos folder across a desktop and 2 notebook would effectively use every bit of storage (and them some on the laptops).

Can you help find a “simple” solution to having photos on a disk array, accessing the array over a network, and keep one local catalog “synced” across each device. In theory LR will access the photos, makes changes to them inside the catalog, and the catalog re-syncs across the computers to work on a different device. I know this is a big ask to figure out, but it has plagued my photography for months, and external drives are not an option.

In researching the net people are using dropbox to “sync” their catalog with some success in this manor, however, my catl. has grown beyond the 2GB’s drop allows for free.

Any solutions, hints, or just general conjecture you can give me to get me moving in the right direction would be very helpful. I downloaded Mesh b/c I believed it could sync my large catalog across computers while I access the photos on a shared networked raid but immediately ran into the multi-catalog save issues. – so live mesh + photo XMP side car + large photos files = not possible for me.

Care to comment?