16 Oct 2014 - Münster (by Dominik)

Why we don't trust Google's CardDAV API

WAT

Syncing fruux with Google Contacts as well as syncing multiple Google Contacts accounts via fruux is possible since almost a year now in our product. So it's only reasonable that we get asked, why this feature is still in beta after all this time.

Bi-directional sync is one of the harder problems to solve properly, but we were very confident that Google's CardDAV API combined with our experience developing fruux and our underlying sabre/dav technology would be the right ingredients to do it.

It turns out, that we were wrong and by implementing with Google's CardDAV API we've made a huge mistake.

Google couldn't be further from being CardDAV compliant

Google's implementation of the CardDAV protocol has massive problems and until they fix these issues, you shouldn't use their CardDAV service for pretty much anything. Here's why we don't trust Google CardDAV:

1. Data-loss

When sending a contact to Google (in CardDAV contacts are represented as vCards), Google will silently discard data they don't care about. This violates what you'd expect from a proper standards compliant CardDAV system (being able to retrieve exactly the contact you previously stored). Effectively certain information that has been added to a contact gets discarded when storing the contact in Google's system and the next time the contact gets fetched again this data is lost for good.

fruux has a versioned data store, which keeps old versions safe.

2. Rejection of valid vCards

We've sent hundreds of thousands of valid vCards to Google's CardDAV server. A whopping 15% of them get rejected by their system.

3. Bananas error handling

Rejecting certain vCards wouldn't be an unsolvable problem in itself. With proper error handling, we'd be perfectly able to apply Google specific workarounds, but sadly the Google CardDAV way of telling us that valid vCards are rejected is just timing out our HTTP requests (their server just never replies).

There's not even a clear pattern to what's not supported. vCards with attachments -which are quite common, because people love contact photos in their addressbooks- often get rejected, but there are also many vCards that get rejected for one Google account, but work perfectly fine in another account.

Our system keeps track of vCards that Google rejects to make sure that it does not retry forever if a vCard is not accepted.

4. Slowness

Every single write operation (e.g. creating a new contact or updating an existing one) via Google's CardDAV API takes 10-20 seconds (no that is not a typo, we actually meant seconds and not milliseconds) to complete. If that does not sound like a big deal to you, imagine a user with 2000 contacts: If a request times out after 60 seconds, we retry requests two times before giving up and 15% of our requests fail by timing out because Google rejects valid vCards for no reason, this means the initial sync will take over 22 hours to complete. Now also imagine that we have more than one user...

We're using a queue system and a fleet of worker servers that process many jobs in parallel to deal with this problem.

5. Unique identifiers get changed

When syncing with a CardDAV system, especially with one like Google's that silently discards data it doesn't care about (see "1. Data-loss" above) you have to keep a reference to the vCards you are sending, so you can keep the data local that would otherwise get lost.

Luckily in CardDAV, every vCard has two unique and stable identifiers for each vCard - one of them is basically the 'filename' of the vCard, the other one is more or less like a unique serial number. When a CardDAV application stores a vCard on a CardDAV server, it tells the server the desired identifiers for the vCard and the server stores the card with these identifiers. So the application will always know "where to find" and be able to recognize that vCard again.

Unfortunately Google CardDAV discards these identifiers and replaces them with their own. To make things worse, when storing a vCard in Google's system it will not only get new and unexpected identifiers, it will also not even be the same vCard anymore, because Google's system also most definitely silently deleted some data (see "1. Data-loss" above). So the application needs even more workarounds to keep track of all this.

Even for fairly basic CardDAV applications this is a problem: Apple's OS X Contacts.app for example uses these identifiers to organize vCards in groups. Once the identifiers of a vCard got changed by Google's system, the group association is lost.

Our system keeps track of Google's changed identifiers to reduce the impact of this problem.

6. Lack of documentation

Google's CardDAV documentation just refers to the specifications of the open standards, but since Google's system couldn't be further from being a CardDAV-compliant server, this is not helpful at all without clear documentation of all the things that don't follow the standards in their system.

Conclusion

The fact that Google advertises that they support CardDAV is an insult to anyone who works hard to be standards compliant. If you care about your data, you also shouldn't trust Google CardDAV until they fix these issues.

Luckily the work we've put into this feature so far is not futile. There are plenty of great CardDAV servers out there and we'll adapt our server-to-server sync feature to allow syncing with generic CardDAV servers.

This blog post is based on Why Google's CardDAV server isn't by Evert Pot, which you should read if you are interested in more technical detail.

About Dominik

Dominik started fruux back in 2007. He's a lawyer that writes code and has Amazon subscriptions for stuff that normal people buy in the supermarket. Follow him on Twitter or contact him via e-mail.

fruux is a free service that looks after your contacts, calendars and tasks so you don't have to. It makes sure that they are always in sync, no matter which device or operating system you're using. If you've not tried it yet, then why not check us out and let us know what you think! And if you're already using fruux, then we'd love to hear your thoughts and comments. You can also suggest a feature for any upcoming releases or tweet us: @fruux.