Google Contacts API Update
Google announced a new version, 0.3, of their contacts data API yesterday. It adds a significant amount of flexibility, several new fields, and enhances existing fields. Some of the major changes include:
- Structured Postal Addresses – It is now possible to keep the structured address fields that Thunderbird has without converting them.
- A birthday field – I am currently working on enhancing the birthday field that will be present in Thunderbird 3, so the birthday values will be synchronized by default. If you use Thunderbird 2 the values will still be stored and displayed once you upgraded to 3.
- Website fields with various types (list)
- More phone number types
- A nickname field
It doesn’t look like all of the changes are available in Gmail yet, but you can see a few of them here.
I am going to start working on Version 0.3 sometime this week which will use the new API and hopefully include some of the most frequently requested enhancements. For those of you familiar with Bugzilla, I have a bug tracking my changes. I am going to focus heavily on Thunderbird 3 starting with version 0.3 and may drop support for 2 with future versions. If you are opposed to this please let me know and I will reconsider.
Here are some potential features. Please keep in mind that I have a full time internship that involves programming and am starting an online class soon, so I may have to delay some of these. The links are to bugs where I will track my progress.
- Code refactoring. This isn’t the most popular feature request, but I would like to like to reorganize the existing code with the new API and these enhancements in mind.
- Contact photos. I am working on contact photos in Thunderbird 3 and have a modified version of gContactSync 0.2.4 that will download photos from Google when adding contacts for the first time. Due to time constraints, 0.3 may be limited to downloading photos only in TB 3.
- Synchronizing all fields that only Thunderbird has and allow for customization of that data.
- Adding a new ‘Relation’ field. This describes the relationship between you and the contact (brother, sister, mother, father, child, coworker, etc.). I haven’t found an existing request for Thunderbird in Bugzilla, so maybe I could add it in Thunderbird rather than just gContactSync.
- Several more types of phone numbers. (list)
- Website field types, and possibly more fields.
- More advanced synchronization with a more intuitive wizard (pick an account, then all groups w/ mailing lists, one group, or no groups/all contacts, plus a few options). Syncing individual groups with address books instead of mailing lists would prevent nearly all error reports that I get.
- A button or context menu to synchronize only one directory.
There have been a few new versions of gContactSync 0.2. It is currently up to version 0.2.3 that handles mailing list bugs a bit more gracefully. Download.
Since the development of 0.3 is going to take some time, I am going to slow down development of 0.2 and let it get translated to more languages.
- ZaZy was kind enough to translate gContactSync to Italian (it). Updates to the locale will be in 0.2.4
- A Russian (ru-RU) translation is being worked on courtesy of Glk63 on BabelZilla.
- A French (fr-FR) translation is in progress by afroxav. Another translation was done by LaGrange.
- Mailing lists are evil. Just kidding, but they can be broken easily rending them (and their parent address book) useless. The cause of nearly every error report I have received so far is directly related to mailing lists.
- The nobody<id>@nowhere.invalid workaround helps but it is still possible to break mailing lists with this workaround.
- People would like to see gContactSync in their native language.
- People have good ideas and valuable suggestions. I have received a lot of feedback and requests. While I don’t have the time to implement every suggestion, I can at least try to add the more popular suggestions
- Creating a forum was a good idea.
Helping with Development
Or if you are good with icons and could make at least a sync button image I’m sure everyone would appreciate it.
Since I finally got around to learning PHP, I decided to create a quick and simple online tool for viewing your Google Contacts. It naturally supports the extra attributes that gContactSync uses (except allow remote content and preferred mail format since those aren’t too important).
My intention was just to make an easy way to view your synchronized Thunderbird/Google contacts from anywhere with Internet access with the ability to show the custom attributes added by gContactSync. If you don’t use gContactSync you are better off using Gmail’s better-looking interface. 😉
It is read-only (you cannot update, delete, or add contacts or groups) for now.
You can find it here. Click on the Login button to be redirected to a page from Google. If you are already signed into a Google Account then you can click on a button to grant access to your contacts for pirules.net (my website).
If you choose to grant access, a token is stored and matched with a cookie on your computer so you only have to sign in once. This token can be used to access your contacts by retrieving, updating, adding, or deleting contacts and contact groups but your username and password cannot be obtained or modified using this token. You can read more about this authentication method (called AuthSub) here.
You will be redirected back to pirules.net where the one-use token is exchanged with a multi-use token (still only for contacts). Then, your groups and the first 25 contacts will be obtained. Click on a group (left side) to show the contacts in it and click on a contact to view more details. Everything is on the same page right now and contacts are obtained using AJAX so the entire page is not refreshed until you logout.
Only 25 contacts (for now) are shown at a time so there are links to the previous and next 25 contacts, if applicable, at the bottom of the list.
Click the Logout button when you are finished, which will de-activate your token unless you want it to remain valid for future use on the site. Logging out takes a few seconds before giving any feedback.
As a sidenote, if it doesn’t work try waiting for 10 or 15 minutes before letting me know because I am actively working on the page and sometimes make mistakes. I just caught a nasty virus (which ruined my first day off), so I may be slow to reply.
- Cookies enabled for pirules.net (only needed until the page or browser is closed)
- Firefox 3.0.6 or higher (tested with a fairly recent build of Shiretoko 3.1b3pre and Firefox 3.0.6). It does not work in Internet Explorer 6, 7, or 8 right now, and it might not ever since I spend a lot of time at work making a website IE-compatible and it is neither fun nor easy. I’m guessing anyone that uses this already uses Thunderbird and Firefox, but if that isn’t the case let me know and I’ll see what I can do.
- It works on Safari (tested with my new iPod touch)
gContactSync 0.2 takes advantage of Google’s updated and improved API for obtaining and modifying contacts. Duplicates are allowed which, combined with a rewrite of the synchronization code, provides a significantly faster synchronization process. The longest delay now seems to be receiving the contacts from Google. There may be some room for improvement there, but I do not want to make too many changes yet until I make sure that the ones I made did not introduce any bugs.
The new API exposes the system groups, which include My Contacts, Family, Friends, and Coworkers. All contacts from the account, including Suggested Contacts are in the synchronized address book. There is a mailing list for each of the pre-defined system groups and each custom group.
You can download it here, but I rewrote the most important code and there may be bugs I haven’t found in it yet. I would only advise those who have hundreds or thousands of contacts and receive the “Unresponsive Script” warning to try it. Please read the upgrade instructions below.
NOTE: Google allows contacts without e-mail addresses, but Thunderbird will break if a contact without an e-mail address is manually added to a mailing list as explained in Bug 20153. If you encounter that bug in 0.1.x please let me know because it is somewhat difficult to fix and all attempts to synchronize after it breaks will fail. To avoid this, 0.2 adds an email address “firstname.lastname@example.org” to any contact without one. However, only unique e-mail addresses are allowed in mailing lists, so only one contact with an empty e-mail address is allowed per group/mailing list is allowed until o.2.0b1. The other contacts will appear in the address book only.
- All the features of 0.1
- Duplicates (multiple contacts with the same e-mail address) are allowed
- Much faster synchronization
- Only US English (en-US) until I finish adding features and strings
- Customization of how addresses are synchronized
- Default groups are enabled (Family, Friends, Coworkers) as mailing lists in Thunderbird
- Thunderbird 2.0 or Thunderbird 3.0 (a1pre through the trunk build b3pre) or Seamonkey 2.0 only (not 1.1.x)
- A Google account or Google-hosted account
- An Internet connection faster than dialup (DSL, cable, satellite, etc)
- More locales
- Maybe addons.mozilla.org?
- Let me know what else you would like, but I can’t promise anything
Upgrade instructions to 0.2.0
- In the address book, synchronize contacts one last time.
- Rename the synchronized address book(s), this is just so you have a backup of your contacts.
- In the main window, go to Tools -> Add-ons -> gContactSync -> Options/Preferences
- In the Accounts tab, select one account at a time in remove them.
- Install the new version and restart Thunderbird. There is no need to remove the previous version first.
- Open the Address Book window and fill in the information for your first account.
Bugfixes/enhancements from 0.1.x
- Bug 20527 – Avoid HTTP Request errors when the Address Book window is closed
- Bug 20509 – Add Seamonkey support
- Bug 20508 – Consider adding overlay.css to customizeToolbar.xul
- Bug 20487 – Add shortcut to preferences in the Address Book
- Bug 20486 – Improve Address Preferences
- Bug 20352 – Update gContactSync to use Contacts Data API 2
- Bug 20148 – Default groups dont appear in Thunderbird
- Bug 19786 – Support for Google’s ‘Most Contacted’ & ‘Suggested Contacts’
- Bug 20153 – Partial fix only – Groups containing contacts without email addresses break in Thunderbird
I have slightly improved support for Groups. Instead of just showing the groups, you can now either remove a contact from groups that it is in (by deleting the groups in the Edit Card Dialog) and add a contact to existing groups by adding a comma, space, and the name of the group to the end of the Groups textbox.
For a simple example, pretend John Doe is in two groups, Group 1 and Group 2 and you want to remove him from Group 1 and add him to the existing group named Group 3.
Go to the edit card dialog and change this: Group 1, Group 2 to Group 1, Group 3
I found a bug where the card view pane on the bottom would show some extra attributes from the last card viewed (third and fourth email and added screennames) while viewing a mailing list.
And, as a reminder, please check out the test release (which doesn’t have the fixes mentioned above) and let me know what you think. I have instructions in my previous post. The deadline for GSoC code changes is August 18, so I’d like to submit code that doesn’t have any major bugs… I will still continue working on this extension after that date, of course.
I wanted to test some Unicode characters to see if they would sync properly. So, I made up an e-mail address: üḯǟôÕ@host.com. The e-mail address synced without any error messages, but when I tried to edit the contact in Gmail I was given the following error: “Please enter a valid email address.”
So, while gContactSync and Thunderbird fully support Unicode, it appears that Google supports Unicode in every field except for email.
I finished adding support for localization to the extension. It previously used l10n for the XUL files, but I just finished localizing strings in the scripts. If you or somebody you know would like to help translate this extension please let me know.
Contact Conversion Rewrite
I completely re-wrote the methods that convert contacts from Google Contacts to Thunderbird Address Book cards and vice versa and went from nearly 700 lines of code to just under 275 (with comments). To do so, I defined an object that stores Thunderbird’s name for the property (like DisplayName), and information on how Google stores the contact data in Atom/XML (the namespace, tag name, whether the data is stored in an attribute or in the child node, and more).
As of now, the supported attributes in Thunderbird include:
DisplayName, Notes, PrimaryEmail, SecondEmail, CellularNumber, HomePhone, WorkPhone, PagerNumber, FaxNumber, HomeAddress, WorkAddress, _AimScreenName, Company, and JobTitle
I finally decided on a possible name that some initial searching on Google tells me is unique: gContactSync. Any feedback on the name is welcome and I’ll gladly change it if I find a better name.
I removed the need for the “shortDelay” preference.
Name of the extension
I haven’t figured out a name for the extension yet. I initially named it gcontacts, but that name already exists for several programs. gcontactsync is a potential name. Any suggestions are welcome.
While syncing some newly-created cards, I encountered the following error:
org.xml.sax.SAXParseException: The entity name must immediately follow the “&”; in the entity reference.
The cause was an oversight on my part during the conversion from an nsIAbCard to an XML representation of the contact. I had forgotten to replace special characters, such as &, <, >, ‘, and ” with &, <, >, ‘, and “, respectively. I was fortunate to catch this error before it caused any real problems. I also wrote a function to do the opposite when converting from XML to an nsIAbCard.
Code refactoring and reformatting
While browsing through the code, I realized that I had changed many of the functions and their parameter types without changing the name and the contents of some of the global variables had also changed, so I started refactoring the code. I am also reformatting the code.
lastModifiedDate/Address Book Listener
Since I added support for lastModifiedDate, I no longer need the address book listener.
I didn’t like my previous HTTP Request methods, so made a few slight modifications. I currently have a generic method for sending an XMLHttpRequest with arguments for the type (POST, PUT, GET, DELETE), URL, body, header label(s)/value(s), and arrays of commands, as strings, to be evaluated depending on the response (OK, 0, or offline, and any error) since the XMLHttpRequest used is asynchronous. This flexibility allows me to call the same function for any HTTP Request that the program must send and act on the outcome.
More contacts supported
When a request is sent to Google for contacts, it only sends the first 25 by default. I now use the max-results query parameter to get the first 250 contacts. 250 was arbitrary, but having that many contacts to sync would probably take a long time. There is now a preference for the number of contacts supported by the extension.
I made a few miscellaneous improvements that resulted in a slight performance increase. More global variables were removed as well.
I have been busy recently, but I did manage to remove all the custom helper methods from the script that works with Google Contacts and just use one generic method instead. I also sped up the sync process by removing the need for one of the for loops I had written. I rewrote the function that updates a Google Contact, but it is still a little rough and will be rewritten again after I figure out the best way to do it.
Well, I finally finished adding offline support, so it will should work well offline. It just sets text in the status bar to let the user know the sync cannot be completed while offline. Once the user is back online it works like normal. Any changes made while offline are synced since the listener still adds the last modified date to the custom4 property. My next goals are to handle errors more smoothly, get some refactoring and additional commenting, and to add support for locales in the scripts. Once these are done it might be ready for an alpha release. Let me know if you are interested.
I have been busy the last few days and will be busy for the next day or two because there is a wedding in my family.
I was attempting to change my sync process today when I opened the Address Book and was greeted with 925 new Address Books. In the end, cleaning obj directory and rebuilding fixed the problem. Strange, but at least my extension wasn’t the direct cause of the problem.
I found and fixed an error that would sometimes occur when trying to sync after adding several new contacts.
After fixing that error, I re-defined what two cards must have in common for the sync function to consider them identical. Since Google only allows one contact/e-mail address, it first checks if the cards have any e-mail address in common (primaryEmail or secondEmail). If there aren’t any e-mail addresses (which is allowed in Thunderbird and Google) it checks the name.
I rewrote my old method to sync individual cards now that I use custom4 for the last modified date. It checks to see first if both have changed (and will eventually have a preference that can ask the user what to do in this case or pick one to always update) and then checks to see which (if any) changed and updates it. Unfortunately, updating a Thunderbird card isn’t as efficient as it could be. I kept getting NS_NOINTERFACE errors while trying to call the modifyCard function in the address book, so I have to actually delete the card and then add the card from Google… Updating a Google contact needs to be re-done soon, too.
In my casual tests I didn’t find any other errors. It now works well offline and syncs every 30 minutes by default or manually. Out of the 11 average contacts I added on one computer, every one synced perfectly with Google and with my other computer.
The only catch so far is that it has to use the card’s custom4 property to store the lastModifiedDate.