Pi’s Blog

My blog about Thunderbird and GSoC 2008

Duplicate contacts in Thunderbird 3

As the author of a synchronization extension for Thunderbird 2 and 3, I get a lot of questions about duplicate contacts.  Duplicates are more common than ever  with the explosive popularity of mobile devices.

There is a fairly popular Duplicate Contact Manager Extension on AMO, but it hasn’t been updated recently, won’t work in Thunderbird 3, and comments mention some bugs, possibly including one that was present in gContactSync 0.2.0a1.

I have not found a way to contact the author, Marian Steinbach, to offer my assistance with updating it for Thunderbird 3.  It looks like the Google Group doesn’t have much activity.

I can reuse a good chunk of code from gContactSync which would allow it to work in Thunderbird 2 or 3, but I would most likely package it as a new extension to not force everyone to install gContactSync just to remove duplicate contacts.  I had some primitive duplicate detection code in 0.1.x, but it was inefficient and only checked e-mail addresses since that’s what Google used to identify duplicates.

So, I have three questions.

  1. Would there be enough interest to warrant new duplicate contacts extension?
  2. Would anyone else like to work with me to develop one?  I already have plenty of code for working with contacts and the address book; it would just need an intuitive, customizable front-end and an efficient way to find duplicates.  I’m just a bit busy with work, gContactSync, online classes, and a possibly emerging social life.
  3. Does anyone have a way of contacting the original developer?

April 6, 2009 Posted by | GSOC, thunderbird | , , | 8 Comments

gContactSync 0.2.0a2

About

gContactSync 0.2 takes advantage of Google’s updated and improved API for obtaining and modifying contacts.  Duplicates are allowed which, combined with a rewrite of the synchronization code, provides a significantly faster synchronization process.  The longest delay now seems to be receiving the contacts from Google.  There may be some room for improvement there, but I do not want to make too many changes yet until I make sure that the ones I made did not introduce any bugs.

The new API exposes the system groups, which include  My Contacts, Family, Friends, and Coworkers.  All contacts from the account, including Suggested Contacts are in the synchronized address book.  There is a mailing list for each of the pre-defined system groups and each custom group.

You can download it here, but I rewrote the most important code and there may be bugs I haven’t found in it yet.  I would only advise those who have hundreds or thousands of contacts and receive the “Unresponsive Script” warning to try it.  Please read the upgrade instructions below.

NOTE: Google allows contacts without e-mail addresses, but Thunderbird will break if a contact without an e-mail address is manually added to a mailing list as explained in Bug 20153.  If you encounter that bug in 0.1.x please let me know because it is somewhat difficult to fix and all attempts to synchronize after it breaks will fail.  To avoid this, 0.2 adds an email address “nobody@nowhere.invalid” to any contact without one.  However, only unique e-mail addresses are allowed in mailing lists, so only one contact with an empty e-mail address is allowed per group/mailing list is allowed until o.2.0b1.  The other contacts will appear in the address book only.

Current Features

  • All the features of 0.1
  • Duplicates (multiple contacts with the same e-mail address) are allowed
  • Much faster synchronization
  • Only US English (en-US) until I finish adding features and strings
  • Customization of how addresses are synchronized
  • Default groups are enabled (Family, Friends, Coworkers) as mailing lists in Thunderbird

Requirements

  • Thunderbird 2.0 or Thunderbird 3.0 (a1pre through the trunk build b3pre) or Seamonkey 2.0 only (not 1.1.x)
  • A Google account or Google-hosted account
  • An Internet connection faster than dialup (DSL, cable, satellite, etc)

Planned Features

  • More locales
  • Maybe addons.mozilla.org?
  • Let me know what else you would like, but I can’t promise anything

Upgrade instructions to 0.2.0

  • In the address book, synchronize contacts one last time.
  • Rename the synchronized address book(s), this is just so you have a backup of your contacts.
  • In the main window, go to Tools -> Add-ons -> gContactSync -> Options/Preferences
  • In the Accounts tab, select one account at a time in remove them.
  • Install the new version and restart Thunderbird.  There is no need to remove the previous version first.
  • Open the Address Book window and fill in the information for your first account.

Bugfixes/enhancements from 0.1.x

  • Bug 20527 – Avoid HTTP Request errors when the Address Book window is closed
  • Bug 20509 – Add Seamonkey support
  • Bug 20508 – Consider adding overlay.css to customizeToolbar.xul
  • Bug 20487 – Add shortcut to preferences in the Address Book
  • Bug 20486 – Improve Address Preferences
  • Bug 20352 – Update gContactSync to use Contacts Data API 2
  • Bug 20148 – Default groups dont appear in Thunderbird
  • Bug 19786 – Support for Google’s ‘Most Contacted’ & ‘Suggested Contacts’

Known Bugs:

  • Bug 20153 – Partial fix only – Groups containing contacts without email addresses break in Thunderbird

February 8, 2009 Posted by | GSOC, thunderbird | , , , , | Leave a comment

My Recent Lack of Activity

I apologize for my recent lack of activity both on gContactSync and Thunderbird.  I have not vanished from the Open Source scene.  The past month was tough (very busy and not fun) due to some strange medical issues that eventually forced me to go to the hospital.  During that time I had to devote my energy to catching/keeping up with homework, papers, and projects for college as well as my part-time job to help pay for college.

I’m doing well now, and will hopefully find some time to work on gContactSync and a few Thunderbird bugs (like an extended XBL datepicker for the Birthday and (not yet created) Anniversary fields) between homework, finding an apartment, and visiting relatives and friends over my one week off for Thanksgiving break (I also have several computers to fix, remove malware from, and setup for relatives).  I accepted a co-op position (not related to Mozilla) from January until August 2009, so I may have more spare time to work on my extension and Thunderbird.

My first goal is to catch up on bugs, e-mail, etc.  I apologize again for my recent unresponsiveness and inactivity.

Status update

  • Thunderbird
    • Bug 456024 Turn the adapted birthday datepicker into an inherited/extended xbl widget – I’ve had a basic patch for some time now that introduces an extended XBL datepicker widget that allows for blank values for the month and year (with M/D empty text) and removes the year portion of the datepicker
    • Bug 456220 Birthday and month shown in card view aren’t localized well – I just need to have some input on how to solve this.  Using the system settings is fine if the year is present, but if not there are two main choices that I see:

      • Just as the system describes (MM/DD/YYYY for me in US English on Linux, Day Name, Month Name D, YYYY in Windows in US English), but with the YYYY, separator, and  day name & separator removed (if present)
      • Look at how the system does it (find the month/day relative order and the separator between them) and print the month name and day in the detected order with the separator.  However, in Linux this would be inconsistent with the MM/DD/YYYY that I get.
    • Bug 456026 Add day/month order indications to the birthday field on address book contacts – The datepicker bug should fix this
    • Bug 458591 Consider improving the labels on the Birthday Field – I just need some recommendations on how to improve them.  There is an outdated image of the new datepicker here (the card dialog was since changed).  Localizer feedback would be awesome since I only know English and some Spanish.
    • Bug 456025 Implement an anniversary field in the address book – Waiting on the datepicker
  • gContactSync
    • Considering new name (gContacts as it originally was).  Any ideas are welcome
    • Working on replying to recent feedback over e-mail, the mailing list, and blog comments
    • Contrary to what some believe, the gContactSync “team” consists of one busy undergraduate Computer Engineering major only as a volunteer project (started as a paid Google Summer of Code 2008 project)
    • Anyone who wants to help program, fix bugs, or add features is welcome to do so (shoot me an e-mail).  The code has a lot of comments in it.
    • Checking out the poll results from my last post and deciding on what to do after fixing some bugs
    • Bugs
      • 20188 Card Dialog overlay is broken on trunk builds – Trying to figure out the best way to have two totally different overlays
      • 20169 Extra attributes should be disabled for read-only cards – The textboxes and types added are not disabled for read-only cards (like cards found through LDAP)
      • 20153 Groups containing contacts without email addresses break in Thunderbird – Have not figured out the best way to start this yet

November 21, 2008 Posted by | Uncategorized | , , , , | 1 Comment

gContactSync available for download

The first–and possibly only–alpha version of my extension is available for download.  You can watch a flash tutorial on basic use and download it herePlease watch the tutorial before using it or you might run into problems. Remember that it is an alpha version, so there might be bugs, and I might add more features.

You can read about the extension on my website.  I also have a mozdev project for it where you can view the source and download the extension.

Reporting Bugs and Suggesting Enhancements

I welcome any sort of feedback, enhancement request, or bug report.  If you encounter an error, please e-mail me with the details and the log file after removing any personal information from it.  There are several e-mail addresses you can use to do it.  gcontactsync at pirules.net is probably easiest.  You can also use the contact form on my blog or website.

If you encounter an error, you can find the log file in the extensions’s folder.  Read this to get to the profile directory, then go to the extensions folder and the folder named gcontactsync at pirules.net.  It is the file named log.txt, and you will probably want to remove any personal information in it before sending it.

Keep you eyes open for updates.  I’ll try to get a new version out soon after I receive some feedback.  Updates will have to be done manually, for now.

July 31, 2008 Posted by | GSOC | , , , , , | Leave a comment

More Address Book Card Attributes Synced

After a little bit of programming last night (early this morning, actually) and this afternoon, I can now sync every attribute of an Address Book card that is visible in the Edit Card Dialog and then some.

So, here is the most recent list of attributes synced from Address Book cards (excludes attributes added by my extension).  2 of the following marked with asterisks cannot be synced.

  • First*, last*, display, and nick* names
  • Primary and second Email addresses
  • Screen Name (_AimScreenName)
  • Home and work phone number
  • Preferred e-mail format*
  • Allow remote content*
  • Work phone number
  • Fax number
  • Pager
  • Mobile number
  • Home & Work: Address, Address Line 2, City, State, Zip, Country, and webpage
  • Title
  • Department*
  • Organization
  • Custom1*, Custom2*, Custom3*, Custom4*
  • Notes

Attributes marked with an asterisk (*) are not visible using Google Contacts.

July 8, 2008 Posted by | GSOC | , , , | 1 Comment

Bug 439819 – LDIF import does not include mozillahomestreet

I have been writing unit tests for importing address books and finally performed my first tests of verifying the imported data. It turns out that LDIF doesn’t import mozillahomestreet (Home Address line 1) even though it worked with mozillahomestreet2.

So I filed Bug 439819. The problem was in mailnews/addrbook/src/nsAbLDIFService.cpp. For whatever reason, the first line of the address isn’t included in all the else ifs. The patch is simple and is included in the bug report.

Status:  Patch waiting for sr.  Test waiting for bug 437556

June 17, 2008 Posted by | thunderbird | , , , , , , | Leave a comment

Bug 437556

A while ago I mentioned that I accidentally crashed Thunderbird while figuring out how to import an Address Book because I forgot to set the fieldMap before I called BeginImport. So, I filed Bug 437556: Mailnews crashes while importing an address book if a field map is required but not set. After receiving comments on previous patches, I believe the following code should stop the import, log an error, and return false if the field map isn’t set when it needs to be.

  PRBool needsFieldMap = PR_FALSE;

  if (NS_FAILED(m_pInterface->GetNeedsFieldMap(m_pLocation, &needsFieldMap)) ||
      (needsFieldMap && !m_pFieldMap)) {
    nsImportStringBundle::GetStringByID(IMPORT_ERROR_AB_NOTINITIALIZED,
                                        m_stringBundle, error);
    SetLogs(success, error, successLog, errorLog);
    *_retval = PR_FALSE;
    return NS_OK;
  }

Included in the patch is a simple unit test to check that BeginImport returns false if the fieldMap isn’t set. I am in the process of writing more unit tests for importing address books.

Here is the bug report with the patch and the test.

Update: The patch was checked in, but the original test failed on Windows builds and the new one is waiting for feedback.

June 9, 2008 Posted by | Uncategorized | , , , , | Leave a comment

Offline Support Finished

Well, I finally finished adding offline support, so it will should work well offline.  It just sets text in the status bar to let the user know the sync cannot be completed while offline.  Once the user is back online it works like normal.  Any changes made while offline are synced since the listener still adds the last modified date to the custom4 property.  My next goals are to handle errors more smoothly, get some refactoring and additional commenting, and to add support for locales in the scripts.  Once these are done it might be ready for an alpha release.  Let me know if you are interested.

I have been busy the last few days and will be busy for the next day or two because there is a wedding in my family.

May 30, 2008 Posted by | GSOC | , , , | Leave a comment

May 27, 2008

I was attempting to change my sync process today when I opened the Address Book and was greeted with 925 new Address Books.  In the end, cleaning obj directory and rebuilding fixed the problem.  Strange, but at least my extension wasn’t the direct cause of the problem.

I found and fixed an error that would sometimes occur when trying to sync after adding several new contacts.

After fixing that error, I re-defined what two cards must have in common for the sync function to consider them identical.  Since Google only allows one contact/e-mail address, it first checks if the cards have any e-mail address in common (primaryEmail or secondEmail).  If there aren’t any e-mail addresses (which is allowed in Thunderbird and Google) it checks the name.

I rewrote my old method to sync individual cards now that I use custom4 for the last modified date.  It checks to see first if both have changed (and will eventually have a preference that can ask the user what to do in this case or pick one to always update) and then checks to see which (if any) changed and updates it.  Unfortunately, updating a Thunderbird card isn’t as efficient as it could be.  I kept getting NS_NOINTERFACE errors while trying to call the modifyCard function in the address book, so I have to actually delete the card and then add the card from Google…  Updating a Google contact needs to be re-done soon, too.

In my casual tests I didn’t find any other errors.  It now works well offline and syncs every 30 minutes by default or manually.  Out of the 11 average contacts I added on one computer, every one synced perfectly with Google and with my other computer.

The only catch so far is that it has to use the card’s custom4 property to store the lastModifiedDate.

May 27, 2008 Posted by | GSOC | , , , , | Leave a comment

An overview of my extension

The extension that I am working on for GSoC synchronizes an Address Book in Mozilla Thunderbird with Google Contacts. This is going to be a quick overview of how it works so far. This post assumes some basic knowledge of the Hypertext Transfer Protocol (HTTP), Thunderbird, and programming.

Basically, my extension adds an overlay to the Thunderbird Address Book, so when the Address Book opens, it starts going.

Authentication Check

When it starts, it registers a preferences observer (so it can modify behavior as preferences change) and obtains the current preferences. I will explain preferences in more detail later. It then checks to see if there is a valid authentication token in the extension directory. If not, it shows a login prompt. When OK is pressed, it attempts to authenticate the user using Google’s Client Login method of authentication. So, it sends an HTTPS POST request to Google which then returns an authentication token that can be used repeatedly and does not expire. This token (not the username and password) is stored in a textfile in the extension directory.

Initialization

Once it gets or finds a valid authentication token, adds a listener (nsIAbListener) to the Address Book Manager (nsIAbManager), sets up the file that stores the date and time of the last synchronization, and begins the synchronization process.

Preparing for Synchronization

The sync process is the longest and the one that I change most often. Right now, it gets the Address Book first (does this every time in case it was deleted, changed, etc.). Then, it gets the time of the last sync (if any) to be used later and sends an HTTP GET request to Google using the authentication token to obtain all of the user’s contacts. Once it has these, it also obtains all contacts in the specified Address Book.

Synchronization

Right now this process is costly in terms of asymptotic analysis (or Big O notation) with a nested for loop.

The outer for loop iterates through every contact from Google while the inner for loop iterates through every unmatched card in the address book. When it finds two cards that it considers “identical,” if the current card from Google hasn’t been matched then it sets the TB card to null (so the loop knows it was ‘matched’ and calls a method that checks the cards’ last modified date to see if any of them is new since the last sync.  If so, it replaces the old card with the new one.

If the card from Google was already matched, however, that means that the card from the Address Book is a duplicate and my current implementation removes the card. In the future I may add a prompt asking the user what to do about it.

Once the inner loop is finished, if the card from Google wasn’t matched it means one of two things. Either the card is new and should be added to Thunderbird (the card’s last modified date is more recent than the last sync) or it was deleted from Thunderbird and should be removed from Google.

Once that is done, it checks any unmatched cards from Thunderbird and figures out if they should be deleted from TB or added to Google, and then does that. Due to some timing issues with HTTP Requests, there are three helper methods. There are three global arrays treated as queues for cards to delete, update, and remove.  The sync method adds cards to the arrays as necessary.  When the sync method is done, it calls the delete method, which then calls the add method, which in turn calls the update method.

Finishing up the sync

When the hard part is over, it writes the current date/time to a file (the last sync time), schedules another sync after the preferred interval and sets a synced boolean to true (which is checked by the listener to make sure it doesn’t act during synchronization)

Address Book Listener

Since TB 3 doesn’t have the lastModifiedDate property, I made my own rough version with an Address Book Listener. The address book listener updates the card’s ‘custom2’ property whenver a contact is modified or added to the Google Contacts address book. Using this approach, my extension should work offline.  I am working on adding last modified date to Thunderbird 3, so soon an address book listener and some of the booleans required by it may become unnecessary.

May 26, 2008 Posted by | GSOC | , , , , , , , | Leave a comment