Data Migration: CRM's Overlooked Process

Simon's picture

What's the most important element of any CRM?

Is it the functionality?

Is it the ease of use?

Is it the flexibility?

Well, yes all of those are important factors but there's one element of a CRM system that is essential. You cannot have a CRM system without it.

I am reminded of the old saying:

"If a tree falls in a forest and no one is around to hear it, does it make a sound?".

So what is this most important of all CRM system elements?

DATA!!

Stating the Obvious

Now to some this may be somewhat obvious.

Without data, the CRM system becomes the proverbial tree.

"If there's no data in the CRM system, is it really a CRM?"
However, as obvious as this is to most people, it quite often doesn't cross their minds that to get their existing data into a new CRM system they are going to have to get their hands dirty.

Let's be honest, when selecting a new CRM the data migration is rarely considered up front by the client as it's not exciting, they want to see new technologies and commercial advantages, not think about all the old data they already have.

Data migration is often seen as just a necessary and costly evil and the effort required to successfully migrate data between CRM systems is usually vastly underestimated by the client.

"Well, it's just shifting data from one place to another, you must do this all the time so it must be simple right?"

Well no, not really.

As a technology company we deal with companies from all sectors and of varying size but one thing is universal, and that is this:

No one's data is universal.

Let me repeat that...

NO ONE'S DATA IS UNIVERSAL!!

Every company has their intricacies and every source system we migrate from has it's nuances.

Some, such as Salesforce are a relatively straight match to SugarCRM in terms of data structure.

Others, like Goldmine, seem to have been written by a masochistic self-harmer hell-bent on storing data in a manner that's as convoluted as possible!!

Seriously, who stores an email address across two different fields none of which have the term "email" in their names, in a table that also uses those exact same fields to store website addresses, contacts and various other items like organisational trees!!??

I digress.

Data Migration Considerations

As I've already mentioned, a lot of clients don't realise just how influential a successful data migration can be on the user acceptance and ROI of their new CRM.

So what should they be thinking about before they start?

Quality AND Quantity

Two big elements to consider are both the Quality of the existing data and the Quantity of it.

A lot of companies are blissfully unaware of the issues within their existing data.

Fields containing inconsistent and often incorrect data or even data in the wrong place are common place.

This is especially true when migrating from multiple disparate systems into a single Sugar CRM instance.

Sugar CRM also has it's own way of doing certain things, such as multi-select fields, very few traditional style foreign keys and the way email contents are stored across multiple tables, which means that quality becomes even more important.

Quantity also has it's part in data migration to play though.

If the migration graphs are not planned with the relevant data volumes in mind, it can bring a server to it's knees.

There are different ways to join data as part of a migration each with it's own pro's and cons.

Do you query the Data server for each record in turn or do you pre-load the related data and join in batches within the migration graphs? One batters the data server and one eats up memory on the server processing the graph.

Are the fields within Sugar large enough to handle the depth of the incoming data?

Quality of data and volume both need considering before, during and often after a migration.

Data De-duplication

The next major consideration of data migration is de-duplication of the data.

Both within each of the individual source system and across the systems if being merged.

Here's a typical conversation about the de-duplication of data across multiple systems...

Enable: Do you need us to de-duplicate the data in your systems?
Client: Ah... yes, we'll need to remove duplicates in accounts and contacts modules.
Enable: Fine that's no problem, do you know the criteria we should use to do this?
Client: .....sorry what do you mean?

Simply put, a lot of clients do not consider this process and that's no big surprise as it's not an approach to CRM that most people take.

To successfully migrate and merge data though we need to consider the criteria on which we should compare records and not just highlight, but automatically select the master records where duplicates arise. Migration graphs are fully automated, there's no human intervention to inject business logic, we have to build it into the migration process.

Data Decay OR "I had a great idea for our CRM a few years ago"

Over time data can decay and lose it's relevancy.

Some people are very good at keeping their data fresh and up to date, others not so much.

Often a source system can contain data that's over 10 years old so just how important is this data to the operation of the company today?

Also over the years the way a CRM system is used can change. Fields can become obsolete, new fields can be created and methods can switch back and forth.

All this can lead to data in the source systems that is simply not needed in the new Sugar CRM system.

In turn this means it can be excluded when building the graphs, saving time in generating the metadata and data migration mappings.

User Association

The final biggest consideration, and the first thing we look to migrate, are the users in the new Sugar CRM System.

OK so what's the big deal there?

If you are unfamiliar with Sugar you may not know that every record has to have an assigned user, as well as a created by user, and modified by user associated with it.

When migrating data from source systems some of these associations can already exist, however, what if those users are no longer with the company? What if they do not require a user in the new Sugar CRM system?

How exactly do you map data to non-existant users?

Well, there are a number of approaches depending on what the client requires, but knowing the exact user base up front along with defining rules on how existing data is to be assigned to users is vital.

Users are migrated first as that way we can associate all the other records with the correct user as they are imported.

Involving the Client in Data Migration

So take all of these factors into account and maybe you can start to see why data migration really shouldn't be treated as a burden or non-essential part of investing in a new CRM.

It can take days if not weeks to migrate existing data into Sugar CRM.

No, that's not an exaggeration! Days is pretty much guaranteed, weeks is not uncommon.

No one wants to log into their shiny new Sugar CRM system to find data that's broken, missing or in the wrong place, which means that without a strong lead at the client, we need to be something akin to a mind reading, data-analyst-Jedi.

The simple fact is, that when it comes to migrating the data from existing systems, no one knows the data like the client does.
People sitting around a table forming a puzzle togetherNo matter how many times we have done a migration from a particular system there is no better way to get it right than for the client to be involved from the outset.

If a client can invest time in the early stages of a migration it can often save a lot of time further down the line when a migration is more mature and therefore takes more effort to modify.

Having a single person at the client end who is willing to invest their time in communication, investigation and testing is a god send, and the most successful migrations we have completed have all had this in common.

Very often though clients expect to hand their data over to us and see a Sugar CRM system containing their data within a matter of days with no further involvement.

They don't always realise that their data is the most important part of their existing and new CRM systems and needs treating as such.

Data Migration Tools

At Enable Technologies we have performed hundreds of migrations from a number of different systems into Sugar CRM, so we know what we're doing.

We have a whole plethora of tools to help us in this task and we're always looking to use technology to improve the process.

We use ETL (Extract, Transform and Load) software to create re-runnable data mappings and transformation graphs which allow us to run an import over and over again, tweaking the process until it's just right.

We use Google Docs to create field mapping and data structure documents for ourselves and our clients, and Trello to create collaborative boards to define migration rules and highlight data issues to be dealt with as part of the testing phase of a migration.

We've also written our own functions within the ETL software and our own php ETL libraries to help us bend existing data into shape for continual automated Sugar CRM data imports where required.

The Bottom Line

With all of these tools at our disposal we can always make sure a clients data is migrated into sugar CRM in the right way.

Being an Elite partner with an intimate knowledge of Sugar CRM, gives us a real advantage when it comes to migrating data from other systems and ensuring our clients have a usable and easily adopted Sugar CRM system at the end of the process.

That said, our client's involvement is still the best tool in our arsenal, hands down.

If you want to read a more in depth white paper regarding the various pitfalls and processes that should be considered in a data migration you could do worse than read this from Oracle: Successful Data Migration

Simon's picture
About Simon

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd> <img>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.