Why is data migration so hard? If you’ve ever changed systems, whether it’s a CRM, LMS or Finance system – you’ll understand the pain I’m talking about here.
If not, data migration is when you take information from one system and move it to another system. Oftentimes, the data doesn’t perfectly align between the two systems, resulting in a lot of work to make it fit into the new one.
My first data migration project
I remember my first major IT project. We were implementing SAP’s Finance & Procurement modules for the first US federal agency to do so.
I was supporting the data migration and data clean-up teams as part of my job. Because the legacy systems were internally built over time, the data structure didn’t look anything like that in SAP.
The implementation partner (Accenture) would run a migration test every night, and we would look at the reports to see what information failed to load into the new trial system…again. We had millions of records to migrate. So, you can imagine the size of these reports!
Examples of common data migration issues for Not for Profits
When we received these reports, I would then work with the client’s team to try to fix it in the legacy system. Here are some common examples of data migration issues I often see with Not for Profits:
- A membership or donor ID is only 7 digits in one system, but a mandatory 9 digits in another one
- Physical addresses were not verified in the old system (by Google Maps, etc), resulting in the address format not translating properly during the migration.
- Large open text boxes were used in the legacy system, but there are reportable fields for this information in the new system.
- There are attachments stored in the old system that are not easy to move.
- Recurring payments (regular givers, membership renewals) are in a payment gateway that is not compatible with the new system.
These structural issues can easily impact the ability to migrate the data from your old system to the new one.
Lessons learned for data migration
Having been involved in numerous data migrations now, I have a few lessons that you may find useful:
- Ask the short-listed new vendors if they have experience with migrating data from your current legacy system to their system. Their lessons learned from other clients could save you a lot of pain.
- Request the data mapping template early. This will give you a sense of how different your current data structure is to the new system.
- Invest in data clean-up sooner rather than later. There’s no doubt that you have duplicate files, just like every other Not for Profit. Cleaning that data now will not only give you a more accurate view of your stakeholders or data, but it will likely save you money since many vendors (especially CRMs and email systems) will charge you by the number of contacts in the system.
- Don’t be afraid to get some help for data clean-up. If you don’t have enough internal resources to clean-up your data, look externally. Your new vendor may be able to help with this during the data migration process. However, there are also 3rd party vendors that have tools for this specific purpose. Just know that someone on your team will still need to make business rules or decisions about what data to keep, merge, delete or archive.
- Minimise the amount of data you migrate. Some of your internal stakeholders may think you need to migrate all historical data, but this is rarely the case. Instead, it’s recommended that you migrate as little as possible and to keep an archived version of your previous system until either the statute of limitations has expired for holding that data and/or you are confident that you no longer need it.
Summary:
Data migration is hard, but a necessary evil when you move from one system to another. By proactively managing it and starting the data clean-up early, you’ll hopefully make it a little easier.
I regularly help Not for Profits with strategic IT decisions, especially around IT investments. Let me know if you need some help.
Tammy Ven Dange is a former charity CEO, Association President, Not for Profit Board Member and IT Executive. Today she helps NFPs with strategic IT decisions, especially around investments.
P.S. If you found this article helpful, you might find these ones too: