At some point when considering implementing a Laboratory Information Management System (LIMS) your thoughts may turn to your data. You might be new to LIMS or extending an existing system to encompass another team or stream of work. Either way, the data you’ll need isn’t in there yet, but you’ve already got it in your old system or in spreadsheets. It should be easy enough for IT to just transfer it, shouldn’t it? Data migration is a notoriously difficult process to get right. The availability of easy-to-use tools with your new system is certainly important but it’s far from the only issue, or even the biggest!
If we talk a little more about the whole process it should become clear that getting the data into your system is only one step of many. That process, if done well, will have a massive impact on the ongoing success of your Laboratory Information Management System (LIMS).
One-off LIMS data migration or regular data transfers?
First, you need to know whether you want to carry out a single data migration process into LIMS. Are you trying to get started with a new system?
Do you also expect to do regular data imports? You might be regularly sent samples with a “manifest” spreadsheet detailing what’s in the shipment. If that’s the case, then you’ll want to import the data rather than manually entering it yourself. Remember, all manual entry is at risk of human error and, let’s face it, manual entry can be a bind!
So, if your new LIMS has an import tool that lets you save an import profile and run it whenever you need that’s a real plus – “Load it up and go” is the way to go. Recycle/Reuse is the modern mantra.
Finally, if it’s a data import tool that supports your own customised data structures – extra fields and tables – then that’s a real bonus. It also means that your investment is protected as your system grows.
What’s involved?
We’ll park the regular transfers for a moment and consider the initial data migration into LIMS scenario. The process is often known as “ETL” to IT professionals. That’s “Extract, Transform and Load”.
Basically, this means:
- getting the data out of the old system – “Extract”,
- knocking your data into shape – “Transform”
- and, finally, getting it into your new system – “Load”.
So, who does what? It’s all to do with data and computer systems, isn’t it? That means it’s all a job for IT, yes?
No. The IT team is highly trained at managing servers and systems and, sometimes, formatting data files. But – they don’t understand your data. Only you do. They can’t make decisions about which values in your old system should be translated into which other values in your new system. That’s your job. IT may be able to help you apply the changes but all decisions about the data are your responsibility. And doing it well is a real challenge that takes time!
For the regular transfers the Extract and Transform steps have been taken care of by someone else – Phew! So, back to the more difficult data take-on…
Data quality
Now, we’ve an idea of the end-to-end process but usually the most difficult step is the one in the middle – Transforming the data. That’s basically the “knocking it into shape” phase and, unfortunately, that’s the one that’s in your lap.
Don’t question whether it’s worth cleaning your data. It is. Always. And clean your data before you load it – you’ll never do it once it’s in!
Good data quality maximises Return on Investment (ROI) and user buy-in leading to a successful implementation. Conversely, poor data can kill a project dead. If users lose confidence in what the new system is telling them they will immediately fall back to workarounds and those pesky spreadsheets. Your system needs trust from day one and that means the data needs to be squeaky clean.
So, how do you go about turning the questionable data that was in your old system/spreadsheets into fully-compliant carefully classified robust data? Well, it involves a lot of hard work.
Are you on your own?
You shouldn’t be. One piece of valuable advice here – try to use your LIMS supplier’s hard-won experience. Chances are they’ve done this many times! And if they offer a data migration support package then give it serious consideration.
Some suppliers believe in working with you as a team to share knowledge and make things easier. They may offer consultancy services as part of the package implementation. This gives you access to their expertise and good advice about what to do and, just as importantly, what not to do. You need the reassurance that your hard work is focussed on the things that will bring the biggest benefit for your team.
Guidance on ‘best practice’ when migrating data into LIMS
Look for advice on the steps involved in the whole ETL process. Guidance on tools and techniques to use when processing the old data can be invaluable. Let’s face it, most legacy data is in pretty poor shape and less structured than you need. You need to know the steps to take to turn all of those misspellings, acronyms, alternative terms and missing data into something that your new users can rely on in the new LIMS. And don’t forget that the process needs to be repeated at least once as you do a test extract/transfer/load and then a live one. So, take copious notes of all steps taken. Your supplier should go through all this and even provide you with template documents, meeting agendas and process flows to help you.
How do you know you’ve set it up correctly?
Firstly, the import tool should perform some data validation prior to import. This may be as simple as checking that the data matches the target data type and that no letters are going into a numeric field. It could also check for blanks going into mandatory fields and might even check that all values to be imported into each field match the fields associated picklist. All of these checks should really have been done manually for the one-off data migration but it’s still nice to have it double-checked. Where this really comes into its own is for regular imports of data prepared by others. If a customer or partner doesn’t follow their SOP properly it shouldn’t fill your LIMS with poor quality data that you have to then fix.
Importantly, the process should include a test cycle where the data is imported into a Staging system for testing and sign-off before the final import goes into your Live Production system. Your users should then carefully check the data in Production before signing it off and starting work with your shiny new system.
Impact on user perception
The whole data-migration-check-signoff process gives the users confidence in the quality of the data and in their initial use of the LIMS system. This means that data validation prior to import is very important indeed. Good quality data maximises user buy-in, helps the new operating procedures to bed in and maximises the system’s Return on Investment (ROI).
A final thought on LIMS data migration and import processes and services
Successfully data migration between systems is fundamental to a successful roll-out of a new LIMS system. Successful means allowing users to start with clean, complete and reliable data. For that reason, data validation and cleansing are vital and deserve to have time and energy applied to them. This is always the part of an implementation where people always underestimate the required time.
Choose your new LIMS vendor carefully. The software might look brilliant but if the vendor doesn’t value its role as a partner in getting your data successfully into loaded then you could seriously risk your project’s success.
Good data means everything! And your vendor should recognise that.
To read more go to ‘Things to consider when choosing a Laboratory Information Management System (LIMS)‘.
Catch up with the rest of the series so far
- Series introduction – [Just in time for the pantomime season – it’s the oh no it isn’t! Not another] ‘Guide to choosing a LIMS’
- Part one – Avoid going out on a limb* by clearly identifying your acceptance criteria when choosing a LIMS (* pun intended!)
- Part two – Don’t believe the hype? Or do? How to get the most from LIMS recommendations
- Part three – Believing that tomorrow can be better than today. How to implement best-practice and future-proof your LIMS
- Part four – Protecting what’s important – choosing a LIMS that protects your sensitive information
- Part five – What’s in a name? The difference between a Laboratory Information Management System (LIMS) and a Sample Management System
Comments are closed.