A look behind the scenes of a major data archiving project with AVIVA
We chat to Andrew Crouch from AVIVA about his experiences of implementing a data archiving solution:
Can you start by telling us a little about how the data archiving project first came about?
Back in 2000 the NUI (Norwich Union Insurance) claims and workflow systems had been running on the current platform for 7 years and policy for 2 years. The Capacity Management team began to have concerns about the levels of data growth and the cost of a serious system failure to a rapidly increasing user base. This kick-started the project with problem definition and research of options being undertaken.
You say that the levels of data within NUI were growing at that time, what were the main drivers for that growth?
Data from legacy systems was converting to the new platform, this was also the period that Norwich Union was merging with CGU. The user base was increasing and with it the need for greater availability. The whole business was starting to move to where it is today.
What sort of pressures were IT facing from around the business with regards to the growing data volumes?
The business weren’t experiencing any issues as a result of the increased data at that time. The Capacity Management team could see problems on the horizon so the pressure was convincing the business that action was required. We were a bit ahead of the game.
Excellent, so you weren’t facing pressure from the business, this must have made your lives easier whilst you were researching the solutions.
Not really because the pressure was reversed in a way. As an IT department we needed a business case to justify the need to run the project. The fact that the business were not experiencing any issues relating to data growth at that time meant that we faced some resistance.
So how were you able to demonstrate to the business that a data archiving strategy was needed?
The two main drivers were the Data Protection Act and system recovery times.
The DPA convinced the business that we needed to implement an archiving strategy. The business was fully aware of the need to comply and potential costs and damage to the brand if Norwich Union were to be found in breach. DPA compliance was high profile and resulted in a separate group wide project being set up to deal with the issues.
The other major factor was system recovery times. Due to the large amounts of data we were dealing with we concluded that should we suffer a major system fail we would not be able to keep within our SLA for recovery. Given the increased user base mentioned earlier, any outage would mean the costs incurred would be far greater than previously.
When presented with these two strong arguments the business agreed and the project began in earnest.
How did you go about choosing which tool you would use?
We worked in conjunction with our partner CSC. They did a lot of the initial research into the market for us and came back with 3 options- IBM, Compuware and Princeton Softech. (This was prior to IBM’s acquisition of Princeton Softech).
Norwich Union used a standard checklist for software acquisition which enabled us to list functional and non functional requirements, this helped us to measure the products against one another. It was clear very early on that the Optim (formerly Princeton Softech) solution was far superior in terms of functionality. The selection process only took around 6 weeks.
So, at this stage were you ready to begin implementation?
We should have been but we came across some resistance during this process as the question of “build vs buy” was raised. A number of alternative ideas and theories for internal solutions were put forward but this was from a support area with limited resource and not a development area, it was impractical to have them tied up in a build project like this which would take up a considerable amount of their time.
It took a significant amount of time to come to an agreement that the best solution would be to buy Optim rather than try and build something bespoke.
Once you had reached an agreement on this how did you find the process of implementing Optim?
We had a big advantage in that we held existing ERD’s (Entity Relationship Diagrams) for the data. This enabled us to build the relationship model in Optim in just a couple of days. The business rules in some cases were quite complicated though, particularly around retention of data for claims with injury.
The solution was implemented into production in 2002 and we began with simple criteria to get us up and running. We then moved onto more complicated business rules which needed supporting batch suites built to handle the diverse criteria held across multiple DB2 tables and objects. The batch suites did all the work in identifying cases that qualified for archiving and Optim did the rest.
You mention the business rules; I guess you had to involve quite a lot of other departments in this project?
Indeed, this wasn’t a project which sat just within IT. We had to consult with Legal, Business Users, Data Owners and users and various teams within the IT department.
To give you an idea of the scale, we had sign offs from over 50 impacted areas who had an interest in the data (or copies of the data)
A few years down the line, what have been the main outcomes?
There have been three main outcomes.
- Recovery times – this had been one of our key objectives. We’re now confident that recovery times will be kept within SLAs now that we have less data in production.
- Data Protection Act – The implementation of data archiving means we’re now happy that we’re meeting DPA regulations.
- Performance – We’re now comfortably managing system performance. Although, as with most things there is always room for improvement!
What would your “top tips” be for anyone embarking on a data archiving project?
My top tips would be:
- Make sure you have buy in from everyone with an interest, the “build vs buy” issue we hit caused delay for many months.
- Know the data model & database, with the up to date ERD’s and some knowledge of the database we built the model in Optim in a couple of days. If you don’t have this you could be looking at a significant amount of extra development time although I understand there are moves to automate this in Optim.
- The business rules for what you can archive may be more complex than you think, we have to code in black and white but the business may present a lot of grey.
- You need to know who is querying the data or extracting the data or working on copies of the data. They will all want a say on what gets archived.