An Ageing Internal System: Repair It or Rebuild It

The situation companies face today

Companies running an internal system that is more than ten years old are generally in the same position. The system still runs. There are no major outages. It is simply that a small request takes two weeks, and only one person in the company can handle it.

This state is dangerous because the problem worsens gradually. A system that stops suddenly gets a budget immediately. But a system that becomes a little slower and a little more demanding each year passes with an annual verdict of "we got through this one as well". Let enough of those years accumulate and three things arrive at once.

First, the person who understands the system retires. The undocumented business rules leave with them. Second, security support for the underlying technology ends. Once a language or database version is out of support, no patch appears even when a vulnerability is found. Third, new requirements can no longer be attached. Requests for mobile support, external service integration or data analysis are turned away with "the current system cannot do that".

By the time all three arrive together, the only option left is a full rebuild. And a full rebuild is the most expensive and most dangerous option available.

Five signals that replacement warrants consideration

If three or more of the following apply, it is time to begin considering replacement.

1. The cost of change is asymmetric. Adding a single field to a screen takes days. If the development time for a request that looks trivial to users differs little from that for a major one, the structure can already no longer absorb change.

2. Maintenance rests on one person. A system only one individual can touch turns that person's holidays and resignation into a business risk. This is a structural problem, not a personnel problem.

3. Support for the underlying technology has ended. Check the official end-of-support dates for the language runtime, framework and database in use. If they have already passed, a security incident is a matter of time.

4. Data cannot be got out. If producing the figures needed for a management decision requires someone to write queries by hand, or to open several screens and total them manually, the system is holding the data captive.

5. Workarounds are proliferating. If the area that business users manage in spreadsheets rather than in the system is widening, the system is failing to reflect actual operations. Because these shadow processes are recorded nowhere, the problem is invisible if you look only at the system.

Calculate the cost of keeping it

Replacement discussions usually stall on the question "it still runs, so why spend money". Answering that requires showing in figures that keeping the system also incurs cost. Total the following four items.

The first three are costs already being paid but booked to no account. The fourth has not yet occurred but accumulates as a probability.

The following is a hypothetical example intended to illustrate the method of calculation; actual values will differ according to each company's circumstances. With 30 change requests a year, an average wait of 10 days per request, and the opportunity cost of that wait set at 150,000 KRW per day, the delay cost alone is 45,000,000 KRW a year. Add two departments each spending five hours a week on duplicate spreadsheet management and, at a fully loaded hourly labour cost of 25,000 KRW, a further 13,000,000 KRW a year.

If the total is 58,000,000 KRW a year, over three years it is 170,000,000 KRW. Only at this point does a figure exist that can be compared with the cost of replacement. Keeping the system is not free; it is expenditure for which no invoice arrives.

Why a full rebuild is dangerous

Once replacement is decided, the first approach that comes to mind is a full rebuild: stop the old system, build the new one, and switch over in one go on a given day. It is intuitive, but it carries three risks.

The benefit is zero until the project ends. On a twelve-month rebuild, the organisation pays for eleven months and feels no improvement at all. If the business environment shifts during that period, the project comes under pressure to be cancelled.

The requirements go stale midway. The requirements settled at kick-off differ from what the business side wants a year later. Reflect the difference and the schedule slips; do not reflect it and you complete an outdated system.

Risk concentrates at the cutover. Because every function changes at once, there is no ready way back if something goes wrong on the day. And something almost always goes wrong, because the existing system invariably retains exception rules that nobody remembers.

Incremental migration as the alternative

Instead of wholesale replacement, there is an approach that leaves the existing system in place and moves functions across one at a time. The sequence is as follows.

Stage 1 — Draw the boundaries

Divide the current system into blocks of functionality. Split along operational lines such as orders, inventory, settlement and personnel, but draw the lines on the basis of who owns the data. Where several blocks write directly to the same tables, that point becomes the largest obstacle later.

When drawing boundaries, follow the flow of data rather than the organisation chart. Two departments that jointly modify the same data form a single block; conversely, a single department can be split where its data is entirely separate.

Stage 2 — Separate reads first

The safest first task is query functionality. Features that only read data, such as dashboards, statistics and reports, can be built in the new system without affecting the existing one. If they fail, the existing screens remain in place, so rolling back is easy, and business users feel the improvement immediately.

There is a further side benefit at this stage. Building query functionality exposes problems in the existing data. Duplicate customer records, dates in broken formats and rows with empty code values are all discovered here. Learning of these problems before write functionality is moved matters a great deal.

Stage 3 — Migrate write functionality

Once queries have validated the approach, move the entry and edit functions. Two systems will handle the same data for a period, so designate one side as the source of truth. Leave both editable and inconsistencies arise, and tracking them down takes more time than the migration itself.

It is safest to sequence the migration starting with functions of low usage and narrow blast radius. But moving functions nobody uses provides no validation, so a function that is genuinely used yet could stand a day's outage makes the best starting point.

Stage 4 — Shrink the existing system

Remove migrated functions from the existing system. Leave them and some business users will carry on with the old screens, and you will end up maintaining two systems permanently. Deferring this stage is the most common reason incremental migration fails.

If removal is difficult, at minimum block access and switch the function to read-only. Then fix a date for taking it down entirely. A decommissioning plan without a date is not executed.

Which approach to take

A full rebuild is not always wrong. Where the scope is small, the business rules are documented, and a few hours of downtime are acceptable, changing everything at once is faster and cheaper. The deciding criterion is not the age of the system but where the rules are recorded.

What actually causes problems in data migration

Schedules slip because of data rather than feature development. Older systems accumulate conditions such as these.

• The same customer registered several times under different spellings
• Dates, telephone numbers and business registration numbers formatted differently by period
• Rows with mandatory fields left empty
• Data still referencing code values that no longer exist
• Rows merely flagged as deleted but in fact still present

Discover these problems during the migration stage and the schedule will certainly slip. Investigate in advance, before work begins, and decide first what will be cleaned up and what will be discarded. Attempt to clean all historical data perfectly and the migration itself will never finish. Cleaning only the last few years and retaining the rest as read-only archive is often the realistic course.

An opportunity to redesign security and access rights

Replacement is also a rare opportunity to put the security regime in order. Older systems generally have loose permission boundaries and are run with most staff able to see more data than they need. Settle the following items as part of the migration.

Minimisation of access rights. Separate visibility so that each role sees only the data it needs. Carry the existing system's permission structure across unchanged and the old problems come across with it.

Storage location and retention period for personal data. Document which personal data is stored where and when it is destroyed. Where the move is to the cloud, the country in which data is stored must also be confirmed.

Retention of processing records. Record who changed what and when. Older systems frequently lack these records, leaving no way to trace the cause when something goes wrong.

How to persuade the executive

Replacement budgets are rarely approved on technical arguments. Explanations such as "the architecture is outdated" or "technical support has ended" do not register as urgent with the person signing off. Recast them as the following three.

The money leaking now. The total of the delay, workaround and error costs calculated above. The essential point is that this is already being spent even without replacement.

What cannot be done. Compile a list of the requests turned down over the past year on the grounds that the current system cannot support them. Where any of them connect to revenue or customer attrition, put those first. Lost opportunity is a stronger argument than maintenance cost.

The worst case. Estimate the duration and cost of recovery if the sole maintainer resigns, or if an incident occurs in an out-of-support component. Low probability still moves a decision when the scale is large.

Having presented all three, request approval for the first stage of the incremental migration only. Ask for the entire budget at once and the review period lengthens, while the situation deteriorates further in the meantime.

Setting the schedule and the staffing

Schedule is what most often goes wrong in an incremental migration. Account for three things in advance.

Put the business side's time into the schedule. The scarcest resource in a migration project is not developers but business users who know the rules. They participate while carrying their day jobs, so unless the available hours are agreed in advance, the schedule slips at every validation step.

Include the parallel operation period in the calculation. Even after a function has been moved, both systems must run together for a time. Operational load actually increases during that period. Fail to reflect this in the schedule and budget and you will be short of people at the final stage.

Allow separate time for data clean-up. The data integrity problems discussed above can proceed in parallel with development, but they require their own time and their own owner. Bury them inside the development schedule and they will certainly slip.

What to check when working with an external supplier

It is often impractical to carry out a migration with internal staff alone. If you are considering an external supplier, confirm the following before signing.

Does the deliverable include documentation. Receive only code and the same problem recurs a few years later. Business rule specifications, data structure descriptions and operating procedures must all be included in the deliverables.

Is the handover arrangement defined for after the migration. Internal staff must be able to operate the system once the build is complete. Specify the handover period and the scope of training in the contract.

Can the contract be split by stage. Contract for the whole thing at once and it becomes difficult to change direction midway. A structure in which the first stage is performed and the remainder decided afterwards is safer for both parties.

Are the rights to our data clear. If real data is used during development, document what data moves where and how it is destroyed on completion.

What to prepare before starting

Whichever approach is taken, secure the following three things before work begins.

Documentation of the current business rules. Rules that exist only in code are invariably lost during migration. A perfect specification is not required, but exception handling and approval conditions must at least be written down.

A data integrity review. Survey the current state against the items set out above and decide the scope of the clean-up.

A rollback plan. For each stage, define how to reverse it if something goes wrong. A stage that cannot be reversed is by that fact too large a stage, and is a signal that it must be broken down further.

Summary

The cost of an ageing system appears in the form of delay and dependency rather than outages. That is why it is recognised late.

• Review the five signals: cost of change, concentration of expertise, end of technical support, data accessibility, and workarounds.
• Calculate the cost of keeping the system across four items — delay, workarounds, errors and risk — to produce a figure comparable with the cost of replacement.
• If the rules survive only in the code, a full rebuild is dangerous.
• Move query functionality first, and always remove migrated functions from the existing system.
• Investigate data integrity before work begins and fix the scope of clean-up in advance.
• Take the opportunity to redesign access rights and the personal data retention policy at the same time.

More important than whether to replace is not deferring the decision. If three of the five signals apply, beginning the review this year is cheaper than a full rebuild next year.