Excavation, Innovation and Transformation – decommissioning a legacy platform at the Department for Education
Where we used to have emergencies, now we have options
“World-first’ digital evidence system for policing”
“£4m released to fund AI in schools”
“New digital campus for 7 000 government officials”
To go by policy announcements and funding decisions, the UK civil service is clearly at the technological cutting-edge of recent innovation.
For those closest to the day-to-day realities of public administration, such announcements might come as a surprise. In many parts of the civil service, creaking code infrastructures and large, poorly-understood legacy estates are the norm. Keeping these running and patched day-to-day can consume most of or even all the budget and technical capacity a department has at its disposal – with nothing left over for developing a more maintainable solution, let alone more advanced technologies like machine learning or AI.
Confronting KIM at the DfE
This was the situation we faced when we were first contracted by the Department for Education (DfE) to redesign their digital service that supports schools to transition to academies in England. Managed by the Regional Services Division (RSD), the systems supporting the service were ramshackle, consuming large amounts of time and money to maintain, while scoring rock-bottom on usability assessments.
In one way this shouldn’t have been surprising. At the time, dxw was only the latest in a long line of agencies brought in to improve the Academisation service. Each had created some form of bespoke solution, none of which quite worked, and none of which integrated well with the others.
Digging down through these various abandoned legacy items furthermore revealed that all of them had some kind of dependency on an ailing piece of software – basically a Microsoft database product with some custom functionality added – termed the Knowledge and Information Management (KIM) system. KIM was the very definition of unmaintainable: long out of Microsoft support, the DfE also no longer had any engineers capable of updating its bespoke functions.
On one hand, it was nice to discover that in a way, we only had one problem to solve: replace KIM. On the other: as we were to find out, it was a big problem.
The surprising power of legacy tech
Ultimately, what makes legacy tech hard to replace is: it works. It might not work well. It might not do everything you need it to. It might impose crippling overheads. But it works. If it didn’t, it wouldn’t have lasted so long. And, in the case of KIM, it wouldn’t have come to underpin so much of the rest of the technical estate. Dig deep enough into almost any application within RSD, and you would eventually hit KIM at the bedrock. In addition, almost all staff also input data into it directly.
Fortunately, the DfE is not the first organisation to confront this kind of problem, and there are several well-explored approaches to solving it. The one eventually adopted is known as the Strangler Fig pattern. In this design pattern, individual items of legacy functionality are replaced, until coverage is complete and the legacy system can be turned off.
Fig-strangling is a team sport
One of the nice things about the Strangler Fig pattern is that it lets you avoid ‘big-bang’ system switchovers – and the risks associated with these. You replace functionality as and when you can, allowing upgrades to be accomplished in small increments.
dxw’s approach to project delivery and team formation made delivering these increments relatively easy. Multidisciplinary teams – user researchers, designers, delivery leads, and developers – worked with DfE end users to determine and prioritise the KIM functions that needed to be replaced. This, along with our Agile approach to development, meant we were able to deliver value to RSD from the first quarter of the contract, providing enhanced user experience and technical reliability almost immediately.
As our understanding of the RSD technical estate grew, we realised we needed to make more radical changes to team structure. Previously, there had been no single group responsible for infrastructure.The creation of a centralised Common Architecture Team helped us to standardise, simplify, and integrate the different applications previously unified only by their shared dependence on KIM. Unifying our architectural, hosting, and data infrastructure quickly led to obvious benefits – not just cost savings on cloud infrastructure, but reduced maintenance overheads and much-increased transparency and reliability throughout the technical stack. It also meant that we could adapt quickly to a last-minute discovery of several sets of users relying on KIM for data reporting purposes: not only have we been able to migrate them off dependence on KIM, we are also now in a position to offer them enhanced and simplified reporting capabilities through PowerBI.
Looking ahead
A little over two years into our contract with the DfE, it’s safe to say KIM has now been tamed. Direct end-user reliance on the system has ended, and we’re finally in a position simply to turn it off. This is a huge and long-anticipated win.
But much more has been achieved than the important but purely-negative step of shutting down a legacy system past its sell-by date. Along the way we’ve created usable and accessible applications for DfE staff. We’ve created a standardised approach to managing deployments that’s been influential in the DfE even beyond the RSD. We’ve improved and simplified reporting. And, through the development of a robust data architecture, we’ve laid the necessary foundations for improving data quality throughout the Department.
What we do with this improved and reliable infrastructure remains to be seen. We’re now in a position to start considering advanced data analytics capabilities, and perhaps machine learning. We can start exploring AI use-cases. But whatever the future holds, the point is that assembling multidisciplinary teams, engaging with stakeholders, and delivering at pace alongside strong technical capability have been key in this process. Where we used to have emergencies, now we have options.
To find out more about our work or continue the conversation, please drop us a line.