On Friday 15th April 2016 the school email systems, the internet and SIMS went down; we then systematically lost access to the servers throughout the day. IT confirmed early on that we had been hit by a virus and they were working on clearing it. The virus could be seen by a name on the files, and all staff were advised to save unaffected files as soon as possible. As an aside I mentioned to a colleague that so long as it wasn’t Ransomware we were fine. This proved to be prophetic in all the wrong ways. By the end of the day IT had confirmed that it was Ransomware and all the servers were affected and encrypted. The attack had come through the new BT server which had passwords that were not as strong as the rest of the network, and it had come through a brute force assault – a password cracker run for a month against BT lines.
No chance of a quick recovery
On Monday, it was clear that there were significant recovery problems, as the server backups saved into the servers and as such were compromised as well. By this point we were copying the paper standby fire registers to use for am/pm registration, and this situation was to continue for the next 6 weeks. All of the servers were unrecoverable and had to be rebuilt, Capita came in on contract to assist. One external backup from the previous year had been located and could be reinstalled, but the date for it was early August 2015. As a consequence it would be missing any data for the current academic year, results data and the year’s timetable.
Over the next few weeks the servers were rebuilt, SIMS reinstalled and the only backup uploaded. This gave a draft timetable that was too unstable to amend, no attendance data and all current calculations in SIMS for the first year of the new GCSE grade scores were gone. Fortunately, the student and staff drives and the work in them could be recovered to a degree. The Local Authority was initially unable to return the attendance data, and we ended up having to pay for them to reverse engineer the upload. This did not come back until the start of the next academic year. The exams data could be reimported but as we use SISRA Analytics, the collection data was secure on their website and could be recovered and dropped back into our systems.
The effect on exams and data
During the six week period this happened in, life was not pleasant in the student office, as all our systems were down, and very little could be done to rebuild the missing SIMS calculations, input the growing pile of paper registers for all year groups, or check the behaviour logs. The exams officer had to recreate approximately 4,500 individual exam entries and special circumstances, as our records for the GCSE and A level exams no longer existed. Curriculum staff were informed SIMS was back up and running after about five weeks, though this statement omitted the level of damage to records and led to some acrimonious exchanges when asked to provide attendance or behaviour data.
The timetable was frankly shot, and could not be amended in any way for the rest of the academic year without causing its total collapse. Every calculation needed in SIMS had to be rebuilt from creating grade sets onwards. All of that year’s data had to be reimported from SISRA Analytics, and at the same time the final data collection and reports for the year collated and disseminated. Timetable migration and the results day preparation also had to be completed before the end of July.
The damage sustained continued to be felt into the next academic year with errors being found in SIMS, calculations, and processes. Initial repairs had focused on those areas in use and as new events rolled round more compromised systems were found. The class migration completed in July failed on 1st September leading to all groups having to be recreated by hand from records.
What did we learn?
A number of lessons came from this process. External backups are essential, as are strong passwords and not opening any suspicious emails. Regularly saving and exporting essential data, documents and reports should be done. Likewise, class membership can be exported and saved, which provides an extra level of backup when the year dates roll over. Finally, cloud based or externally hosted systems like SISRA Analytics are fantastic, because you don’t lose all your data!
Nathan Page, Data Manager, The Ferrers School