Schedule data displaying incorrectly
Incident Report for OpenTrainTimes
Postmortem

We have to laugh about this - it’s the perfect case of a single missing character screwing everything up.

On the evening of Wednesday 10th November, we had to restore schedule data to our database server after a problem importing it. These things happen from time to time, and there was due to be no issue. However, the process we carry out is to re-import the data on another server and then ship two SQL tables of data over to the database server. Here’s where the fun starts.

When exporting the data, we have to specify the names of the tables on the command line - ‘schedules’ and ‘schedule_locations’ in this case. However, we’d mis-typed them as ‘schedule’ and ‘schedule_locations’, meaning that only one of the tables was exported, not the second. We failed to spot the problem until after the data was restored, when that night’s timetable file was imported - creating a set of schedules for Thursday 11th November which were also wrong.

We ended up with a schedule database where all of the timing points of schedules were intact and grouped correctly, but the ‘header’ records were pointing at the wrong set of timing points. This is why you saw trains at stations destined for unfamiliar locations.

Once we’d identified the problem and were sure of the cause, we re-extracted and imported the correct set of schedule data, and baby-sat the site to validate everything was fine - which it is now.

Posted Nov 12, 2021 - 09:08 GMT

Resolved
This incident has been resolved.
Posted Nov 11, 2021 - 21:35 GMT
Monitoring
We have restored the database and are waiting for a backlog of messages to be replayed
Posted Nov 11, 2021 - 15:41 GMT
Update
We have tracked the problem down to a single missing character in one of the scripts that manages our databases. A fix will be in place in the next two hours.
Posted Nov 11, 2021 - 13:50 GMT
Update
We are continuing to work on a fix for this issue.
Posted Nov 11, 2021 - 08:57 GMT
Identified
A problem with our database server means trains are showing up at incorrect stations. We aim to have this fixed in the coming hours.
Posted Nov 11, 2021 - 08:57 GMT
This incident affected: Website (Train movement data).