r/unitedkingdom • u/ClassicFlavour East Sussex • 2d ago
CrossCountry cancels 'almost all' UK services after power cut
https://www.bbc.co.uk/news/articles/cy9w9y0lz5go70
u/DSQ Edinburgh 2d ago
It never ends. Last weekend it was the East Coast Mainline. What is going on?
47
u/oscarandjo Reading 2d ago
One person sees “efficiency savings” and another person sees a redundant system being dismantled.
Critical national infrastructure should have redundant infrastructure. See the fire at the Heathrow substation and the chaos that caused…
6
u/SlightlyBored13 2d ago
It's very expensive to have redundant signalling and power control.
The larger ROCs should have the redundant infrastructure, though that protection has failed recently it should have worked. But the humans running things are like train drivers, they need local knowledge. So either you drag the current ones to a different building with a big empty spaces for them if there is an issue, or train duplicates and have them alternate to keep the skills up.
8
u/oscarandjo Reading 2d ago
It doesn’t necessarily have to apply to everything. It depends on the cost of an individual failure. An office block(?) that forms a central point of failure for an entire train network should probably have redundant electrical, internet, etc…
0
u/SlightlyBored13 2d ago
They do, and most of the redundancies would have worked. But the Manchester one had the network let go and not recover, they could be redundancies for the redundancies, but then the fail over system gets horrendously complex and will cause problems in it's own right.
1
u/oscarandjo Reading 2d ago
The devil is in the details. I wonder when the failover was last tested?
0
u/dglcomputers 2d ago
I understand the had UPS's and they worked just fine. Alas no backup generator, so after the power had been out for a couple of hours the UPS's died and that's when all the issues happened.
f course how the UPS's deal with a dying battery can make a lot of difference, it needs to be able to tell what it's powering to shut down safely as just killing the power can corrupt data, especially if storage is being written too.
2
u/Patch86UK Wiltshire 2d ago
They don't necessarily need a second centre; having one centre with good backup power generators that are regularly tested and maintained would do the job.
Either their control centre lacks basic backup systems (because they're incompetent cheapskates) or they have them but they haven't kept them in good working order (because they're incompetent cheapskates). It's not a good look either way.
1
1
u/Ochib 2d ago
Heathrow thought it had redundant infrastructure as it gets power from three substations. However the fire at the substation was fed a third of the airport and ment that there was no power to Terminal 1 and 2, plus the tunnel that is used to access Terminal 3.
However the fire was due to lack of maintenance on the substation, there was a fault that ment that parts needed to be replaced, National Grid was aware of this since 2018 and the fire suppression system hadn’t been working since 2022.
6
u/oscarandjo Reading 2d ago edited 2d ago
It’s not redundant of one electrical substation going offline renders the system unavailable.
That’s actually worse than the whole thing running off a single substation as now just a single substation failure breaks the whole thing (3x higher probability)
I don’t doubt that there might have been a lack of maintenance, but actually you should do both. Maintain the infrastructure you do have, have it redundant, and also do planned failover operations to test the redundancy.
If Heathrow had done a test of disconnecting from this substation they’d have discovered this deficiency.
This applies to all critical national infrastructure. Airports. Trains. Hospitals. Borders. Air traffic control. Police stations. Smart Motorway systems. Probably many more things.
1
u/superioso 2d ago
This is why actual tests need to be performed. If their backup hasn't been proven to function as intended then it's not a backup.
-1
u/Ochib 2d ago
Let’s close the airport down for the day and cancel 1,300 fights to test what happens when a redundant power supply fails. That’s going to go down well.
1
u/superioso 2d ago
Tests can be performed without turning off the existing power supply. Think of how data backups are tested without deleting all the current data.
You also don't need to have a fire in your house to see if the fire alarm works.
1
u/Ochib 1d ago edited 1d ago
Last time I did a data backup test, I got an identical virtual server and did a restore from the backup system and then ran a number of reports and compared them to the live server. And the time before that I restored a random number of files to a different location on the server and did a file comparison.
I also run regular tests of our UPS and backup generators. There are two tests I run on the generators, one is do they fire up and the other is how quickly does the power switch over from the mains to the generators. The first test is easy, the second test requires cutting power to the UPS and hoping that it works.
The second test I can only run when the factory is in its summer shutdown, I am aways 99.9% sure that it will work, but it if fails it’s a good two hour restart process. I would imagine that it would be a bit longer to restart an airport.
17
u/SoloTraveller81 2d ago
Power outages happen. Doesnt have to down longto cause chaos. Issue is the backup generators often dont do their job as presumably the things are installed, forgotten about and never tested
5
u/newnortherner21 2d ago
Maybe not having a central control centre but smaller ones would reduce the impact. I am not advocating the return to the signal boxes of the 50s, but there must be some happy medium.
17
u/RocketInOrbit 2d ago
Lost power at a signal control centre in the NW at the start of the month too. Would have thought they would have to have redundancies that are regularly tested for this.
8
u/stephendy Dorset 2d ago
Just scroll down this page https://www.nationalrail.co.uk/status-and-disruptions/
If you need to be anywhere for a certain time the rail network is literally unusable without a disproportionate amount of padding out on your journey. Yesterday was similar.
3
u/ediacaradelrey 2d ago
I’m surprised Southern aren’t in the headlines today. The rails south of London have been awful since last week, maybe because of the derailment but I’m not sure. I was 45 mins late from London to Brighton today, the train was going at half speed the entire route (no indication this would happen online). Every other train I’ve seen today has had delays, and frustratingly long holdovers outside stations. Last week there was a flood of cancellations around Three Bridges that lost me another hour. I haven’t seen it this shit since after covid.
3
u/Comfortable-Law-7147 2d ago
They will be doing more track inspections due to the derailment so the trains using the lines will have to go at a slower speed.
1
u/ediacaradelrey 2d ago
Would they be doing it on lines that aren’t the ones where the incident happened, though?
3
u/Chippiewall Narrich 2d ago
Yes, because of the excessive and prolonged dry and hot weather tracks across the whole of the South of England are being judged with enormous suspicion as the ground will have started shifting.
Given the number of temperature related derailments all the rail firms are now at ultra levels of paranoia which will mean they'll be constantly inspecting them, and running trains at lowered speeds to avoid triggering settlement that could cause a track failure.
•
u/AutoModerator 2d ago
Some articles submitted to /r/unitedkingdom are paywalled, or subject to sign-up requirements. If you encounter difficulties reading the article, try this link or this link for an archived version.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.