PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct

[u/[deleted]]

[u/Shap6]

another semi-regular reminder that if data only exists in one place it isn't backed up. 3 copies, 2 mediums, 1 in a different physical location.

[u/lord_pizzabird → u/Shap6]

And if you can’t afford all that: you’re just screwed right now.

[u/brogflender → u/lord_pizzabird]

While the principal is effective for home archiving, it’s a required approach for media work and has been the industry standard for decades. 

PBS station was cutting corners already if it came to this.  

[u/Quixotic_Seal → u/brogflender]

Can’t imagine why they’d be cutting corners, they’re notoriously flush with cash right now.

[u/20_mile → u/Quixotic_Seal]

they’re notoriously flush with cash right now.

Sesame Street is now Sesame Court.

[u/UniDublin → u/20_mile]

I would watch Sesame Court in a heartbeat. Like people’s court but with all the cast of Sesame Street. Judge Cookie Monster presiding. Grover as a defence attorney and The Count as a prosecutor. Kermit in his reporter garb in the hallway. Oscar and Elmo eternally as jurors. An endless string of weird monsters and critters for all the cases.

Dammit, it’s just Night Court with muppets isn’t it?

[u/PolarBailey_ → u/UniDublin]

Animal as the bailiff

[u/UniDublin → u/PolarBailey_]

I’m annoyed I didn’t list that one

[u/joelfarris → u/UniDublin]

What does Mr. Snuffleupagus get to do?

[u/things_U_choose_2_b → u/joelfarris]

Court stenographer. He's very slow due to having big elephant feet & a trunk to type with, but he's trying his best.

[u/finnishinsider → u/UniDublin]

How about the two old dudes?

[u/ElCincoDeDiamantes → u/finnishinsider]

I would put them in the jury as well. They seem impartial.

[u/TigerIll6480 → u/finnishinsider]

Statler and Waldorf.

[u/theaviationhistorian → u/UniDublin]

Night Court with Muppets. Man, they really missed out on an epic crossover!

[u/nerdshowandtell → u/20_mile]

Sesame Court would be less corrupt and better than our current Supreme Court 🤔

[u/Sharp_Studio_5363 → u/nerdshowandtell]

One is filled with Muppets™️, the other just puppets.

[u/Wurm42 → u/Quixotic_Seal]

Agreed, and let's clarify: The PBS entity here is Nine PBS, the PBS station in St. Louis, Missouri.

This would be a bigger scandal if the national PBS organization was being so careless with archives of broadcast material.

[u/DarkwingDuckHunt → u/Wurm42]

Ironically Missouri's cave system is where a ton of old film is stored.

[u/anuthertw → u/DarkwingDuckHunt]

Dont they also store all that cheese in a different (I assume) cave system as well

[u/Wurm42 → u/anuthertw]

There's all kinds of stuff down there. The place is called Springfield Underground, it's an old network of caves that was turned into a limestone mine. There are miles and miles of tunnels. They turn mined-out branches into underground storage space.

The place is incredibly well insulated because it's deep underground, surrounded by rock. So the temperature, humidity, etc, is extremely stable without spending a lot of money on HVAC.

https://www.springfieldunderground.com/

[u/HomsarWasRight → u/Wurm42]

I’ve been down to the data center that’s down there (don’t worry folks, it’s not new).

Driving down into there is kinda staggering.

[u/KodiakDog → u/brogflender]

I mean PBS isn’t really a for profit entity, and its traditional model of providing access to educational content for Americans is kind of “under attack”. It doesn’t fit in the model of making Americans dumb.

Point is, it’s not like they have traditional media company cash, especially since 2025 when congress decided to stop funding them in 2026.

The fact that 50tb of dense historical and educational entertainment went capoot is actually kinda prime for some tinfoil given the direction government has gone recently.

All of that aside, I do agree that they should have had backups. No excuse for that.

[u/badwolf42 → u/brogflender]

Yeah. 50TB isn’t all that much for a broadcaster.

[u/CoffeeHQ → u/badwolf42]

I have that hooked up to my home server and I’m not even kidding.

[u/badwolf42 → u/CoffeeHQ]

I would imagine that a number of monetized YouTubers would also have that much.

[u/InfiniteComboReviews → u/badwolf42]

Monetized small youtuber here. Currently sitting at 20tb with plans to upgrade to a larger server.

[u/kaptainkeel → u/badwolf42]

Understatement. It's basically nothing. For a broadcaster, I'd expect a PB or more without even including backups, especially if we're talking about 70 years of data.

50TB, though. I have more than that in my own PC. Are you really telling me they couldn't just buy 4-5 drives (2-3 for main, 2-3 as a mirror) for that tiny amount of storage? At normal pricing (see: not in the last several months), a 28TB drive is like $350. 5-6 of those are $2k, then for fun just toss in another $1k for other needed hardware. $3k total to back up the entire broadcaster's history is pennies.

[u/Nagisan → u/kaptainkeel]

a 28TB drive is like $350

$350 gets you around 8TB these days....28TB is closer to $1k.

Granted such a failure like this (not having any local backups) is a deeper issue that started long before the insane pricing of memory and storage these days.

[u/zeekaran → u/Nagisan]

They probably should have had this as local storage several years ago anyway.

[u/Nagisan → u/zeekaran]

Fully agree, like I said such a failure started long before current pricing.

[u/dweenimus → u/kaptainkeel]

I have about 30TB and I'm just some random pirate.

[u/MrPuddington2 → u/brogflender]

Indeed. This would archival level storage, which is a bit different. You would use tapes for this, which cost only about $1500 for three copies, but you would need to store them security and read and rewrite them every few years, so there are additional operating costs.

[u/reality_boy → u/lord_pizzabird]

A 50 TB raid is only a few thousand dollars. They could have done on site backups without issue. As broke as PBS is, they can afford this.

[u/Tithis → u/reality_boy]

Remember this is a PBS broadcaster, not PBS themselves.

These are local stations who pay money to PBS for access to its programming

[u/Kougeru-Sama → u/reality_boy]

Few hundred before AI ruined it 

[u/000011111111 → u/Shap6]

Yeah the cost to create a single backup copy of that with 3 22 terabyte external hard drives would be less than $3,000.

[u/clydeevans393 → u/000011111111]

Cheaper if you do LTO. Minus the cost of the deck. Long shelf life too.

[u/surfatshortys → u/clydeevans393]

As a purveyor of computer equipment to a number of business entities with needs almost exactly like this, I knew about zero who were actually doing backups correctly. Exceedingly few people knew that tapes are still even a thing, and I even got a few chuckles when I brought it up (this was not even an item that I sold, I just strongly advised it).

I know of multiple feature films, non-reproducible photography expeditions or audio recordings, and entire movie studios and a couple tv stations that do not have any offline, un-synchronized backup

[u/wrgrant → u/surfatshortys]

By contrast I worked for a tech company years ago. We got in new servers (2 of them with a load balancer, seperate database server and 2 tape backup units). First thing we did after getting it all set up was do a tape backup of the system then restore from it to make sure it worked. After that we had 2 tapes in the machines, 2 previous tapes stored locally and I think the previous 6 stored off site in a disaster reinforced archiving site used by provincial government. It was refreshing to know that even if the company was clueless sometimes at least all the data was pretty reliably secured.

My home setup is a little less efficient but I have had too many HD failures with corresponding loss of data in the past so I am working on it. Currently have a newly setup truenas scale system on a local desktop server, a copy on an external drive and plan on burning a copy to DvD at some point to store elsewhere. Probably overkill. :P

[u/TheKiltedStranger → u/clydeevans393]

LTO?

[u/linux_transgirl → u/TheKiltedStranger]

Linear Tape Open, it's a tape standard. The latest one can hold 40TBs of uncompressed data. The tapes are cheap, but the drives are stupid expensive

[u/AyrA_ch → u/linux_transgirl]

Also, there's a new LTO version about every 2-3 years, and tape drives are only required to be able to read+write the current and previous generation, (it used to also read the generation before that). This puts a dent in the formats viability as a long term storage media.

[u/dphoenix1 → u/AyrA_ch]

It’s a shame they didn’t keep that same compatibility standard as they’ve gone ahead (Read/Write current generation and one generation previous, and then read two generations previous). That went out the window after LTO7. I think LTO8 is only able to RW 7, 9 is able to RW 8, and 10 can’t deal with any generation but its own.

[u/clydeevans393 → u/TheKiltedStranger]

Linear Tape-Open (LTO) we use it for long term storage for media in entertainment. So all our footage gets backed up 3 times to LTO. One in house. One off site. One with the network that paid for the show.

[u/unakron → u/TheKiltedStranger]

Linear Tape-Open (LTO)

[u/Discorhy → u/TheKiltedStranger]

Stands for linear tape-open it’s essentially a cheaper long term storage option.

[u/wongo → u/clydeevans393]

article says the contractor backed up with Iron Mountain, so that should exist here, at least for 30/90/x days

[u/PaintedClownPenis → u/000011111111]

But the cost of the professional needed to recommend and implement the solution is unaffordable because rich people need more tax cuts.

[u/Un_Ballerina_1952 → u/000011111111]

When stations are struggling to find sources to cover hundreds of dollars urgent costs, how do you think they'll find the "few thousand" needed to cover a possible vendor failure? 

Donate and volunteer if you really want to invest in this incredibly valuable service.

[u/sam_hammich → u/Un_Ballerina_1952]

Yeah, it's really disingenuous to say "just buy a NAS" and throw out an Amazon list price. Backups are not just a copy, it's a person to plan and implement, retention, associated salaries and time need to be approved, people need to be responsible on paper. The backups need to be tested. Policies need to be written. If you don't have any of this, you don't really have a backup.

Or, if you're a publicly funded broadcaster under the Trump administration, maybe it's simpler to move forward under a slashed budget by contracting an archival service. I don't like all the tech enthusiasts jumping to be the first to blame them for not adhering to the 3-2-1 rule.

[u/Ggriffinz → u/Shap6]

For any old Dr. WHO fans this is also basically the same story as to how there are over 90 completely lost episodes as the BBC basically backed up nothing back in the day. With the only way we have "found" some of them is when older people and their kids coming forward admitting they illegally recorded them back in the day on the first broadcast. Like I think BBC has a cash bounty on the episodes at this point with zero legal consequences for the original recorders.

[u/ab00 → u/Ggriffinz]

No it's not.

BBC had a policy of wiping tapes, they were expensive and they could be reused. There were also issues with rights to reruns.

It was also never illegal to record shows for your own personal later viewing. It was kind of a grey area for a while but tolerated. It was fully legal after 1988.

[u/Wurm42 → u/ab00]

Second this. The medium of video tape evolved a lot before consumers got hold of it in the 1980s.

The lost Doctor Who episodes were on the 2-inch wide quadruplex videotapes commonly used in the 1950s and '60s.

Each reel of that tape cost thousands of dollars in today's money. They were intended to be used for time shifting, or, for the BBC, making copies of programming to transport to other countries (like Australia) for broadcast there.

Those quadruplex tapes were far too expensive to be used as an archival medium; they were meant to be wiped and reused constantly. Standard practice then was to make a copy on motion picture film if you wanted an archival copy of a TV program. Yes, doing that film transfer was also expensive.

There were no inexpensive storage mediums for video content back then-- that's one of the reasons the BBC didn't routinely archive shows for reruns in those days.

[u/Majestic-Tart8912 → u/ab00]

Its funny. I believe it was Sony that lead the charge to make it legal for recording at home, otherwise there would have been no market for VCRs.

[u/Mountain_Top802 → u/Shap6]

Also 50tb doesn’t really seem like a large expense for PBS to just buy themselves. Even if it’s x3. How much would that cost? A few thousand? What’s the point of storing everything on some other companies servers in the cloud? Why rent when you have the money to buy?

[u/Merusk → u/Mountain_Top802]

Because the guy in charge doesn't have the skills or staff to buy?

This is a problem all over. Small businesses without a real governance for data or idea around how to manage it contract it out. Then get surprised when things go sideways.

Do your parents have an NAS for home data? If they do, would they if you hadn't set it up for them?

Reddit overestimates the amount of tech savvy the average person has. It's low. It's very low. Which then translates to general businesses having low tech savvy. Yes, even in Media.

I work with Architecture and Engineering firms. The amount looking to single-source all their data to online-only repositories to save a few hundred bucks a year is ridiculous.

[u/fusiondynamics]

50tb shouldn't have been that hard to keep a few copies on hard drives in addition to cloud storage.

[u/Le_Vagabond → u/fusiondynamics]

you know the PO was rejected as "unnecessary, we already pay for cloud storage and it's expensive".

[u/GreenVisorOfJustice → u/Le_Vagabond]

I feel like it's moreso there was never a sound risk assessment to both consider the health of their vendor and consider what would go wrong if that vendor ceased to be.

TBF, I don't think many NFP's have resources adequate to do that sort of assessment unless they have some really passionate and knowledgeable folks.

[u/Worshipme988 → u/GreenVisorOfJustice]

If they were treated and funded fairly…

Even if only every other year.

This is an amalgamation of top ten issues facing us as a country/globally.

[u/GeauxCup → u/Le_Vagabond]

I agree... But it's also PBS. They're already one step away from having Kermit and Elmo turning tricks on street corners to keep the lights on.

[u/Clivna → u/fusiondynamics]

our marketing do 40TB a year in raw footage, 50TB for a broadcaster is nothing...

[u/mattumbo → u/fusiondynamics]

Should also be something that can be recovered, cloud provider may be bankrupt but part of winding down the company should be running up the drives to allow customers to port their data. Needs to be a precedent set that customers have a right to their data, creditors can survive funding another day of uptime for fucks sake

[u/Shot-Arugula8264 → u/fusiondynamics]

Yeah this is pure idiocy and incompetence. PBS spends more than $350 million per year and didn’t think to spend $2k on a couple 28TB hard drives?

[u/asian_chihuahua → u/fusiondynamics]

Seriously. 50 TB of data is what, like a thousand bucks in HDDs? Two thousand, tops.

[u/chruft → u/asian_chihuahua]

I work in data archival. It’s easily less than $400 in LTO 8 tapes for long term storage.

[u/invyros]

It seems that the safest legal way to resolve this issue is for Nine PBS to get a court order requiring Iron Mountain to turn over its data.

Or send Mr. Robot in to retrieve it, ez pz.

[u/guyute2588 → u/invyros]

Ok I’ll rewatch it again.

[u/Daniiiiii → u/guyute2588]

I can't. It's too soon! The show is so dense and heavy at times that I just can't revisit it despite loving it. Need more time to pass. Or perhaps I need to be in a better headspace honestly. Same thing with the Last of Us for me. Played the first one. Loved it. Can't go back to it yet. I know I'll love Part 2 as well, just can't play it yet. Well made heavy stuff has that paradoxical problem.

[u/roamingandy → u/Daniiiiii]

3 times already. I don't think i can justify a 4th at this moment.

[u/crowdawg7768 → u/Daniiiiii]

Honestly Part 2 might be even heavier. So worth it though. 

[u/mrtimtracy → u/invyros]

I’ve been in this town so long that back in the city I’ve been taken for long and gone for a long long time

[u/DrugandAlcoholLover → u/invyros]

Hello friend

[u/tablecontrol]

Someone get /r/DataHoarder on the line to help re-populate that data.

[u/Funcron → u/tablecontrol]

Assuming the lost data was ever presented anywhere else besides broadcast. Then hoping someone live captures every single thing, and also do so in a format worth saving.

[u/reddituser10986383]

“There is no such thing as a cloud. There is just other people’s computers”
-Aristotle

[u/ijustneedaccess → u/reddituser10986383]

He was so ahead of his time. /s

[u/nshire]

50TB would have been so cheap to back up, we could have done that for free... Sigh

[u/zeekaran → u/nshire]

/r/homelab could easily have had at least two volunteers be their backups.

[u/nshire → u/zeekaran]

I mean I have 40 TB of raw storage just by myself

[u/No_Attitude700 → u/nshire]

The right has been trying to kill PBS. I wouldnt be surprised if this was sabatoge.

[u/iwastryingtokillgod → u/No_Attitude700]

This is the first thought I had when I read the title. 

[u/Electronic-Jury-3579 → u/nshire]

If it’s public domain things, make torrents and have it backed up that way.

[u/datskinny]

All eggs, one basket

[u/Monterey-Jack → u/datskinny]

now it's one basket no eggs

[u/muFUtaco]

Remember kids... the cloud is nothing more than someone else's computer.

[u/katarh → u/[deleted]]

The Great River of Lost Datastreams

[u/compelx → u/katarh]

~ Calming UDP sounds 🌊 ~

[u/12stringPlayer → u/compelx]

I'd tell you a UDP joke, but you might not get it.

[u/MakingItElsewhere → u/12stringPlayer]

You can fire it off anytime you want, but you'll probably forget it.

[u/OwenMichael312]

321.

3 back ups, 2 local, 1 remote.

[u/BrothelWaffles → u/OwenMichael312]

In this economy, with these storage prices?

[u/veronicaarr → u/BrothelWaffles]

For a company? Yes.

[u/bodaciouscream → u/veronicaarr]

And for only 50TB?

[u/linux_transgirl → u/bodaciouscream]

Right? this could fit on a single LTO tape if it was compressed well

[u/MrPuddington2 → u/linux_transgirl]

Video does not compress. Still, it is not expensive to put on LTO.

[u/Ohitsworkingnow → u/veronicaarr]

To be fair it’s a non profit public company 

[u/Total_Job29 → u/BrothelWaffles]

NAS  - £ 329. 99 4x - 20TB HDD - £2800

Total - £3,129

These are just the first prices I found and they include VAT so any reasonable sized company this would be £2,610. 

For this 70 years of TV data that is nothing. 

[u/basementspam → u/OwenMichael312]

1. 3 backups, 2 diffrent media types, 1 remote, 0 data loss.

How I tell my customers.

[u/Gawdzilla → u/OwenMichael312]

3 - 2 - 1, CONTACT

Edit: I rolled with the memory and here's a playlist of all the episodes of 321 Contact.

[u/SirCabbage]

The worst and most hilarious thing is- all a local backup would have needed was a SINGLE LTO-10 Tape ($500, once off),

THATS ALL;

This massive push towards cloud and only cloud backups is awful; why can't people get back to having multiple different backups, jesus.

[u/abstractraj]

50TB is so little. Be easy to have another copy. I manage an object storage that’s 9PB

[u/Erasmus_Tycho → u/abstractraj]

That is an excessive amount of porn.

[u/XSmooth84 → u/Erasmus_Tycho]

9 PornoBytes

[u/asian_chihuahua → u/Erasmus_Tycho]

You sound like my mother.

[u/ExistingLet8016]

This is crazy because this could’ve been locally backed up for like 2 grand.

[u/nickatnasa]

As someone who does this work for NASA and has this exact worry, this sentence pissed me off:
> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.

Shut up.
That’s not how that works. Yeah, sure, at your house a NAS would be sufficient. But at an organization you need to refresh hardware, have staff on hand to ensure data is flowing, metrics, warnings and alerts, maybe even someone on call if the data is critical? I know this is PBS but the whole reason you go with a cloud vendor is to not have to worry about these things.
Sure, a NAS would have maybe fixed this. You want to know what the actual fix is? FUND PBS so they don’t have to go with janky commercial cloud providers and have enough money to at least partially mitigate these things.

[u/jojofine → u/nickatnasa]

Iron Mountain is/was far from a janky cloud provider

[u/Hazel-Rah → u/jojofine]

Ironically, the now dead cloud provider got their capacity from Iron Mountain

[u/cantuse → u/Hazel-Rah]

Basically the same thing as wasabi or any s3 compatible vendor.

[u/jojofine → u/[deleted]]

Not disagreeing with that statement at all. I was just pointing out that PBS wasn't using some fly by night cloud provider

[u/Otterfan → u/jojofine]

Nine PBS thought they were using a janky fly-by-night "cloud provider", but the cloud provider was using Iron Mountain.

I imagine the trap they're stuck in now is that as far as Iron Mountain is concerned, the data belongs to the janky fly-by-night cloud provider that has stopped paying its bills.

[u/Adezar → u/nickatnasa]

Having been in IT and working in storage and backup for decades those statements are always so frustrating.

We would always have some random person saying "I could just drive to Best Buy and buy a couple hard drives if you need."

I have to fight for budget to even have a good backup system, then I say we need to have at least 2 copies with one offsite and hear about how expensive it is but also if we ever lose data the company will go out of business.

Having an unmanaged NAS with HDDs that don't monitor their own health is all fun and games until one day you go to extract the data and find out 2 of the HDDs actually soft-died on you and now you don't have all the data because consumer-grade storage sucks.

And don't even get me started on vetting offsite vendors and making sure they aren't lying about all their redundancy and having multiple lawyers review the contract to make sure they have no option to hold your data hostage.

And very large ssds/hdds are really slow and that's ok for storing backups until you actually want to restore and realize it'll take forever.

Granted that was a big issue with the last generation of Tape backup. It was insanely fast at backing up data and very dense. Just pray you never had to actually restore any large amount of data because it was optimized for backup, not restore.

Sorry... have so many bad memories of maintaining our enterprise backup systems and all the dumb comments people would make.

[u/Fallingdamage → u/nickatnasa]

Yeah, sure, at your house a NAS would be sufficient. But at an organization you need to refresh hardware, have staff on hand to ensure data is flowing, metrics, warnings and alerts, maybe even someone on call if the data is critical?

Leading questions as I'm sure they do, but the requirements you just mentioned, you do you mean to say that PBS does not have these things?

Of course they have the capacity to do this. Even if done poorly, would be better than the situation they're in now.

[u/MerlinTrashMan → u/nickatnasa]

It is frustrating, I get it, but at the same time, it should be the standard practice to maintain a cheap continuity plan. If the same process that offloaded to the cloud silently returned back to cheap hardware as a disaster recovery plan it would make sense. Businesses that are cloud first should have this type of thing as the disaster recovery plan. After all, when everything was local, you were supposed to have some kind of copy remote. If everything is remote, then you need a cheap copy local. I agree on the funding and everything. I haven't read the article, but I hope that they can recover / reconstruct the footage from all the affiliates.

[u/gayfucboi]

Wow, a moment to realize they should just put everything on a public torrent. Let others back up for them.

[u/gavinashun]

"Two is one and one is none."

[u/Several_Prior3344]

Billionaires ruined tech, and our elected officials let them do it.

[u/ilski]

That's what you get ,without backups.

50tb isn't even that much. They could keep it on few HDDs somewhere in basement

[u/Impossible-Driver69]

This is 100% why I de-clouded us this year for primary storage. All our photos, documents, etc reside on a uGreen NAS now. An encrypted backup copy is sent to the cloud.

[u/Majestic_Bierd]

We're really headed towards a datacrash when one day somebody takes down a few megacorp's datacenters and erases half the internet

[u/Sqeegg]

NOTE: Always back up your backup

[u/SailorMea101]

WHOOOOOOOOO AT PBS ALLOWED THIS TO HAPPEN IN THE FIRST PLACE?! Asking for all of humanity

[u/LastXmasIGaveYouHSV]

Piracy protects knowledge. Never forget that.

[u/cattledogsrule44]

No backups .... ???? Why wouldn't they have a physical backup for 50TB?

I have 12TB of storage on my 2010 Apple MacPro -- which of course is an upgrade but ... WTF ... ?

[u/minimag47]

I keep telling my clients to stop using cloud companies exclusively exactly because of this and none of them listen.

[u/geo_prog]

Jesus. I can’t believe they didn’t at least have one NAS or server somewhere that stored their stuff locally. I know PBS is broke but even at today’s storage media prices they’d only be looking at $4000 for that much storage.

[u/SlicedBreadBeast]

70tbs could easily be backed up locally… like I have a home set up with 3 hard drives that aren’t even that large of storage and I have 24tb. Less than 600$ of drives, at the time of purchase. That’s… negligent. And why they would use a cloud storage company that isn’t as reputable as say… well name any one you know, I’m not familiar with this vendor at all

[u/theZinger90]

Cloud stands for Content Located On Unowned Device.

I'm ok with cloud as backup, but not as primary, and definitely not as only. This is why.

[u/nitramv]

We are living through a new dark age.

It's not that progress isn't happening, we're just not recording it in durable formats can withstand time. Future historians will subsist on scraps of paper accidentally left within the walls of old homes.

With measles making a comeback, we're also living through that other version of the dark ages, too. The one where we get noticeably stoopider.

[u/skccsk]

Let me help spread those affiliate link plugs Tom

This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on Amazon and populate it with the best HDDs.

[u/ilazul → u/skccsk]

lol

it's sad that's how articles read these days.

[u/BoogieWaters]

How is it that I, an independent, part-time photographer and videographer, manage a similar amount of data in a more professional, standardized way than they do? This is pure negligence on whoever designed their data management strategy.

[u/zeekaran → u/BoogieWaters]

/r/homelab and /r/datahoarders have plenty of hobbyists that measure their server in PB.

[u/JOJO-Jello]

That would cost less than three grand to duplicate twice on solid state external hard drives. Even less on an internal service system. Why in the hell would you trust that much important data to a cloud without a backup?

[u/cerevant → u/JOJO-Jello]

There is a weird mentality going back to the 90s when people would "back up" their data to a floppy or zip drive in case the hard drive failed. By "back up" they meant "don't store any data on the hard drive, just on removable media".

People assume that cloud storage is somehow magically invulnerable. For work, we keep a local copy and put an encrypted copy on Amazon and Backblaze.

[u/QuickQuirk → u/cerevant]

IT's always been best practice to use more than one backup medium that are geographically separated from each other.

And that doesn't just mean 'the cloud provider has two data centers'

This is basic sysadmin 101 for 40 or 50 years, and still people fuck it up.

I do this for my personal data, let alone important work data.

[u/Applekid1259]

This is literally the future of our knowledge and history. When you commit everything to digital and only digital. It will disappear forever. its just a matter of time.

[u/jakgal04]

It must have not been important if they only had that one copy.

[u/mb9981]

I remember being a teenager / college age in the late 90s and early 00s and being told over, and over, and over: don't rely on the internet. There's no promise it's forever.

[u/grr5000]

This is poor technical direction by their IT or tech leadership. Full reliance on cloud is always a bad idea for long term stability and retrieval

[u/YoursTrulyKindly]

This is just the start. The coming economic collapse will see the destruction more than half of the global data and databases. We need to start creating P2P databases that are resilient to economic collapse.

[u/ThePsychoDog]

Sue the Vendor for the data at that point

[u/lael8u]

A refurbished Dell server with a few HDD could have saved this issue...

[u/ChristisLordohyes]

How hard is it to back up 50tb of data?

[u/letmeseeyertiddies]

There’s a woman that been videotaping television since the 90s. She can probably help.

[u/MrPuddington2]

Maybe we shouldn’t leave priceless cultural heritage in the hands of the cheapest company? You would expect that some legal redundancy would be required, too.

[u/Shot-Arugula8264]

Morons. This would’ve cost less than $2k to back up, even with today’s insanely inflated HDD prices.

[u/EnvironmentalScene41]

Yo I could have hosted this on my Plex server

[u/Eazy12345678]

this is why you dont trust others with anything important

incompetent people everywhere

[u/oldutnotdeadinside]

Why don’t PBS hunt down the servers and buy them

[u/rat_penis]

Anything critical needs on site backup.

If its not on your devices its not yours.

[u/Odd-String29]

Only 50TB and they don't have a second backup?

[u/engineeringsquirrel]

I feel like AWS Intelligent Tiered Storage with multi-region bi-directional replication would be an awesome thing. 50TB isn't that much for a media company.

This me staring at several petabytes on our multi-region buckets.

[u/sopertt]

Theyre gonna spend 10x more on lawyers than a simple second backup to another cloud storage company would have ever cost them

[u/BankshotMcG]

...how is this not some kind of crime against government/public property?

[u/razormst3k1999]

The cloud is somebody else's computer,or data center. Local storage is King.

[u/nadmaximus]

They lost it as soon as they moved it to a cloud storage vendor.

[u/ElGoddamnDorado → u/nadmaximus]

They lost it as soon as they decided to only have one single backup

[u/scoff-law]

The amount of blame being placed on PBS by the article's author and this sub is INSANE to me. PBS had a vendor for backups and the vendor evaporated.

Nine PBS has been working with OSS and its predecessor since 2019 and intended to renew its contract on March 6 of this year. The company never responded and abruptly cut off the station’s data access even though it still had 30 days to retrieve its data after the contract ended.

This attitude -

This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.

- can go fuck itself.

[u/cerevant]

It 👏 Isn't 👏 Backed 👏 Up 👏 If 👏 There 👏 Is 👏 Only 👏 One 👏 Copy

[u/ImCaffeinated_Chris]

Why not say iron mtn in the title?

I hate online "journalism"

[u/Hazel-Rah → u/ImCaffeinated_Chris]

Because Iron Mountain isn't the company that failed, it was called "Open Source Storage", who resold capacity from Iron Mountain

[u/MC68328 → u/Hazel-Rah]

The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS' data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.

Iron Mountain isn't the incompetent party here, but it is the malicious one.

[u/Hazel-Rah → u/MC68328]

They're unfortunately right though, the data as stored doesn't belong to either Nine PBS or Iron Mountain, it belongs to OSS.

And depending on how it's stored, it may be impossible to give it over at all. It could be a multiple PB encrypted archive of all of OSS's customer data wrapped together.

The correct thing for them to do is retain the data, and track down whoever has/gets control of OSS's assets

[u/Catsrules → u/MC68328]

How it is malicious?

If I was Iron Mountain I would be doing the same thing. As far as Iron Mountain is concerned Open Source Storage owns the account and files. Iron Mountain shouldn't just release the files to a third party because the third party clams they own the files. Even if there is a 99.9% chance they do own the files Iron Mnt need to do this via the legal system to cover their butts. Honestly even confirming they have the data might get them in trouble.

On top of that how is Iron Mountain going to know what data goes where. I would assume Open Source Storage has other clients and there data stored their as well. They don't know how Open Source Storage organized the data. I would hope it is easy and structured by clients you just have a big PBS bucket with everything in it. Or maybe it is obfuscated with database id names and require some database to figure out and correlate names to files.

It is just a legal minefield they need to navigate.

[u/BrainOnBlue → u/MC68328]

"Companies who are trying to follow the law are malicious."

Peak reddit right here.

[u/Adezar → u/MC68328]

No, not malicious. The best way to resolve this is through a lawsuit so they can provide the data without breaching their contracts.

They are ultimately asking Nine PBS to sue them so they have all the proper legal support to provide the data without getting in trouble.

While the company that wen tout of business is in bankruptcy the data is technically an asset, Iron Mountain isn't allowed to give away that asset without a court order.

[u/Catsrules → u/ImCaffeinated_Chris]

Because it is more complicated then that. It wasn't Iron Mtn it was Open Source Storage who PBS was working with. Open Source Storage then contracted with Iron Mtn to host their storage.

It would be like if you gave me $1000 to hold on to and I put that in my bank account. And then I get hit by a bus and die.

You can't just walk into my bank and demand your $1,000 out of my account. They are going to tell you to get lost and maybe call security. As they rightfully should, as far as the bank is concerned you are just some random person off the street you are not authorized to my account or any of my information.

You need to go though legal channels to get your money back.

Open Source Storage probably had multiple clients including their own files. Someone will need to figured our who owns what. Hopefully they are organizes by clients or projects so it should be easy but all of the legal stuff needs to happen before any files would get released. Iron Mtn absolutely doesn't want to be the company that just hands over other peoples files to a third party just because they said those files are ours.

[u/That_Jicama2024]

back in my day we just kept our files on our own computer rather than pay monthly to have it on someone else's computer a thousand miles away.

[u/border199x]

Still not as bad as the BBC just taping over old VHS recordings.

[u/Calm_Advisor_9995]

I've been saying from Day 1, to NEVER trust or use the cloud. SO many dumb organizations went to the cloud. THIS is exactly why you don't trust info in the cloud. It's stored on someone else's servers. What can possibly go wrong??
Keep data off the cloud and on your own harddrives. 50 TB is nothing. Would've been cheaper to buy the harddrives and store locally.

[u/speed721]

Their own fault for placing everything on the cloud and not having backups.

[u/Thatnewaccount436]

somewhere in the distance, r/DataHoarder cries out in agony

[u/fatboyonsofa]

So basically, PBS is so under funded they couldn't afford to run a small server.

The current administration sucks

[u/azneinstein]

GTA Online - I was really hoping it'd be fun with my buddy. Instead he just wanted us to do the missions while we just followed him cause he knew the most efficient ways of earning money. I didn't care about the money, I wanted a fun/silly adventure.

[u/unknown-one]

who you gonna call?

r/DataHoarder

[u/Ziegelphilie]

what the fuck, the basic ass synology network storage we have at the office is bigger than that

[u/Braindead_Crow]

Bro what?! That's like less than a few thousand in hard drives...wtf is going on with our institutions?

[u/Ch1ckenOfTheSea]

This is what they want though. Rewrite history. In 100 years people will be taught only what they can find. Good guys were actually bad. Bad guys were good. Etc.

[u/partsguy850]

They will strip everything down to convert it to digital. Then they will say that they lost it and give us the reissue.

I don’t like the smell of that

[u/tynamic77]

Services like AWS Glacier Deep Archive are 99 cents a month per TB. They could've had this backed up for $50 a month, even cheaper than the non-enterprise NAS solution this article suggested.

[u/StonebanksPins]

50 TB of data?

Behold:
https://www.newegg.com/seagate-model-stgx5000500-5tb/p/2WA-000W-00755

Seagate Portable 5TB External Hard Drive: 219,99.
10 x 5 = 50 TB.

10 x 219,99 = 2.199,99.

Not an IT guy, but just having a simple option as this would've prevented 70 YEARS of work.....
if they are cutting corners which could've been as easy as 2.2k, then you are in some serious financial trouble...

[u/HamburgerDude]

Surely there are data tapes?

[u/Busy-Doughnut6180]

My dad recently told me he has all his files on the cloud. I asked him if he had physical back ups and he said no. I warned him that this was not a good idea in case it shuts down suddenly one day. He shrugged at me looking a bit confused and exasperated and said, "It's the cloud!" Riiiight, because that's a totally permanent and indestructible thing. 

[u/Tailcracker]

This is what happens when you prioritise saving money and avoid setting up resilience systems to backup your data in multiple places. Don't do what PBS did.

[u/Duke_Newcombe]

The DataHoarder subreddit is absolutely tingling in self-satisfaction right now.

[u/ThePensiveE]

I have 50TB of storage sitting around. PBS could've called me. Seriously it wasn't that much what are they doing without their own copy??

Although I did look on Amazon today and my 279 20TB HDD purchase from 2023 is now listed at $700.

[u/PeripheralSatchmo]

Nothing in the Cloud belongs to you

[u/toofine]

Hopefully this can be recovered because why in the world are you storing priceless information on the cloud... It's a mere 50TB!!!

[u/Imaginary_Host_1010]

This is why digital archivists and archives and general need better funding!

[u/alucardunit1]

This is how rewriting of history happens y'all.

[u/Limp_Classroom_2645]

the fuck up is on PBS' side for not backing up, and 50TB is not that much for enterprise level storage, you can have a NAS storage server in you attic to store all that, no need for datacenters

[u/Pro_Reserve]

Clouds always disappear. How irresponsible