You must log in or register to comment.
10 daysI plead the fifth on that. But how come no backup on this. I am pretty sure they could got at least 50 people to get 1 tb hard drives so their data wouldn’t go this route. And probably more since PBS shaped whole generations of kids and adults into better people.
- Theoriginalthon@lemmy.worldEnglish10 days
15x8TB drives gives you about 72TB in raidz2, for about £4k in disks, add a server for say £2k
So by not spending £6k they lost everything, they spent $19million on IT in 2025
Someone fucked up
- BillCheddar@lemmy.worldEnglish10 days
A tiny PBS office in St Louis has a $19 million/year IT budget? I really doubt that.
- Theoriginalthon@lemmy.worldEnglish10 days
I may have taken the whole PBS accounts, I’m not from the US
- mrgoosmoos@lemmy.caEnglish10 days
you forgot the $10K in labour to handle the backups
changes everything on that $19M budget!
- takeda@lemmy.dbzer0.comEnglish9 days
That’s budget for the whole network, this happened to a small local station.
- Jason2357@lemmy.caEnglish10 days
Heck its pbs. They should have publically archived it so anyone could keep a copy. At least the old stuff.
- alternategait@lemmy.worldEnglish9 days
I’m having a vague memory of some (cartoon) creator or artist who was thankful to torrents because the show that was the cornerstone of their “portfolio” got delisted from a streaming platform’s catalog.
Is this true?
- Mantzy81@aussie.zoneEnglish10 days
Oh sweet child you have a rather myopic view of the weird shit pirates find and their willingness to data hoard.
"What do you mean I should* remove data?? I might need this! " - definitely not me
I once put a ton of effort into tracking down episodes and information of a random Spanish Telanovella I caught on the Spanish channel once. Including recording and digitizing episodes.
I don’t even speak Spanish at all at the time, the show just seemed interesting.
I’ve lso converted a bunch of old Zines from a UK school to PDF because a musician I like was involved and the only copies were behind some weird paywall magazine site. Bunch of print screens and a script to crop double pages to single pages then merge them together.
Maybe I should try putting them somewhere like archive.org.
- SabinStargem@lemmy.todayEnglish10 days
Please do. The more copies that are spread, the less likely a given work will truly vanish from humanity’s history.
- Duamerthrax@lemmy.worldEnglish10 days
The problem is connecting the people who have the files with the people who want copies. I have some weird stuff I’m sure someone might want, but no way to know who.
- lost_faith@lemmy.caEnglish10 days
weird shit
I once found a bunch of the old Disney Nazi propaganda cartoons, wonder if they are still out there
- njordomir@lemmy.worldEnglish10 days
I knew someone who had multiple seasons of both Mr. Rogers and Reading Rainbow!
But I also probably know someone who has Thanos and Fast and Furious.
- boonhet@sopuli.xyzEnglish10 days
I found a bunch of obscure movies from the 1930s and 1940s when a friend asked for some. I’m still seeding them. And I do mean obscure, not like Citizen Kane or something that everyone knows.
- thegreatloofa@lemmy.worldEnglish10 days
There’s a few 90s movie torrents I think I’m the only person left seeding. Things get forgotten about so fast.
The private tracker I use have random stuff like reality shows for Norway television, all the Olympics transmissions, and BlueRay rips of movies from the 30’s. There’s also the popular stuff, but you would be amazed by the random stuff people upload.
- Rai@lemmy.dbzer0.comEnglish10 days
We’re not young people, your original comment is just wrong and makes no sense. Or tells us what YOU preserve.
- ignominous_wombat@lemmy.mlEnglish10 days
I’m not completely wrong, it’s a joke! Isn’t it funny how I got all of my facts incorrect? Upvotes, please!
- ignominous_wombat@lemmy.mlEnglish10 days
I know, dude. It’s an inside joke with a friend, if you care.
Wat… The lone seeder is elevated to mythical status in torrent culture. People take great lengths to preserve old content - whether that’s by seedboxing, ripping physical content, or even the more backend stuff like trackers who have entire systems devoted to “reseed requests”.
- Funkt4st1c@lemmy.worldEnglish10 days
Deleting a torrent is probably the most heartbreaking thing i can imagine. My most prized possession is a ratio 1.00 torrent from 2014 that was 0.00 until 2021.
Rechecking, it is now at 2.00 which makes me unreasonably happy
Edit for more happiness: A copy of Monkey Shines (1988) that took me about 14 months to download is now at a 2913 ratio (since February)
Lmao I just watched the trailer for Monkey Shines. Rest in peace George, you legend.
According to the article, the data is not clearly lost. It’s that their provider, OSS, had a subcontract with Iron Mountain to host the data storage. When OSS abruptly went defunct, PBS had no direct legal path to access the data in Iron Mountain, and no assurance Iron Mountain would preserve it. It filed the lawsuit to make sure the data was preserved and affirm a legal basis to recover the data.
So the headline’s use of “loses access” is accurate but a little misleading. That said, it is a great argument for reliable data preservation.
Iron Mountain is aware of the situation so ideally they’ve flagged it to avoid the bad press of deleting it.
- N0t_5ure@lemmy.worldEnglish10 days
Well, yes, but consider that others using the OSS service may not have the resources of PBS to file suit and litigate the issue, and they may well lose their data forever.
- 10 days
The whole company is sketchy
https://web.archive.org/web/20250318131517/https://www.osstorage.com/cloudPricing
There is nothing open source, their address is the iron mountain denver datacenter address
That page looks like it was directly ripped from https://wasabi.com/pricing, including the “Ball appliance” which appears to be a Wasabi-only product. This reeks of “I know a guy who does storage” and it’s just a few open source products bolted to another storage API with upcharging and a corporate website
- db2@lemmy.worldEnglish10 days
Redundancy would have made this a non issue. Even just local cold storage.
- tburkhol@slrpnk.netEnglish10 days
50 TB isn’t even that much space, certainly not for a broadcaster. No reason not to have a local copy. 50 PB, I could see a small PBS affiliate not having the resources to manage locally, but you can put 50 TB in consumer-grade NAS.
- BassTurd@lemmy.worldEnglish10 days
That’s less storage than my Plex storage. PBS could and should definitely do better.
More specifically, this PBS affiliate (Nine PBS St Louis). It wasn’t PBS proper. This only impacted local programming archives. But that’s still a loss for the local community.
PBS likely does much better than this. Heck, a lot of their stuff is backed up in the Library of Congress.
- ignominous_wombat@lemmy.mlEnglish10 days
Either way, like under $2000 would have bought them the storage necessary to keep these backups on-site. They still would’ve needed another backup solution for redundancy purposes, but this wouldn’t be the huge issue it seems to be now, where the data is sort of in limbo.
Agreed. Although sometimes these smaller affiliates don’t have super-seasoned IT talent. The jobs for the department heads at these organizations often make what an intern or college grad would make in the private sector at an unremarkable mid-size tech company.
You often get people making questionable decisions simply because they don’t have seasoned IT people in the building.
All in all, this was dumb, but it also feels like a story I’ve seen before. The nonprofit media isn’t funded, and the cracks are showing.
- BassTurd@lemmy.worldEnglish10 days
When I started at my last job, I was fresh out of college, only Best Buy Geek Squad experience, and I was the sole IT guy at the company. All “backups” were shadow copies and VM snapshots. Shadow copies are great for simple accidental deletions, and the VMs would work as a backup 90% of the time, but it was not a valid backup solution. When I left we had onsite backups to hard drives, cloud backup in Azure blob storage, and remote replication geographically distant.
It still could have been better, but it was something. Also, we worked on government contracts and stored confidential and ITAR controlled military and government drawings and data. It was one of those things that I’m absolutely shocked that in my 9+ years there that we never had a major breach or lost any data.
50T isn’t even that much data anymore. You’re talking about maybe $2k in hard drives.
Refurbished Exos X22 drives with 24GB Capacity cost around 650€ these days, those 2k are the high end even nowadays.
Refurbished
If you’re a multi-billion dollar broadcast parent company doing long term storage for a 70-year archive catalog, you can spring for new. Maybe even go Big Fancy and get some RAID drives and tape backups, for extra redundancy.
- fuzzzerd@programming.devEnglish10 days
In the pre-AI days maybe, today 50TB is expensive due to increasing prices of all computer gear.
today 50TB is expensive
I don’t believe you can go back more than five years and find per-TB rates under the current rate. Ten years is straight out. Twenty years is laughable.
Meanwhile, PBS has an annual operating budget of around $340M. They can afford to manage their own data, at the scale of tens of terrabytes. And I guarantee you that whatever they’re paying Iron Mountain to host the data is multiples the cost of self-hosting.
- bad1080@piefed.socialEnglish10 days
it sucks but they didn’t follow the basic 3-2-1 rule of back-ups (3 copies, 2 types of media, 1 off-site)
- CluckN@lemmy.worldEnglish10 days
That’s smart, My last company did 4-3-2-1. When the servers went down all we had was 4 calling birds 3 French hens and 2 turtle doves.
- bad1080@piefed.socialEnglish10 days
but it wasn’t even a back-up when they no longer had the original files
- TheOctonaut@piefed.zipEnglish10 days
That’s kind of how bankruptcy works. You can’t give notice that you’re going to be bankrupt. The moment you’d give notice of bankruptcy, everything would collapse anyway as lenders call in debts and customers cancel orders that are now in jeopardy.
Other statuses that exist are examinership (an outside appointee trying to save the company by any means) and receivership (probably the closest to what you’d like - the company is entrusted to a third party to try to pay its debts before it is wound up, usually trying to determine which debts to pay for the greater good, and which ones to default on).
- TheJesusaurus@piefed.caEnglish10 days
Someone just tell me that there are backups of most of this properly archived somewhere. Just tell me that, I’m not asking if it’s true, I just need someone to say that.
The data is all there, the vendor they were using to manage the storage just went OOB, and they’re not yet being allowed to retrieve it from the vendor’s storage vendor, Iron Mountain:
The lawsuit filed in Denver District Court alleges that the station’s cloud-storage vendor, Open Source Storage, abruptly cut off access to Nine PBS’ data earlier this year without warning. It states OSS, which had a separate relationship with Iron Mountain to provide data storage, went “defunct,” leaving Nine PBS’ archives in a data center operated by Iron Mountain.
- eleijeep@piefed.socialEnglish10 days
Open Source Storage
Anyone heard of this company before? Their name is impossible to search for!
- eleijeep@piefed.socialEnglish10 days
Thanks. This is the latest archived copy: https://web.archive.org/web/20250328163113/https://www.osstorage.com/
End-to-Endless Digital Storage Solutions
Endless? lmao
Iron Mountain is obviously aware of the situation and probably doesn’t want the bad press of deleting it. PBS just needs the court order that makes them hand the data over. Because PBS isn’t iron mountains client, they can’t hand them the data without a court order.
Does the local St. Louis affiliate produce anything that lots of people would want to seed? They look like a smaller affiliate compared to the WGBH and WNETs out there.
I imagine KETC is a really small shop that mostly broadcasts stuff others have made, plus hyper-local content that isn’t expensive to make.
- Aceticon@lemmy.dbzer0.comEnglish10 days
Risk analysis in IT Management nowadays is spectacularly bad.
If you become dependent on a 3rd supplier then their survival as a company is now a risk for you. It doesn’t matter how good the reliability you have contracted with them - if they go under whatever penalities in the contract for failing to deliver said reliability will never be paid.
You might even be at risk of them simply chosing to close that division - small company signing a service supply contract with a big company: good luck getting a contract where the parent company (rather than whatever paper company they spun up for that business area) actually has to pay you stiff penalties if they fuck you over.
As soon as mission critical anything is outside your company, a lot of otherwise irrelevant things become operational risks for your company.
This is why you don’t put your mission critical anything on an external Cloud supplier.
You know, stuff like the only copy of 70 years of TV history that’s actually just 50TB and would’ve fit a bunch of Hard Disks or Backup tapes.
- Test_Tickles@lemmy.worldEnglish10 days
This is something I kept pointing out to management at my last company, but they were always so hellbent on putting everything in the cloud. Then construction across the street cut some fiber or something going to our building. When I told management that we had no Internet and I was sending everyone home, they lost their shit. I mollified them a bit by pointing out that at least some people would be able to work from home. I then waited until the meeting the next morning to point out that only 1 QA/support person and 1 dev had laptops (because of course management didn’t see any reason to spend the extra money for laptops over desktops). It was also nice to be able to make the CEO and VP to drive all the way home and back to get their wireless data pucks they used as their personal Internet connections.
My overwhelming joy at all the petty little ways I was able to rub their bad decisions in their faces was a big part of what made me actually start hunting for a new job rather than just thinking about it.- Aceticon@lemmy.dbzer0.comEnglish9 days
Yeah, even having learned to explain this shit in the IT Consultancy language of “supply contracts” and “mission critical”, it’s still not guaranteed that I’ll convince a manager to avoid or at least account for the higher exposure to outside risks when subcontracting certain things.
Nowadays, with 3 decades of experience in the Industry, I honestly think that the majority of managers, no matter how high in a company, are either incompetent as fuck or simply manage with as priority maximizing their next bonus in the full knowledge that they’re leaving traps for the next one to solve and if they do blow up during their own time, they’ll still personally end up better off by going for bonus-today-problem-tomorrow than refraining from doing so.
- BarneyPiccolo@lemmy.todayEnglish9 days
The Peter Principle states that a manager will get promoted to his level of incompetency, then get stuck there. As time goes on, more and more managers reach their level of incompetency, until the entire management level is populated by people who are bad at their jobs.
Say a guy is a great salesman, the best in the company, beats all his targets. He’s just great at it, and all his customers love him. So he gets promoted to sales manager, with a nice office, and a big raise.
But just because he’s a talented salesman, doesn’t mean he knows how to manage sales people. Also, the guy who replaced him isn’t pulling in near the revenue, so the department he’s managing is now WORSE than before he took over. He becomes just another mediocre manager, in over his head, and he never gets another promotion again.
Or maybe he moves up a few more levels, but eventually he hits the point that he can’t do well, and that’s where he remains, being mediocre. After a while, ALL the managers are people who were good at their old jobs, but mediocre at this one, and here they stay, until they are swept out in their 50s.
- Aceticon@lemmy.dbzer0.comEnglish9 days
I think that there are roughly 4 causes:
- As you mentioned, people promoted all the way up to the first position they are truly incompetent at, i.e. the Peter Principle
- Nepotism and cronyism - people are getting the jobs because who they know (i.e. “the old boys network”) or who their family are, not competence.
- People get the job because they’re good at personal image management and social networking - they mingle with the right people and project the right impression with things like speaking in a firm confident manner about things they know nothing about - which are fine skills to have and even applicable to some of the job of management (at the mid and above management levels), just not enough for the rest of the job of management, and that part missing is in most management positions the biggest chunk (and includes the stuff for recognition of risks and accounting for them).
- Upper level managers hire people like themselves to work under them. This partly overlaps with number 2 but goes further. When said upper level management is incompetent because they got in due to one or more of the other 3 problems, they hire people the same profile as themselves even if not due to those reasons directly (a simple example: managers who look and behave posh because they attended a posh private school will tend to hire managers who look the same hence are likely to have attended such schools themselves - this isn’t directly cronyism but ends up producing a similar effect as they’re selecting people from their social domain even if they don’t know them personally)
I would say that nowadays in management the first one is a pretty small fraction of it (except maybe in the less prestigious and less paid low level management positions) because people are rarely “promoted from the ground floor” nowadays and hiring for more prestigious positions is very heavy on the impression people project, so most of the problem are numbers 2 and 3 plus a bit of number 4.
Number 4 is especially pernicious because it means that once an institution is managed at the top level by people who do not recognize the importance of certain skills in management (like caring about and understanding risk analysis) it tends to mainly hire people who are the same for other management positions and even for replacements for the top job - people who don’t think a certain skill or mindset is important for the job are not going to have as a hiring criteria that candidates have such skill or mindset.
- northernlights@lemmy.todayEnglish9 days
Yeah I did risk analysis for 9 years and the job gradually devolved in useless box check-in with people checking nothing, until all of us were laid off and replaced by an AI and a few people in India checking the output. It’s all for show.
This looks like this is isolated to KETC’s locally produced shows for Missouri. They appear to be a smaller affiliate and they don’t make a ton of stuff that contributes to national and international broadcasts.
Since they have a smaller audience I imagine it’s going to be harder for KETC to find community-created backups of their stuff.
I also imagine that if Missouri gave more of a shit about public broadcasting, this affiliate would be operating with better funded / more experienced IT staff. These little affiliates really rely on a lot of volunteers and green IT folks who do the work for the experience.
- LordCrom@lemmy.worldEnglish10 days
I do volunteer work for nonprofits. Its amazing that normal people think we are swimming in cash… newsflash, we are not.
The IT budget is nothing so alot of what i do is setup open source stuff to meet needs…think libreoffice erc.
External storage for large backups is not on the table. And no one wants the reaponsibilty of shuffling external backup drives to and from.
And IT staff is often a lot of young folks who are building their resume and make really dumb rookie decisions. I was once that rookie.
All the people in the thread dissing the station and suggesting all they needed is “X many Y terabyte drives for $2.50 each and give them to people to take off site and store and rotate somewhere and all this wouldn’t be a drama”… Those people have yet to discover that manual processes using fallible humans does not work long term.
The station’s IT staff no doubt looked at all of this and decided that off-site backups via a third party was the simplest and most cost effective route. I have no doubt the middle-company advertised that all the actual storage was in Backblaze, a quite reputable storage company. The only thing that has bitten them in the ass is that the middle-company has folded unexpectedly, requiring them to scoot to court to get Backblaze to give them access to their data.
After all of this, it’s quite possible that IT will now get enough resources to do the backup via Backblaze directly. If Backblaze wants to put a bit of PR spin on it, they’ll task a few engineers to set this up for them. And if Backblaze was smart, they’d go looking for alllll the other little PBS affiliates in the same boat and offer them the same setup.
- HugeNerd@lemmy.caEnglish10 days
Jesus Christ, what happened to P2P? Just have the files in constant circulation through the world’s computers. And 50TB, while a lot, isn’t beyond the reach of a well-heeled private individual to host on a NAS.
- elucubra@sopuli.xyzEnglish10 days
50TB is not a lot, it’s actually a capacity that some homelabers and data hoarders have or surpass.
It’s a lot if you were trying to source the hardware with today’s process, but 5 years ago that would be nothing to an organisation.
- RamRabbit@lemmy.worldEnglish9 days
It’s a 4-digit number, even today, to build a 50tb local backup box. Absolutely nothing to a company. They were simply lazy and didn’t do the bare minimum.
- BarneyPiccolo@lemmy.todayEnglish9 days
The first time I heard about cloud storage, I wondered why anyone would store their stuff with a third party, for reasons EXACTLY like this. This is the exact scenario I thought of.
And while we’re talking about this specific case, presumably all the other customers of that cloud storage company list all their stuff, too.
- rmrf@lemmy.mlEnglish9 days
Using cloud is not the problem here and never has been an issue from a technical design standpoint, it’s having the data only with one cloud provider that allows this to happen.
The only customers of that cloud provider that lost their stuff are the customers that would have lost their stuff using any other method they’d implement, because the only way you can lose your data this way is through poor design.
Aw, that’s sad. I loved Mr. Rogers and Dragon Tales.
Fuck cloud storage.
They are still around. This specific incident happened to a local PBS broadcaster, and it is only the local programming that was lost. Still sad, though
Yeah, but relying on a shady outsourced cloud storage middleman provider was kind of a terrible decision
I actually have a marble track that automatically syncs with my hard drive to back up all my data in a series of binary pits that either do or don’t contain a marble
- JingoBingo@lemmy.worldEnglish10 days
You can blame the storage provider but ultimately it’s their fault. Anyone who takes archiving serious knows that you need at least 3 backups of your archive in different locations for cases like this.
They should have made better decisions, but if they paid a middleman to outsource their storage to third company and the middleman goes out of business without having any terms with the storage provider regarding the data rights of their clients, then that sounds like somebody committed fraud and should face consequences
- remon@ani.socialEnglish10 days
50TB seems very little for 70 years worth of data … I have more than that it it didn’t even take me 4.
- 10 days
Video files only got big in the last decade or so. Lower resolutions are a lot smaller
