All of our data is GONE!

All of our data is GONE!


Well hello, YouTube Um..we have a bit of a problem today on this section of the moving, but we’ve been moved in for months moving vlog um Something is going on with the server and we haven’t been able to work on a project for more than [like] 30 minutes at a time before the server crashes and Our computers all freeze and then we’re all sad because we can’t work on them. Most of the **** we do is on the server well I do it I’m actually doing some writing right now because I can do that. Can’t you’d see I’d love to be working – anything that doesn’t [see] the server We can do now well that’s not much, but it always seems to go down Yeah, but the worst when there’s nothing else to do you’re kidding, but you know yolo, you only Liao once Let’s see let’s see the stress. That linus is going through It’s probably gonna crash like while you’re doing this How do we fix servers? How do you fix it? Tell Linus, he will fix it That’s your go-to. Yeah Whether you’re running a small [business] or a 1 billion dollar even multi-billion Dollar Enterprise Rackspace has your dedicated storage environment covered check out the link in the video [description] to learn more [alright], so here’s the situation, Whonnock server over the course of the last couple of days has been spontaneously going offline and turning off in spite of my best efforts to turn it back on and Complete the backup that [I’m] in the middle of trying to do to our new unraid vault while I was standing in front of it, I observed One of my raid controllers giving up the ghost, so as a reminder This server is running 3 raid 5s striped together in Windows of a total of 24 SSDs so if one of the raid fives drops out entirely [all] The Data is gone only about 10% of the backup that I was just in the middle of remaking is saved So it’s time to investigate All right, so here’s Whonnock server so far troubleshooting steps that I’ve tried… you never wanna you never wanna rebuild something like this if there’s a possibility you’re going to have to go to a data recovery service because that Can make it much much worse so in terms of [troubleshooting] steps I tried transplanting it into another case so that [I] could use a different [Sata] backplane, that didn’t work. I tried a different power supply a better power supply That didn’t work none of that fixed it And I even went as far as to put in another LSI raid card that I had and try to import the array But while it did detect my drives all of them as unconfigured good and a foreign array, It was not able to import it So there is some good news there though while I’m getting firmware errors kicked out by this card right here when I try to boot All the drives are detected by another card, so hopefully it didn’t write too much garbled Data to the array Okay, so this is basically what I’ve been doing for [the] last 14 hours is sending emails contacting different Data recovery services There’s some that are local there’s some that are not local But I’m not ruling those ones out in fact Right now the most likely solution looks like werecoverdata.com, they’ve got some Some custom tools that they’ve created that will allow them to SSH into the system and actually potentially recover the raid the missing raid 5 and then the other [two] raid fives and Assemble them all together and export them as one gigantic Pile of Data [too] to one of our servers, all without me actually having to send the drives away Which would be pretty ideal Now obviously an approach like that would not work in the case of physical damage to the drives But because it is only a raid controller issue. They’re saying hey. There’s a shot at this, so let’s give it a try So I downloaded their remote recovery client thing here So right now. I’m waiting for their custom Linux based tool to load on a bootable external SSD. So that they can SSH into the server I’m not having a good couple days… Right now I can’t get the utility to detect my… my USB drive Called Tech Support! Oh bloody hell… [uh], I just figured it out. This is linus calling by the way [I] just realized I have to run the WRD disk image as administrator You guys should put that in the documentation Never fails, this is why it’s a good idea to call Tech support Because the second you call them You’ll solve the problem on your own Alright, so in theory. We now have a bootable drive! Okay, so step one is getting this Unnecessary extraneous server out of the way, I’m grabbing a UPS to use now in a perfect world I Would have a motherboard or an HPA card a host bus adapter card that could allow me to plug in all 24 [of] the drives for them to do their data recovery Magic on but what we’re going to have to settle for is having eight of them plugged into the motherboard [at] a time as Each raid five is rebuilt. Raid cards do all kinds of funny funny nonsense And they don’t give other software direct access to the drives at a bit level in some cases So we need to remove all these drives and plug them in directly to the motherboard Okay Now it should boot to that in theory Let’s just make sure all the [drives] are detected in the bios and everything oh That’s a problem. I Need to pull up the other raid controllers as well otherwise They’re going to be looking for their drives, and I don’t have them powered up right now Okay [oh] actually that doesn’t look very good pcieport errors What? So let’s make [sure] all of our drives are even showing up. um.. uh oh… That is extremely worrying. [oh] that is extremely worrying… [oh] That’s not good That means potentially three of the drives in a raid five are dead which means this data is not coming back… None of it If we lost three drives from one of the raid fives So let’s try moving things around I’ve never tested the SATA Ports on this board It’s possible three of them are dead, however extraordinarily unlikely that would be Five six, okay, we’ve got six now Okay, that gives some hope I’m just moving Sata ports around. I was just gonna say right now I’m even having difficulty getting this system to post, so I’m I’m at a whole new [level] [of] concern Yeah, I’ll let you [know] I’ll send an email Okay So I’m at the point now where I don’t want to put the drives on this system Let’s go with this one. I didn’t like all those pcie Express errors that the bootable USB was kicking out I don’t like that some of these drives aren’t showing up. I’m starting to wonder if that’s what corrupted the LSI card [will] pain in the ass so thousand things I can think of that. I would rather be doing right now Okay, I got seven right now really yeah, wow maybe it was bad PCI you line is speaking well, I can tell you what’s happening so far is I put it on the system [that] I was running the raid card and the drives in and that baby started spitting out pcie errors all over the place, so I got everything pulled out of that system and I have plugged all the drives now into a different test bench And I’m trying to see [cuz] I wasn’t even getting all the drive showing up two three four five six seven eight okay Hey, you guys are on the phone for the magic moment where all eight drives are detected by a new motherboard So I’m beginning to think there may be a motherboard issue that caused the firmware on that LSI controller to corrupt in the first place So I’m actually pressing enter and I am booting up to your usB and it is loading correctly on this I think that board is dead um **** Okay, well that’s I don’t know if that makes it worse Probably does doesn’t it you can tell me I can take it that says waiting up to 60 more seconds for Network configuration Good timing with the follow up call If [you’d] called one minute earlier, I’ve been cursing you for dialing my number So I have to answer my phone because I was in the middle of thinking only five of my drives were being detected You guys know what you’re doing right? [I] had a local shop that I was going to physically physically take it to and They basically they were like they kind of freaked out when I said [I] was having someone use their remote tool They were like well unless these guys really know what they’re doing. That’s pretty dangerous, [and] I was like okay Well, I’ll check with them. I’ll make sure they know what they’re doing so so you guys know what you’re doing, [so] [basically] they’re explaining that they’re going to virtually mount the volumes so we’re not doing any writing to my disks [so] even in the event that we cannot recover it by this means that still leaves other options open to us okay, there is Okay, well that’s that’s very wonderful to hear but I’m trying not to get too excited right now. They sound pretty confident This is like the ultimate Roller coaster right now, so to put this in the appropriate context for the viewers right now Whonnock Server has several [in] [progress] projects Stored on it that do not have a backup anywhere else including a fully shot Channel Super Fun that involved a thousand dollars in equipment rental Including lots of footage from Linus Tech Tips videos like it is going to be a huge huge Devastating loss for us to lose this I mean It’s got a lot of art our templates for editing editing videos For for scripting templates for employee reviews like all kinds of stuff like this is our main working server Know the off-site backup server has not ever been built yet the vault was the backup server If I ever have to like act in a in a video where I have to look for Lauren I’m going to think back to this moment Okay, so I do need to reboot it once in order to add that [8] terabyte drive [so] anyway right, okay? Thank you very much. Gentlemen. I hope to speak too with good news very soon. Ah ah ah Twelve SATA Connectors ah ah ah Okay, so let’s make [sure] everything is showing up one two three four five six seven eight of those our boot Device and our eight Terabyte recovery Media. We are ready to rock Okay, so within about ten minutes. They contact me back. They said okay. Yeah, we’re ready to have all the drives in there, so That means we are Gonna have to do some Janky-ass **** right now So basically what you’re looking at [here] is These drives are going to be powered off this power supply With a jumper these cards and all these drives are going to be powered off of this rig So now we can power all those drives without powering up this motherboard or any of that nonsense, [okay]. So we have a grand total of twenty five drives connected to this test bench [sixteen] of which are running in bay’s of this enclosure and eight of which are spread out here So there we go. There’s one of my LSI controllers. There’s it’s eight drives There’s my other one, whoa Where are your virtual drives? I think I remember a troubleshooting step where I tried Swapping all the drives to different bays where I was trying to see if it was the back plane or the cables Wow, panic moment there But it’s okay So now we should see two virtual drives here On the screen that’s one set Show me the second set Thought we lost another raid5 there, okay… It’s good feeling good Reality TV ain’t got nothing on IT work Show me all the devices! Two raid adapters one eight Terabyte drive one two three four five six seven eight Kingston drives boom Okay, I will let the data recovery specialists know All right, so all that’s left now, then I think I got in touch with them And I asked if we can put a 10 gigabit network card in so we can accelerate the offloading of all of our files so while we’re waiting [for] now is for them to reboot the machine and set me up with a network share that I can access and Fingers crossed just start pulling off all the footage and while everything else [alright], so check [this] out here [around] [here] So I’ve got the instructions for how [to] access our data So I go into the tritium Share and There it is Bunch of Eula’s a bunch of stuff. We’ve got What looks like the file structure? Let’s test it How about one that I? That we haven’t finished releasing yet like something from the the mail a battle over here. Let’s just Let’s watch it! hmm… Okay, exhale it is for sure uh oh Okay, here. We [can] copy one of these let’s copy one of these clips to my local machine and see if that helps Nope, oh boy Let’s have a look I wonder if everything’s even here like I [don’t] even see the Linus Tech Tips folder 1 & 2 are missing Useful applications is empty that should be full of things okay. Well, I guess we’d better get in touch [with] them So thanks for watching subscribe follow all that good [stuff] and see you next time [I] just got an email back from them please stand by addressing your emails with the engineer It’s possible the raid definitions were changed from one reboot to the next because of drive letter assignments and the Linux 26 was 26 drive limit we have [I] think we have more than 26 Devices in there, I got another email back – we recreated the block devices Please go ahead and check the videos and content that you are [looking] at again. No way. [oh] The suspense… I’m not enjoying this Nothing there. I’ll let him know [the] [shares] empty This is just like the wildest Roller coaster ever I I don’t even yeah. I don’t even have anything else to say actually it’s almost funny. [I] hope to some of you. It’s funny Here’s the email. I got 17 minutes ago. I was stuck on a call We got everything remounted so you can now go ahead and do the data once more. This is the [moment] do we not have a 2 I don’t think we ever had a 2 oh, okay? um Okay, so let’s go to that channel super fun video that. We were looking at yes Yes way way justing okay? And it was the delivery? I’m getting like tingling Okay, okay, you got to check the source file. Oh cuz that was what we know was corrupted, okay, okay, okay? Okay, so Untranscoded footage X-Ray They did it it’s back. Oh my God It’s back. Here’s another file its back. Do we have everything? In theory yes, really? Holy, **** Oh I think it’s finally over! Woo hu! [that] is fantastic Hey guys, guess what happened with the recovery. What?… We got everything back! Thank goodness Like it’s just been such a stressful few weeks, and that was just such a huge Huge thing all of our pitches and proposals and stuff are on that server, [okay]? Whonnock is back Really? Really? Everything is back as far as I can tell I have played back video files They are not corrupted Whonnock is officially back! Yeah! Swegway jousting footage for the Melee Battle should be fine And we’ll have a better backup Scheme in the future I promise That is all Yeah, to save video shots. We should physically print all of our videos every frame I don’t think so… on paper [you’re] the good news? That Whonnock is back? Whonnock is back! Yay for hiring other people to do our jobs! [I] know, I don’t think it’s back, but it is! It’s back What are you doing? I’m just letting everyone know that Taran [it’s] back the server’s back Okay So it’s not always that our sponsors tie so perfectly into our content But today it happened Today’s sponsor is rackspace the top tier manage cloud computing company They pride themselves on best-in-class service across all platforms They’ve got over 300,000 customers in 120 countries with 10 worldwide Data centers. They’ve got Red Hat, Cisco, Microsoft, VMWare certifications and all of that Amounts to whether you’re running a small business or a billion dollar Enterprise Whatever your needs are in terms of capacity storage or flash-based high performance storage they can take care of it without you going [grey] or bald because you did something wrong and Screwed up a backup and went and lost a raid array or whatever else they’ve got dedicated storage to meet your performance security network capacity and compliance needs everything from direct-attached storage so that gives you the flexibility and scalability of Just a simple [ok] let’s attach some drives to this thing. It’s redundant cool off you go to the races to [san] So that’s high availability and reliability fully redundant for business to even mass with support for demanding workloads like virtualization file sharing and Rich media in fact [their] NAS stuff can scale [up] to 20 petabytes of Capacity they’ve got [private] public cloud and private cloud options as well And you can get your own server a scaleable private cloud in the data center in your data center in their data center And it’s all supported by rackspace and Vmware they call their support Fanatical because these people are available 24 hours a day seven days a week 365 days a year, [and] they’ve got industry-leading service level agreements both managed and intensive, so if you need dedicated storage they got you covered! Go to rackspace.com com/ltt to learn more and Not go through what I did I Just yeah, I don’t really have anything to say So thanks for watching guys if you disliked this video then Hit the dislike button. I guess but come on This is about as real [as] it [gets] around here if you [liked] the video hit that like [button] get subscribed maybe even consider Checking out our [Amazon] affiliate instructions for how to use it whenever you buy stuff there Instructions are up there Buying a cool shirt like this one or even giving us [some] of the contribution through our forum which gives you a little contributor badge Now that you’re done doing all that stuff. Hey. Maybe you want to check out that Channel Super Fun video that we just discovered is back and It’ll definitely be worth your while. We joust with each other on Swegways. It’s pretty awesome, so See you next time

Danny Hutson

100 thoughts on “All of our data is GONE!

  1. 4:30 Is that an HDD you've placed at an angle ? I hope it's an SSD I really hope you're not that dumb and it actually is an SSD not HDD

  2. I would have choke slammed that plant in the corner and finished it off with the peoples elbow if I would have lost that much data. Great video linus

  3. I’ve worked on a couple of computers before, the feeling of something valuable slipping from your eyes is heart breaking. It’s hard to watch all of your time being wasted because you don’t have the hardware, or software. I have a Mac, because my laptop is super slow. Of course I tried Bootcamp cause Mac OS is trash, but when I went to install drivers it just kept on showing the bsod. It was an easy fix but the only thing that held me back was…. A wired keyboard.

  4. I realize this is an old video but this is still without a doubt the best Linus Tech video IMO. Not because I wish him to ever go throughout that again but because for once there was some legitimate drama and no silly scripts. Hey I like the silly, but this one had me on the edge of my seat! 😧

  5. I forgot how I got into Linus videos, but it's been a good few months and it's just now getting recommended to me

  6. HAHAHA he forgots the most obvious thing

    run it as administrator HAHAHAH

  7. Always i see this video wen i deal with any pc that cant logging or have problem on boot, like no hard drive detested! just go around to solve the problem!

  8. reminds me of when my client unplugged the off-site backup server at her home, so she could charge her cell phone… lost like 6 months worth of shit. as if there were no other plugs in the house to use…

  9. Thought for sure this was going to be a massive we choose Kingston ssd drives for our server and they fucked us over video. Hear a lot of horro stories about kingston drives and my main drive is a Kingston but it is one of the higher end drives that so far has been flawless. for your back up raid drives you might need to go oldschool and get physical drives back up might take longer but it would be a lot less stressful as you would not need so many drives to fail . you probably got crates full of those drives collecting dust from when you were thinking how awesome ssd raids would be

  10. I Know I am like 3 years late for this comment, but damn I felt this one. I am my self a System Administrator en just the tension true the whole video untill the end to finally here you guys got everything back even gave my system Administrator Heart a big warm fuzzy feeling.

  11. So did we learn a lesson? It looks like one should not put consumer based drives when you need enterprise class storage.

  12. so funny. finally doing a backup on a raid array, uh.. that is what we call a warm failover. come on Linus, drives are cheap. 😉 the ironicy. my relay for redundant backups hiccuped this morning and I got to watch this video while linus was panicking. i just had to replace the $100 relay and let the rolling and delayed sync catch up with one another. no sweat. 😉

  13. The camera man sounds so casually worried. Like, he's worried about the data, but he doesn't know enough to know why he should worry.

  14. should have used linux software raid, and the hbas in jbod mode… probably faster and you can replace the cards without a problem

Leave a Reply

Your email address will not be published. Required fields are marked *