Storage of video files

[chrismorace]

New P4P owner with a question. How are you guys storing all your video files. Those things are huge. I just bought a 2 TB external thunderbolt hard drive. I'm not sure that will be enough.

The video on this thing is awesome.

Depending on your use case you might want to try the Dragonfly app. Over the last month and a half a few thousand people have joined the app and are storing their drone video collections on it. There is a promotion going right now that allows you to earn a lot of free storage if you're industrious enough. Might be worth checking out--it is free to create an account and try.

 

[ChrisRL]

Gents, greetings.

As a video pro since the 80's, I'm always being asked about my mass storage methods. I've settled on the Drive Robots from Drobo, as have quite a few of my pro associates.
I have a couple of Drobo Pros, that have 8 drive slots apiece.

The original Drobo demo that sold me, and many others, showed an original Drobo playing media from the drive to a computer screen. The demo person turned around - and pulled the drive with the flashing light. The other drives immediately went from green to orange, indicating that they were adjusting for the drive loss - but the clip on the screen just continued on playing, nary a stutter.

It's actually happened to me since then - a drive gave up in the middle of an edit. I popped in another blank one when I had the time to go get one, after the session was over. The data shuttled around and the drive had mended itself in a couple of hours. The drive was a 4TB drive. What would have been a major catastrophe became something of an inconvenience.

So I started with 4x2TBs, then kept on adding 2TBs until it was full - that took me around nine months, by which time I could afford to swap out two of the original 2TBs with 3TBs, which I did.
Now, maybe three years later, I have 8x4TBs in both Drobo Pros - that's 64TB, not too bad for a backup system. And the prices for the 6TBs are going to start to fall pretty soon... : )

No I don't work for them, no I'm not paid to endorse anything, nor are any of my associates.
Just wanted to share what was, and is, for me, a good thing.

 

[GadgetGuy]

It's still being tested but BR is certainly a much better storage format then a hard drive. The longevity of a mechanical HD is _far_ less then a BR disc. A SSD is a different animal and it's still being tested as well.

BR is expected to store date for 200 years. A HD life span is 3-5 years. Of course, this is considering that they are stored correctly. A HD is simply not a reliable long term storage solution.

I doubt that in 200 years you will be able to find a BR player anywhere! The longevity of the storage medium used is of no value if you can't find a player for it when you wish to retrieve it. I have Super 8 movies from my childhood, but no access to a Super 8 movie player, while my black and white negatives are still usable and easily scannable. The Super 8 movies not so easily. Anybody still have a Zip drive to play all those Zip disks we used for archival purposes? No matter what medium you use, best to copy it over to a current medium every few years.

 

[Mix_]

I think, a good option is a RAID5 capable enclosure, filled with NAS or server grade HDD's.
If (when) one disk fails, just put in a new one, and the data will be restored.
The read/write speed will be enough for today's drone's video needs (ok, if You plan to edit uncompressed 4K+ raw from high-end RED Weapon/epic etc, thats another tale).

There are many brands (drobo, owc, promise etc), just be sure to choose an future-proof and fast interface, and keep in mind, that those NAS solutions are different beasts - a personal cloud storage is a good thing, but for video/photo, an local-mounted disk array is better.
A.E., Adobe's Lightroom will not allow to create picture catalog on networked disk, and the read/write speeds are not enough fast for editing.
So - go for Thunderbolt 2/3 or USB-C (depends on Your hardware) 4-bay DAS (Direct-Attached-Storage) case, put in a good set of disks.

And, of course, as previous mentioned - it is important to throw out the not-ok video coverege.

 

[andy_k]

Learn to save selectively and not hoard old files.

The likelihood of you ever viewing them again gets lower every time you add a new file to the archive. The likelihood of them ever seeing the light of day decreases with the ever increasing amount of digital data that is being recorded every day.

As others have said. Edit your files down to a final video and after a reasonable period, delete all the unused raw footage. The same goes for stills - why save 25 versions of the same image - pick the best and delete the rest.

It seems to be a digital photography 'thing' - just save everything in case you need it in the future. Learn that not every image is a keeper and get rid of the 'rubbish'. There are very few amateur photographers that have enough 'good stuff' to fill a 1TB drive, never mind some of the numbers being mentioned here

 

[tcope]

I have Super 8 movies from my childhood, but no access to a Super 8 movie player, while my black and white negatives are still usable and easily scannable. The Super 8 movies not so easily.

https://www.google.com/search?q=sup.....69i57j0l5.4234j0j7&sourceid=chrome&ie=UTF-8
Simple Google search. $50 on ebay.

Anybody still have a Zip drive to play all those Zip disks we used for archival purposes? No matter what medium you use, best to copy it over to a current medium every few years.

$150 on amazon

But none of that changes what I said. HD storage is not a better long term storage when compared to BR.

 

[anotherlab]

[QUOTE="tcope, post: 1247189, member: 21184"...But none of that changes what I said. HD storage is not a better long term storage when compared to BR.[/QUOTE]
Blu-Ray disks should last least a century if stored properly. But if you want storage where you can easily and remotely access your data. hard drives in a RAID array are more useful.

 

[Raptorman0909]

Blu-ray may store 50GB and I believe there are options that increase that to 100GB so that's pretty decent, but if you have 10TB of files you'd need 100-200 disks and since redundancy multiply that by at least 2 and upwards of 4. I'm not saying it's not doable, but do you want to spend that amount of time have at it.

Looking down the road about 5 years or so I think SSD's will finally reach a price point to compete with HD's and with storage densities to permit 10TB to 20TB per drive. Then, you have 3 or so copies and in 5 years you copy to newer 50TB-100TB drives. Assuming you plan to keep files for decades or hope to bequeath them to your children you might expect to do periodic upgrades to newer larger drives so that the old files are maintained on multiple drives no older than 5-10 years.


Brian

 

[anotherlab]

Blu-ray may store 50GB and I believe there are options that increase that to 100GB so that's pretty decent, but if you have 10TB of files you'd need 100-200 disks and since redundancy multiply that by at least 2 and upwards of 4. I'm not saying it's not doable, but do you want to spend that amount of time have at it.

Looking down the road about 5 years or so I think SSD's will finally reach a price point to compete with HD's and with storage densities to permit 10TB to 20TB per drive. Then, you have 3 or so copies and in 5 years you copy to newer 50TB-100TB drives. Assuming you plan to keep files for decades or hope to bequeath them to your children you might expect to do periodic upgrades to newer larger drives so that the old files are maintained on multiple drives no older than 5-10 years.


Brian

The thought of backing up a TB of data by swapping Blu-Ray disks just makes me shudder. I'll stick to RAID and accept that I'll be replacing drives every few years or so. I don't think that SSD will be a viable long term storage solution. You can only write a SSD location X number of times before it wears out. When you buy a 120 GB SSD, it will have 150 GB storage and the the SSD controller will replace bad locations with ones from the extra space.

 

[Raptorman0909]

The thought of backing up a TB of data by swapping Blu-Ray disks just makes me shudder. I'll stick to RAID and accept that I'll be replacing drives every few years or so. I don't think that SSD will be a viable long term storage solution. You can only write a SSD location X number of times before it wears out. When you buy a 120 GB SSD, it will have 150 GB storage and the the SSD controller will replace bad locations with ones from the extra space.

The limited write issue with SSD's turns out to almost never be a problem in the real world and for video/image files the need to change them is quite a bit less than typical PC files. HD's are the preferred solution at this time but they are LESS reliable than SSD's. In addition, saving 100GB of video files can take a long time with even the best HD's that seldom exceed about 250MB/sec whereas PCIe SSD's are more than 6X faster write and 10X faster read.

In my editing PC, a desktop I built last year, I have a 512GB PCIe SSD that has my OS and programs and is the scratch disk for video editing. I also have two 6TB WB Black HD's that store my image and video files. In addition to that I have about eight 2.5 inch external HD's that retain copies.

Again, I expect SSD's will equal the $/GB of HD's in about 5 years and after that they will be cheaper per GB so SSD is the way things are going like it or not. As for me the huge speed increase and greater reliability would make SSD's preferred even at a higher price/GB.


Brian

 

[Erised]

I got a pair of Seagates 4TB portable hard drives at about $100.00 each. I store on one and the other is the back up. They cruise at about 109 MB/sec and that's plenty for me.

 

[GadgetGuy]

https://www.google.com/search?q=sup.....69i57j0l5.4234j0j7&sourceid=chrome&ie=UTF-8
Simple Google search. $50 on ebay.


$150 on amazon

But none of that changes what I said. HD storage is not a better long term storage when compared to BR.

My point was that you need to keep upgrading the media to currently available devices, for a variety of reasons. Anyone still have a floppy disk reader? HD's are cheap, and getting cheaper every year, and should be replaced every few years with larger ones, reducing the number of HD's necessary for your older archives. I had CostCo convert the Super 8 movies to digital files, so I can store them on HD's. I have scanned old negatives and slides to be able to store them on HD's, too. We'll have to agree to disagree on the "better long term storage" solution. It's the principles dscussed that matter, rather than which media.

 

[John Locke]

I personally don't like the file system of 200 BR disks sitting on the shelf, inserting them individually in a reader just to see the content. Trying to find something archived would be virtually impossible, no search function, so difficult that you really don't have a practical backup solution that's reasonably useable.

 

[JarrodF]

Just a quick note about the RAID drive option. Remember that RAID 5 is still good when 1 drive fails. When a second one goes then you loose everything. RAID 6 allows 2 drives to fail without loosing all your media. This may sound unimportant but twice over the years we've lost one drive and then a second has started to go right after. That happens when the reason the drives fail effect them all (like a power surge, brown-out, etc). We switched over to LTO tape backup several years ago and couldn't be happier. The drive is pricey but the tapes are super cheap, backward compatible and bullet proof.

 

[anotherlab]

Just a quick note about the RAID drive option. Remember that RAID 5 is still good when 1 drive fails. When a second one goes then you loose everything. RAID 6 allows 2 drives to fail without loosing all your media. This may sound unimportant but twice over the years we've lost one drive and then a second has started to go right after. That happens when the reason the drives fail effect them all (like a power surge, brown-out, etc). We switched over to LTO tape backup several years ago and couldn't be happier. The drive is pricey but the tapes are super cheap, backward compatible and bullet proof.

Tape is a great option for a redundant backup. Never put all your eggs in one basket.

 

[Zarinbk]

I have seen many replies on this thread concerning the reliability of Hard Drives over an extended period of time. While this IS true, I have found that using Spinrite from GRC.com regularly dramatically extends the useful life and can recover most HD that cannot be read by other systems.

 

[Raptorman0909]

I have seen many replies on this thread concerning the reliability of Hard Drives over an extended period of time. While this IS true, I have found that using Spinrite from GRC.com regularly dramatically extends the useful life and can recover most HD that cannot be read by other systems.

While I can't say that Spinrite is useless the fact is most failures I've had with HD's are physical failures where the drive no longer turns -- you get the click-click sound as it tries to spin up but doesn't. In fact, I have two NAS HD's that failed just recently and neither one will spin up at all.

There are services that disassemble a HD and use there own heads and/or drive mechanism to recover data but even they aren't 100% given the damage that can occur to the platters if there was a head crash or the head fuses to it. So a software only solution is going to have little value if there's a hardware failure as seems to be the case most of the time.

I've owned about 8 desktop PC's, 12 laptop PC's, and 15+ external HD's since 1982 and in that time I'd guess I've had at least about 15 HD failures and possibly more than that, and I'd guess 85% of them that died had platters that no longer spun. Software aint going to fix that!


Brian

 

[anotherlab]

I have seen many replies on this thread concerning the reliability of Hard Drives over an extended period of time. While this IS true, I have found that using Spinrite from GRC.com regularly dramatically extends the useful life and can recover most HD that cannot be read by other systems.

I do't think that Spinrite will extend the life of a hard drive. It does two things, moves data from failing sectors to new sectors and will repeated try to read a bad sector to recover the bad data.

For the former, remapping sectors when they can't be written to, that's been in hard drive firmware when since the late 80's. Spinrite isn't going to extend the life of your drive that way. It wont harm a good drive, but it's not doing anything that the drive isn't already doing on it's own.

If your drive is failing, Sprinrite does a better job of recovering bad data. But there's a caveat. Because Spinrite will try thousands of times to read a bad sector and then write that data to another sector on the same drive, it can make a failing drive worse. If your drive has is failing, use another tool first to get all of the readable data from the drive before running Spinrite. Here's a good explanation of what Spinrite does and the alternatives that should be used first.