Large storage needs

k1962

We will be setting up direct storage, consisting of 450GB drives, in RAID 5 to store and serve 4TB of aerial photo projects. We need almost all the disk space so I was wondering just how full is too full? At what percentage full does performance go down. Storage is not my specialty as you can probably guess.

I'd like to configure the 12 disk enclosure as one RAID 5 with 11 disks and a spare. Others feel that 11 drives is too many and that we should configure it as 2 5 disk RAID 5 arrays. Part of their logic is that if we lose 2 disks at the same time we only lose half the data. And therefore only need to restore that half. Can anyone point me to a white paper on best practices for setting up storage?

The datacenter we are forced to use has never dealt with a project like ours. Their business model is not designed for large volumes of data. The yearly cost for storing our data on the data center SAN is such that the direct storage pays and support pay for for itself in less than a year. Don't you just love government

We also can't aford the $3.50 per GB to backup the data to tape so we are purchasing a second inclosure which will be filled with 1TB SATA drives. We will be using 2nd enclosure for backing up the data. This is not how we want to do things but it is our only option short of not doing the project in the first place. BTW, the nature of the data is that it never changes.

Tyrant1919

How many users are accessing this data at once?

blargoe

If losing 2 disks is a concern, you could just set up RAID6. Writing performance would go down, but if you're primarily serving static content and they're not changing it, shouldn't be a big deal.

Depending on how many people are using this enclosure, even using SATA on the first enclosure might be OK.

Tyrant1919

That's what I was getting at. If the number of users is very low, I'd go to 1TB or 2TB drives. Hell, 4 2TB drives in raid 5 and you're set. 5 Drives in raid 6 if you want two drive protection. But if you're backing it up, depending on your needs, the extra protection may not be warranted.

tiersten

k1962 wrote: »

We need almost all the disk space so I was wondering just how full is too full? At what percentage full does performance go down.

Doesn't affect it. You've got static data anyway.

k1962 wrote: »

We also can't aford the $3.50 per GB to backup the data to tape so we are purchasing a second inclosure which will be filled with 1TB SATA drives. We will be using 2nd enclosure for backing up the data. This is not how we want to do things but it is our only option short of not doing the project in the first place. BTW, the nature of the data is that it never changes.

How certain are you that the data will never change? Does this include whatever else is running off the data? I hope that you're still backing it up on to something other than another enclosure in the same datacenter though.

What level of performance do you need from this? RAID5/6 requires a large number of spindles for decent performance. If its only a handful of people accessing these files at any given time then a low number of drives will be fine but if there are more then you'll need to consider performance.

If you're concerned enough to warrant using RAID, check that the drives you're buying are actually rated to run 24x7. Regular desktop drives whilst cheap aren't actually rated for that.

dynamik

Have you thought about getting larger drives and going with RAID-10?

astorrs

Whenever you're calculating a storage solution you need 3 things.

1st is the space required and approximate growth - it sounds like you have an handle on that.

The second is IOPS (I/Os per second), how much performance do you require from the device - hard numbers are best here - where is the data currently stored and can you provide some number from that? (if you need help gathering this we can give you some pointers if you tell us where its currently stored)

Finally you need to understand the availability requirements, again you have an idea, but putting it in business terms is better - e.g. "Recovering from a backup in 24hrs time and losing a couple days worth of data is okay" vs "Downtime on this data is unacceptable and will cost the company X dollars per second in lost business"

Without those 3 things you are just guessing that a particular solution will meet your needs.

dynamik

Have you thought about getting larger drives and going with RAID-10?

blargoe

How about RAID 10, has anyone suggested that yet?

dynamik

Aw, see how you are. I came back to this tab and saw a DB error, so I just hit refresh

UnixGuy

I always recommend RAID 10 + Hot Spare for our customers. Best performance and highest protection. Try to convince your company, tell them that it gives best performance and best protection and once setup, you practically forget about it (less support).

Tyrant1919

I agree, 10 rocks.

astorrs

RAID 10 makes sense - when it makes business sense - it can also lead to costly overprovisioning; everything has its place, hence why I asked for more details.