Harddrives, Filesystems, Raid, and LVM

Hard Drives

Can you ever have too much disk space? I'll describe some tricks later (transcoding) for minimizing the disk space needed but still plan on getting as much disk space as possible (This saves frustration over having to delete your favorite shows so you can record more -- as you'll find out you record much more than you can watch). If you can get a single hard drive that will fit everything you want then you are in good shape or you don't record a lot of HD material. If you can get two drives, one to hold the OS, music, and movies, and the other to hold TV recordings that is another good option to help increase space with little work. If you want more drive space for TV recordings than what can fit on a single drive you can buy, things get a little more difficult, but I'll explain a few ways how to do it easily on Linux with raid and LVM -- If you're using Windows I'm not sure if you can do this.

What size of drive and speed

I do recommend to get at least 100GB drives as a minimum for regular TV and 300GB for HDTV. 3-5 times this minimum recommendation will be plenty for 99% of any HTPC user. If you plan on expanding your HTPC storage space, I recommend starting with a single large drive that's atleast the minimum size above. I personally run 4 200GB, 1 250GB and one 300GB drive and I wish I had gone with 3-4 400GB drives to save time in configuring, money on buying an extra PCI IDE controller, using more power for more drives, having to have a larger case and having to cool more drives and system... and so on. If you can go with 1 400GB drive for a little (or decent amount) more in cost, I highly recommend this because later when you buy another drive, the price will be lower. 7200 RPM drives should be used for both TV and HDTV. Also, the bigger the cache the better (8MB or 16MB will do).

Brands

There's nothing worse then having everything working perfectly and then have a hard drive goes bad and ruin everything. I'm going to only talk about IDE drives, since SCSI drives are in a league of their own (In performance, cost and reliability). After years of being around just about every brand and seeing how they last I only partially trust one brand of hard drive now, Seagate. Hitachi drives are like gambling, most fail pretty fast and sometimes you get a good one to last 2-3 years. Western Digital, Maxtor, and Fujitsu often prove to last a few months past their warranty and then go out, and they do it pretty regularly. Of 5 Western Digital drives I've had over the last 5 years every last one has gone bad, often after 1-2 years. Of 7 Maxtor drives I've had one last 3 months, one 13 months, one 17 months, on failing after 17 months and the others are doing okay. IBM drives really depend on that model line, since IBM oem's them from someone else.

Seagate drives are the only drives drives you're likely to find without a 1 year warranty, they have a 5 year warranty. Seagate is the only company who's produced SCSI like performance and reliability at IDE cost. Seagate used to be about 20% more expensive but lately they've hopped on the rebate train and I've seen their drives going for the same price or cheaper than other brands in the faster selling drive size areas. I do want to say I have no connections to seagate other than they're the only brand that from my experience I can trust. I did some research to find out why for those curious to read. Producing a good drive production line takes time. In the past every company had to develop their own their own unique way to do things and it cost them a lot. When new better ways of doing things came out most companies stuck to their ways for cost reasons.

Noticing disk failures and Avoiding disk failures

There are three major types of disk failures. Most drives have the ability to monitor disk performance and you can often see when a disk is starting to fail. Even without checking these you should know the basics about disk failures.

• The first is if you scratch off a piece of the disk, that metal could start to bounce around inside, hitting the head and causing it to bounce, hit, and scratch more off. If you notice your disk getting worse and worse over a few hours, days, or weeks this is likely what's going on. It's rare but possible that within a minute of scratching the first piece off it just does a number and the drive chews its self apart incredibly fast. Usually though you can hear randomly (ie, not when you're accessing the disk) pieces of metal bouncing around and you better realize you need to backup important things and get a new drive. My edcuated guess is this is the cause of disk failure 60% of the time.
• The next common problem is the servo that moves the disk heads to different areas of the disk goes out. This is related to how much you use the system. Since I record 100's of GB's a week and delete 100's of GB's a week the heads on my disks move a vast amount more than your average user and this is what I often see in many of my failing drives. You can hear this by clicking only when you're accessing the drive. Heads almost always get worse over time instead of a sudden failure. The clicking usually gets louder as the servo wears out. I've had drives take a year to fail once I began to hear clicking, others were weeks.
• The last common failure is when a motor goes out. This failure is often caused by lots of disk starts and stops. This area is also where companies often put in the extra money for reliability for your average user who turns their computer off most of the time (which is a good idea since a computer can cost $10-$25 a month to run). There's really no way to predict this failure. Usually you go to boot a system and it see's the drive but that's it. I've seen this a few times.
• Okay, I said 3, but this one I've seen happen a few times so it should just be put in for completion of causes. Circuit boards are not immortal, they can have problems -- the problems though are not the drives fault though. Any hardware in a computer can fail if it gets shocked, by static electricity from touching it or by a power surge. If you rub clothing, or sometimes your fingers all over a circuit board you can ruin it, but this is not common. The most common cause is a power surge. Power surges usually come through through the power supply in the computer, and they can fry hardware and often do. I've been victim to this three times in my life. Power supplies do go bad and they can send a surge of power to everything inside (or even nearby). If a fan goes out in the power supply it will overheat fail, I lost count of the computers I've seen with bad power supply fans and their owners are nieve to the problem, if it stops, REPLACE IT! :) If you try to run the newest video card or a couple extra drives from a power supply that isn't made to support them it can overheat the power supply and fry it. Last is when you get a surge from a power company either by lighting, blackout, or brownouts. I have seen hardware inside the computer fry other hardware. I had a western digital drive go out and it took out another drive nearby power connector. When power surges happen there's nothing you can do, but these are all instant killers to a hard drive. I personally shut down systems, even 24x7 ones when the lightning storm of the year comes through, that showing of Lost or those 50 spam mails are just not worth the time to fix things.

To know if a disk is failing you should do two things. The first is install smartmontools on your system and either set up a cron job to watch and monitor, or get into the habbit of checking disks from time to time (Worst case is if you notice one of the problems above and your system is still online you can check a drive health). You can run the commant `smartctl -a /dev/hda` where hda is the disk you want to check. This will return a lot of results and you can tell if a drive has failed, is failing, or just fine. The important thing to look for is any major failures, reported at towards the end of the report, or if the drive is reporting sectors are being reallocated. If sectors are being reallocated, the disk is likely to have been scratched and it's now a matter of time before it goes bad. Hardware error correction is fine, even for a near perfect drive. If then "WHEN_FAILED" column doesn't have a -- in it and has FAILING or FAILED then you're in a bad spot and get off what you can, it's past late. A drive though can still be starting to fail with head problems even if there are --'s the the WHEN_FAILED column. If you see any reports for errors, not performance information, the drive is begining to go bad and you have weeks to months usually before there is a problem.

If you see any errors like below on the console, with dmesg, or in /var/log/syslog you're drive could be losing data or could be fairly soon. (Note: If you're only getting filesystem errors, those errors alone do not mean you drive is going bad _unless_ smartctl also reports the drive has had problems.)

hdc: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hdc: dma_intr: error=0x40 { UncorrectableError }, LBAsect=6006647, high=0, low=6006647, sector=6006639
ide: failed opcode was: unknown
end_request: I/O error, dev hdc, sector 6006639

Cost

Going the rebate route is by far the best thing to do. The cheapest per GB right now are the 300GB drives that often sale for $99 or lower after rebate (or $.33 a GB). For larger drives, I have seen 500GB D drives for $160 after rebate (AR). I currently consider anything under $.35/GB reasonable and worth buying. Anything under $.30/GB is a very good deal. (I've seen 160GB drives go for $29 AR before) Make sure to check out my deals page that has links to online price watching sites. The best online stores are outpost.com (Fry's), Newegg.com and Buy.com (Which you can often find 10% off coupon codes at slickdeals.net). Some companies like Tiger Direct are know for problems with rebates, but slickdeals.net users always warn you if the deal is good, but rebate may be questionable.

How to get more than 3 or 4 drives in a system

Motherboards almost always only come with 2 IDE channels that support 2 drives on each channel which means a max of 4 total drives in a computer. You can go serial ATA and sometimes get more, but not always. What I did to get up to eight drives was purchase an IDE PCI card. An IDE PCI card provides 2 more IDE channels and so 4 more drives total for the price of one PCI card slot. The performance to drives connected to a PCI IDE card is still good because the limiting factor/bottle neck is the hard drive, not this card.

Using a drive as Master vs Slave

There has been a debate about having a hard drive as the slave device on an IDE channel. It is true you do sacrafice some performance as data can only be sent to one drive (either master or slave) at a time. However, I did the following tests. I setup JFS partitions on /dev/hda,hdb,hde,hdf,hdg I ran bonnie++ on each partition alone and again at the same time. (To note, My two DVD drives are on /dev/hdc and /dev/hdd) I lost about 20%-30% performance when writing to two drives on the same IDE channel at the same time, vs writing to the drives at the same time when they were on seperate IDE channels.

Picking a filesystem

I think I've hit about every pitfall possible in the 2 years of HD PVR usage. I've had 2 drives go bad in the past few months (I only by Seagate drives now, they have a 5 year warranty. Every Maxtor, and Western Digital drive I get doesn't last more than 2 years and only have 1 year warranties) and there's nothing worse than losing everything you have. There's a lot places that talk about how good a FS or partition type can be, then you find out the truth when it's too late. The first drive I had go bad was the third drive in a LVM setup. Something very odd happened and even the LVM guys couldn't figure out a way to get any of my data back on any of the 5 drives -- LVM was suppose to do this for all but that one drive that went bad. Okay, tough luck, it's been proven this can work and does, so I was just 1/1000 that couldn't. Two months later I had another drive go out so I decided I would just take it out of LVM for a few months until I can buy another drive -- Nope, I can't take it out. It turns out that XFS, JFS, and Reiser4 do not support shrinking a filesystem (they only support growing). Sure I could shrink the LVM partition but then the XFS filesystem would go bad. In order to get back to working order, XFS on top of LVM requires you to add another drive that's is the same size or larger, add it to the volume group and then "move" the virtual disk space on the bad drive to the new drive before attempting to remove the bad drive from the LVM volume group. This is just a huge pain and something not documented well. I had decided to use reiserfs from then on for the useability factor -- so come 2 months later. I deteremined reiserfs to be about the worst choice for a media filesystem. Reiserfs does very well with small files, but it's terrible with many GB files. ReiserFS was so bad that I rebuilt everything because it couldn't write even 3 HD shows at once and fast enough, and caused a lot of corrupted TV shows. My current recomendation for anyone using LVM is to use your largest drive as rootfs, swap, and media (For music, photo's video clips, etc.) and then store all recordings on other drives using LVM. The main pro to this setup is if your LVM setup goes bad you only lose TV recordings. If you need to rebuild your LVM setup then you can copy all recordings that you want to keep to the largest drive and have the space.

Pro's of FS's
• XFS has shown to have better performance overall when dealing with large (1GB+ files and especially 10GB+ files).
• JFS does almost as well as XFS and slightly better with smaller files.
• reiserfs and reiser4 do the best with small files only.
• Ext3 is slower in most all ways, but you can turn an ext3 into an ext2 fs if for some odd reason this is important to you.
• Ext2 has very good performance in all areas but it's not a journaling filesystem and is much less prefered for use today because of this.

My personal likes, after testing all of them in a PVR use is reiserfs or reiser4 for a root filesystem is probably the best bet -- files on a root FS are not gig files so reiser would work better here. XFS is my top choice for media partitions holding GB's or TB's of data, JFS is a close second, then reiser4 and reiser3 last. I don't put Ext3 anywhere because it's slow all around and fscking a TB ext2 filesystem would take probably an hour or longer. Since LVM is critical for making a single large partition for storing recordings and media files (Since MythTV requires all tv recordings to be in a single directory and all video's to be in a single directory -- but hopefully this will change in a few months?) you need to pick a filesystem that works best with LVM -- This results in the (sadly) best/only pick of reiserfs. I truely hate to admit this, since performance of reiserfs is one of the worst performers for GB sized files. The only thing that makes this acceptable is Myth doesn't need "the perfect filesystem" to work at it's best. If you're doing HD video editing then maybe you should consider XFS and note you'll hate life if a drive goes bad and you're using LVM.

The final note for filesystems is the "best" for any use is impacted (overall) more by your hardware. Your harddrive specs, motherboard, and CPU all can make more of a difference than different types of filesystems may. My example is as follows. In all my testing of various filesystems on my maxtor 200GB drives, every result was worse than any test run on a seagate 160GB drive I have. All drives have the same RPM and memory cache size. What made the difference is seagate used better technology that could write and read data faster than the maxtor drives. Overall I saw between 30%-60% better performance in the seagate drive depending on the tests.

Making one large partition with 2 or more drives -- The LVM way

Since you typically want a single directory to hold recordings, and right now a single drive often doesn't provide enough space (1-3TB's), there are a few ways you can combine drives (In Linux) to do this. The best way I've found is using LVM (Logical Volume Manager). In short, LVM provides what looks like a single large partition and hides to the user how it's mapping data between various smaller partitions or drives. LVM works by having you partition all the space you want for media storage (not your root FS) as LVM paritions. You then format the LVM partition (aka Volume Group) as say an XFS, JFS or reiser FS (Make sure to read about the pro's and cons of different filesystems below before picking one for LVM). You mount the LVM partition like any other drive would be and you're done. Some nice features about LVM are that you can resize the size of the filesystem and LVM partition (depending on the filesystem type) plus add or remove drives and sometimes when the filesystem is even mounted. If one drive fails, you may only lose data on that drive and you move on (Unless it's the first LVM drive then you have to do a bit of work and pray hard to get the other data working again). One of the biggest pluses of LVM is you can mix and match any size of drives or partitions sizes, something that raid doesn't do. It is possible with LVM to shrink a FS, but this may not be worth it. If you wanted to make a new partition for storing say home directories or DVD colelctions, just shrink an LVM filesystem and use the new unused space to make one on the fly. The catch is shrinking only work son EXT2 and ReiserFS, both are bad choices for performance of large (Many GB) files. I've run all FS's under Linux and use XFS or JFS now.

I'll explain what I did with LVM to help you understand things better. I took my 300GB drive and made a 2GB swap partition and a 10GB root FS partition (xfs) to hold the OS and everything but media related files. The rest of the space on the drive (288+GB's) I made an xfs filesystem for music, video clips, etc. On the other 5 drives in the system I only made one LVM partiton on each drive. I installed with Debian Sarge on the 10GB root fs, used the 2GB swap partition for swap. I was now left with 5 unused LVM partitions and a fully running Linux system. I used the LVM tools to build a volume group with the 5 available LVM partitions and told it to make a single XFS filesystem from these. I then made a directory /media and ran `mount /dev/mapper/media /media` to mount the 1.05TB LVM partition. Pretty simple. Sarge does have the option to make the entire LVM setp on install, and I recommend that for anyone new to LVM. Suse also supports building LVM partitions on install, and it's very well done.

Some drawback of LVM are it's striping feature and it's bootability. LVM does support striping (Basically raid 0). However, if you use striping you can not resize the file system. I believe being able to add and remove drives is more important than trying to improve performance with striping. I ran raid 0 for storing recordings until Jan of 2005 because it is no longer needed. In the past, programs such as MythTV wrote data out as soon as it got it. This posed a problem when having 4 or so HD tuner cards. Each card would send about 45Mb/s of data, so 180Mb/s or 22.5MB/s for four cards. 22.5MB/s is too fast for a typical single drive to handle for writing. MythTV now has an internal buffer and writes data out as it can. I've tested 6 cards writing to a single hard drive and an XFS filesystem (Not LVM) and it worked just fine. I have been recording from 4 HD tuner cards, transcoding and commercial detecting streams, and sending the stream to a networked frontend just fine for over a year, all at the same time and to a non-striped LVM XFS filesystem. I just mentioning this drawback because maybe there is someone with 12 HD tuner cards and will need to use striping. If you want striping, or redundancy you can use Raid 0, 1, 5 or 10. You can also use raid 1 and put LVM on top.

Right now, you can't use LVM for your root partition. Although work is being done and there is a Grub 1.5 stage boot loader, it's still a work in progress. In a year or so hopefully we will be able to boot from LVM partitions, but for now you will need to have your root FS on a non-LVM partition.

Raid (Software and hardware) -- The other way of making a large filesystem (This is here for history and just incase, but I don't recommend building a HTPC this way at all, use LVM)

Software raid gives you the ability to make more than one drive look like one large drive or to duplicate data between drives in a very optimized/speed way. Raid is a little tricky to setup though.

When I first built my large MythTV system I chose to do raid 0, 1 and 5 to achieve various advantages. I allocated 256MB of raid 1 for my Linux root filesystem (Fully redundant and to boot from since grub/lilo can only boot froM RAId 1), 160GB of raid 5 for /home /usr /var /music /backup (Fully redundant but you lose n-1 drive space so 5 drives*40GB=200GB then you minus one drives space (40GB) and get 160GB usable), and 800GB of raid 0 (Stripping which is great for speed, but no redundancy so if one drive goes bad I lose all the data) for HDTV recordings. I can save about 200 shows, or 12 days worth of TV, a bit more than most people would care about having around. I record an average of 20 shows a day now and care about keeping 1 or 2 usually (This is averaging 4GB/hr for HDTV).

mediabox:/# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath]
md2 : active raid0 hdg3[4] hdf3[3] hde3[2] hdb3[1] hda3[0]
      799072640 blocks 128k chunks
       
md1 : active raid5 hdg2[4] hdf2[3] hde2[2] hdb2[1] hda2[0]
      156280064 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
       
md0 : active raid1 hdb1[0] hdf1[1]
      256896 blocks [2/2] [UU]

mediabox:/# cat /proc/partitions
major minor  #blocks  name
 
   9     0     256896 md0
  33     0  199148544 hde
  33     1     257008 hde1
  33     2   39070080 hde2
  33     3  159814620 hde3
  33    64  199148544 hdf
  33    65     257008 hdf1
  33    66   39070080 hdf2
  33    67  159814620 hdf3
  34     0  199148544 hdg
  34     1     257008 hdg1
  34     2   39070080 hdg2
  34     3  159814620 hdg3
   3     0  199148544 hda
   3     1     257008 hda1
   3     2   39070080 hda2
   3     3  159814620 hda3
   3    64  199148544 hdb
   3    65     257008 hdb1
   3    66   39070080 hdb2
   3    67  159814620 hdb3
   9     1  156280064 md1
   9     2  799072640 md2

Using hdparm, bonnie++ and other benchmark tools I can sustain a good read of 96MB/s over the 5 drives and writing at 63MB/s. That's 14 times what I need to record 3 HDTV shows and watch 1 all at the same time. I may sacrafice some speed using two hard drives on the same IDE channel but the drives, even sharing IDE channels, provide more transfer bandwidth than I need. Recently I did some more testing, comparing 3 drives on seperate IDE channels and the read/write speeds were almost identical. Someone wrote in to tell me that I am maxing out the PCI data bus and that is the bottle neck and there's nothing you can do to get over this speed, but it's not needed anyway.

Next Page