Using RAID and S.M.A.R.T to save yourself from data loss and lot's of grief

I have been configuring SMART monitoring to lot of my servers and workstations, both work and home, lately. Especially when combined with RAID, SMART can help you to avoid disasters that would occur when you lose your data in an event you hard drives break.

Short primer
RAID means "Redundant Array of Independent/Inexpensive Disks" and is mostly thought something you would do on hardware level via a separate RAID-controller or motherboard that supports RAID. RAID is used to get performance improvement for your disk operations, which is the biggest bottleneck in modern systems most of the time, or to provide redundancy when you lose your drive to hardware failure.

With Linux you can use something called 'software RAID' which does not require expensive RAID-controller nor support from motherboard. All you basically need is more than one hard drive. With two hard drives you can already setup a RAID0 or RAID1.
RAID0 means that you distribute your data on two (or more) drives and get a huge performance benefit, but the downside is that if you lose either disk to HW failure, you lose all the data on both disks.
RAID1 is quite the opposite of RAID0, RAID1 mirrors the data on first drive to second drive, which means that write operations are somewhat slower but the data is now duplicated and thus safe from HW failure.
There are several other RAID levels too, like RAID5 which requires 3 drives minimum and provides some redundancy in case of disk breakage and some performance improvements. My personal favorite currently is RAID10, which is a combination of RAID0 and RAID1, meaning that you get quite good redundancy in case of failures and quite nice performance boost also. The downside of RAID10 is that you need 4 drives to get started and 'lose' two of them for the mirroring.

For more information about RAID levels check this wikipage and for information how to create RAID the hard way (remember it is easiest to create during installation by the distros installer) check out Linux Journal's article.

SMART (or actually S.M.A.R.T) on the other hand means "Self-Monitoring, Analysis, and Reporting Technology", which is a technology most, if not all, modern hard drives support. With SMART enabled drives you can gather information straight from the independent drive(s) and use that information to predict when your disk fails of old age or otherwise. Hard drives have gotten lot better since infamous times of IBM 'Deathstar' hard drives, but still the fact remains: hard drives will die of old age sooner or later. If you can get a advance warning of this impending doom for your drive, you have time to make the necessary preparations, like making backups or even replacing it before you run into risk of losing your data.

For more information check out the excellent article by Linux Journal

Getting started

I am not going to show you how to create a RAID system, it is quite easy to do when you install your operating system, openSUSE installer for example provides nice tool for creating RAID on which to install the system.

Note! You will need super user (root) privileges for (almost) all the steps below.

Step 0

Anyone can start using SMART right away, as it requires neither special hardware or special setup. Some systems might require that you enable the SMART support from your computers BIOS, so check that first. Next step is to make sure you have package called "smartmontools" installed, when you do, you can try running a command like this:

smartctl -a /dev/sda

This should output lot of information about your hard drive, but if you have not enabled SMART testing this information will be of little use. Next you will need to enable SMART testing and data collection to be automatic and done in timely manner.

Step1

First step for this is to edit config file for SMART. Usually it is /etc/smartd.conf. Open it in your favourite text editor and first comment out anything that already exist there, usually this line:

DEVICESCAN -d removable

Now you can add configuration for your drives in the end of the file. Here is example from my workstation which has 4 drives in RAID10:

#Run every Sunday offline, Saturday Long and every evening conveyance and morning short test

/dev/sda \

-H \

-l error -l selftest \

-s (O/../../7/02|L/../../6/02|C/../.././20|S/../.././01) \

-m NotUsedNow -M exec /usr/local/bin/smartd.sh -M once

#Run every Sunday offline, Saturday Long and every evening conveyance and morning short test

/dev/sdb \

-H \

-l error -l selftest \

-s (O/../../7/05|L/../../6/04|C/../.././21|S/../.././02) \

-m NotUsedNow -M exec /usr/local/bin/smartd.sh -M once

#Run every Sunday offline, Saturday Long and every evening conveyance and morning short test

/dev/sdc \

-H \

-l error -l selftest \

-s (O/../../7/08|L/../../6/06|C/../.././22|S/../.././03) \

-m NotUsedNow -M exec /usr/local/bin/smartd.sh -M once

#Run every Sunday offline, Saturday Long and every evening conveyance and morning short test

/dev/sdd \

-H \

-l error -l selftest \

-s (O/../../7/11|L/../../6/08|C/../.././23|S/../.././04) \

-m NotUsedNow -M exec /usr/local/bin/smartd.sh -M once

In above configuration tests that take long time to complete are run during weekends and shorter test are run daily. You can modify the starting time by modifying the last number in the expression.

For example this part "O/../../7/11" translates to "T/MM/DD/d/HH", first letter being the test type (Offline, Long, Conveyance or Short), rest of the fields are for scheduling the test, in this particular case we run Offline test every 7th weekday (Sunday) at 11:00 (AM for those with 12hour time disability).

The second interesting part from the above configuration is the "-M /usr/local/bin/smartd.sh" which basically tells what to do in case of problems, here it is setup to run the script /usr/local/bin/smard.sh which you will have to create in Step2.

Lastly you will need to create the above configuration only for those hard drives you actually have in your system. As mentioned I have four disks (sda, sdb, sdc and sdd) and you might have less. To see your drives you can use the following command:

ls /dev/sd?

Step2

Again with your text editor create a file /usr/local/bin/smartd.sh and paste the following and replace your email to the appropriate place:

LOGFILE="/var/log/smartd.log"

echo -e "$(date)\n$SMARTD_MESSAGE\n" >> "$LOGFILE"

mail myemail.address@somehost.com < $LOGFILE

After you have created the file above, you need to make it executable by running following command:

chmod +x /usr/local/bin/smartd.sh

Extra Step for Ubuntu users

Edit the following file /etc/default/smartmontools and uncomment the following line:

start_smartd=yes

This will make the SMART daemon start automatically during boot.

Step3

Either restart the SMART service or reboot your computer. Restarting the service works like this in openSUSE:

/etc/init.d/smartd restart

Or in Ubuntu:

/etc/init.d/smartmontools restart

That' s it, almost

Now in theory you will get an email if SMART detects problems with you hard drives, but that requires your mail daemon to be in working order. You can test this simply by executing the script we created:

./usr/local/bin/smartd.sh

You should receive email containing the empty smartd.log.

If your email notification is not working, you probably need to enable the mail daemon, but that is quite distro specific and you will need to figure it out by yourself.

Other option is to use somekind of monitor software instead. Ubuntu and Debian users can use Smart-notifier application and KDE users can use Plasmoid called Plasmart.

Now hopefully that will some day save you from disaster of losing your precious files. But in the meantime remember that neither RAID nor SMART replaces making backups, they only safeguard you against hardware failures, you will need traditional backups to safeguard against user, application and operatins system errors!

Beginner's guide to Linux: openSuSE 11.1 exploring the desktop and system configuration

In previous screencast we demonstrated how easy it is to install modern Linux distribution (openSUSE 11.1 to be exact). In this screencast we explore a bit the newly installed system and the new KDE4 desktop. We also learn how to access YaST, a superb system administration tool and how to change display resolution with it.

KDE is one of the major desktops available for Linux, the desktop means in this case the graphical interface that user sees in front of him when he is using his openSUSE system. KDE went through major changes from the "old" KDE 3.5 series to the new KDE4 series, to a point that the new KDE is a complete re-design of the desktop metaphor. KDE has always been my choice on desktop and because openSUSE does great job 'releasing' KDE it has only deepened my admiration for the openSUSE project. KDE.org has a nice visual guide to KDE4, check it out!

YaST is no doubt one of the single most greatest aspect of openSUSE Linux distribution. People that have not used it seem to think that it is yet another package manager (if that was the case it would be called YaPM). Truth is that YaST is so much more than simple software installation tool, it is everything that is missing from most other Linux distributions and that is a centralized system administration tool that makes configuring your system easy. Want to partition your drives? Simple, use YaST. Want to setup your monitor to correct resolution? Again thats simple just launch YaST. Want configure file serving or other server functionality? YES, use YaST!

In video below I'll shortly present the two aspects I like most about openSUSE, as described above they are: great KDE desktop implementation and YaST. By KDE implementation I am not saying that openSUSE devs did all the work, but they compile a great release with fixes and polish to the KDE code.


Notice, the above video should load, by default, in HD mode. So just ignore the recommendation on the video to "turn on the HD resolution".

HTPC reincarnation

Three weeks ago I placed an order for couple of HTPC suitable terabyte hard drives. It seems that hard drive selection has grown quite bit since last time I was shopping for drives. Seagate has long been my favorite, because of their long warranties and my personal experiences with their drives. So this time my choice was Seagate's LP (for Low Power) drives with spinning speed of "only" 5900 RPM. In exchange Seagate is promising lower wattage, less noise, less heat and better reliability for these drives compared to other desktop-grade hard drives. To me they seem ideal to my HTPC usage, they are only Seagate terabyte drives that have five years warranty and quite cheap (84€ each).

So finally getting those drives means that I can start messing around with my HTPC once again. As I mentioned in some older post, it has been working too well lately and I been kinda wanting to get something to fiddle with. I've learned from my last HTPC upgrade though and I am keeping my old installation parallel to my new installation, so we can watch TV and do the rest of HTPC stuff during the next few weeks.

Enter Debian
I've decided to try Debian this time, instead of the old trusted Gentoo. With Gentoo I seem to always to break the Portage in the end to such state that I don't want to install or update anything in fear of breaking something critical. Last episode was unsuccessful Python update and the result a only half working Portage.

As always, it has been a pain in the ass to get my machine to boot from any installation media. This time around it does not seem to care for my boot CD's and I resulted to installing Debian Lenny from USB memorystick. I wanted to install Debian Testing from the start, but I had trouble with the USB installer, which was complaining about disparency between Linux Headers on the installation image and the running kernel from the Debian Installer. I could not find a easy solution, so installed the stable Lenny and upgraded to Testing first thing after installation.

I also managed the most complicated partition setup to date (personally). I wanted to have several RAID1 devices and a LVM volume, which made the partition count reach 12!
So here is what I did:
-/boot 100MB on RAID1 with EXT3
-/ 20GB on RAID1 with XFS
-Swap 4GB on RAID1
-/htpc 20GB on RAID1 with XFS (this partition houses eg. cache data for MMS multimedia libraries)
-/safe 200GB in RAID1 with XFS (this partition is for backups, mainly our photos)
-/video 2x750GB in LVM with XFS (this partition will contain solely PVR DVB-recordings)
The reason for using LVM with /video is that I can add more capacity easily later on once I manage to fill the the 1.5TB. 1500GB sounds quite a lot, but I can fill that up quite easily by running automatic timers for our household. I also do not consider the recordings critical enough to warrant RAID mirroring.

Next up, compiling a custom kernel, hopefully it will be quite easy to have the latest 2.6.30 version.

Beginner's guide to Linux: openSuSE 11.1 installation guide

It is evident from the earlier blog entries, that I've been into making screencasts lately. I've started a series of screencasts called "Beginners's guide to Linux", in which I use openSUSE 11.1 to demonstrate how to install and use a modern Linux operating system. Of course the choice of Linux distribution and the desktop environment can be argued to infinity, I've decided to use what I use currently myself and following videos will feature openSUSE 11.1 with KDE4 desktop.

There are several ways (the case always when dealing with Linux) you can choose to install openSUSE 11.1. First you could download the LiveCD's, so called because you can boot your computer from them without installing anything, although you can actually install a basic system from the LiveCD also. Nice thing about LiveCD's is that you get to see a preview how you desktop looks and behaves, how openSUSE detects your hardware and so on, without actually installing anything. Keep in mind though that booting to a LiveCD environment will present you with slower system and with less functionality than you would get with doing a real hard drive installation. If you want to try out a LiveCD, you need to choose between the two (main) Linux desktops, KDE and Gnome. Second way to install, which I like the best, is to download the DVD and install using that. The DVD offers a better installation program, more choice in software and you can choose between several different Desktop during the installation. Head to http://software.opensuse.org/ to download the CD or DVD image.

First screencast starts where any Linux beginner would, installing the system. While it is nowadays very easy to install any modern Linux distribution, there might be some points like partitioning where beginner can feel out of their depth. The video is fast forwarded during the tedious bits of installation, so expect to spend something like 20-60 minutes when installing it, depending quite a lot about horsepower on your computer. On this screencast we are using 32bit DVD.



Notice, the above video should load, by default, in HD mode. So just ignore the recommendation on the video to "turn on the HD resolution".

Secrets of screencasting in Linux

"A screencast is a digital video recording that captures actions taking place on a computer desktop. Screencasts, which often contain voice-over narration, are useful for demonstrating how to use specific operating systems, software applications or website features."
Definition by: WhatIs.com

To be honest, there is nothing mysterious about screencasting on Linux operating system. Plenty of applications exist from recording your desktop to audio and video editing. But the small advice I can give is in form of applications that I've tried and found out to work best for my needs.

For me the workflow is only three phased, phases being: recording the video, editing the video and adding audio, and finally uploading the video Youtube and creating captions. Because I am not doing any voice-over narration (my english is not that good) I can skip any advanced audio editing that might be otherwise required.

Step one of the workflow is to record the video from your desktop. I've experimented with few applications and found out that application called recordMyDesktop suits my needs best. recordMyDesktop gives me stability, simplicity and performance that I require and most importantly it gets the job done. Changes are that you can find this application straight from your distributions software repository. recordMyDesktop saves the video to Ogg Vorbis Video format (.ogv) but on step two you can convert it to pretty much anything, besides at least Youtube accepts ogv-files just fine.

Step two is to edit the recorded video and add some audio to go with it. The application I use for this is called Cinelerra, it is a long standing video editing software for Linux and might actually be a bit "overkill" for screencasting needs. Previously I've edited a 7 minute making-of documentary of indie film called "Korpinkieli ja Vaeltaja" (website in finnish only) using Cinelerra.
Although it has been over three years since then, Cinelerra still feels very familiar and is now considerably more stable. Check out the tutorials on Cinelerra's website to get going, I assure you that once you get going it is very nice tool. After the video is edited it is time to add audio track. Because I'm no good as narrator, I've only added some Creative Commons music to the video.

Step three starts with uploading the video to the chosen video host. I am using Youtube, but of course there are also alternatives like Vimeo. After the video is uploaded I usually create the captions using the Youtube's caption editor. After that the video is ready for publishing.

Plans for the future

Everything has been working very nicely in my current HTPC installation since it's installation early 2008. Because of that, I think it is time to upgrade, the proverb "Don't fix it if it ain't broken" does not seem to ring any bells for me. I guess there has been too little tinkering lately.

New hardware?
Anyway, I am not planning any major hardware upgrades, I've been thinking of getting myself one of those small Logitech DiNovo Minis. But besides that I am mainly thinking about software upgrades, and plenty of them. I am still thinking of the best way to do the complete re-installation. I will be keeping the old installation as a backup, because from the past experience I know there will be some problems and delays before I get to same state of HTPC-nirvana I am with my current installation.

It remains to be seen if I figure out a way to do this without buying new harddrives, but it might be the safest bet to get two new (and bigger) drives from shop. It would also enable me to change my old Seagate disks to Seagate special DB35 or SV35 class of harddrives, which are designed to be run on HTPC, NAS or other 'embedded' systems. I have those already in my NAS and they seem to do what they promise, ie. low noise, low wattage and low temp, also because aforementioned things they should be more reliable.

New software!
On software side I have some grand plans.
First of all I want to install the latest Gentoo 2008 and the latest Linux Kernel. I want to also create RAID1 that I previously had, but also create separate LVM setup that houses recordings, this LVM shows itself as a single filesystem for VDR which can be quite difficult at times when using multiple filesystems for saving the recordings. Also LVM enables me to easily extend the space by adding new drives to LVM and thus expanding the space available for my recordings.

"Witness the power of this fully armed and ...
eh I mean "full upgraded" and operational HTPC."

With the new Gentoo and kernel I want to test out new ways of making the boot process even faster. Somethings called baselayout-2 and openrc should speed up it quite significantly. Who knows maybe the new kernel has better support for my motherboads ACPI and I can also enter Suspend2RAM and actually resume from it.

New kernel means that I have to once again change my kernel config, which gives my a good opportunity to take some features into use that I have previously disabled. I want to have AutoFS available to allow me easily mount Samba and NFS shares from my network, even if the shares are not available at boot time. I also will need to enable NFS support, NTFS support and FAT support in my kernel, to enable mounting of network shares and USB connected drives.

I also will probably update to latest Nvidia driver which enables hardware based video decoding, this is done with NVIDIA VDPAU support that was added in the latest stable driver 180.22. This also leads me to upgrading my Mplayer and Xine to enable support for the VDPAU. When compiling new Mplayer I need to keep in mind that I should enable mencoder make flag, that way I can use mencoder to automatically encode my DVB recordings via scripts.

Two of my main HTPC softwares are also in for a major upgrade, this is if I can wait until the 1.1.0 final of the MMS will finally by released. VDR has also received a major upgrade from my old 1.4.7 version as the 1.6 branch has gone stable. Especially the new VDR I fully expect to give me lots of headache, I run quite many plugins in 1.4.7 and I am not completely sure of their compatibility. 1.6 version of the VDR also enables me to use UTF8 finally, which should help me with filenaming problems. With these two upgrades I will also need to change my lirc setup and enhance the functionality even further.

Finally I plan to improve the fault tolerance of the system by duplicating the system partitions in that way, that if the first system fails for some reason, it will use the backup system automatically. I have done this with good results with my parents VDR setup, where the main disk is showing signs of ageing.

MMS 1.1.0 nearing completition

MMS picture browser in action.
Release candidate 7 (RC7 for short) of the MyMediaSystem 1.10 was released yesterday. This might actually also be the last RC before the final 1.10. Before final release the build needs to be tested and the bugs need to be squashed, but I have to say I so far have not found any. I have skipped last three of the RC's because my system has been working very well and I've been consciously trying to avoid fixing anything that is not broken. Also there has not been so important features (for me) to be worth the hassle. RC7 is completely different matter though and it actually has two "must have" new features that I could not just pass by.

The user interface in RC7 has had a major upgrade, it finally scales nicely to high definition TV resolutions. I use mine in 1080p (1920x1080 pixels) and previously the interface components, like text and icons were very small. But now the interface scales nicely to higher resolutions, my main menu is still a bit crowded but that is mainly because as experiment I enabled almost every feature in MMS, now I will just cut down to what I use.

Also the selection indicator in the browsers (video-, audio- and picture-collections) is now much clearer, it zooms the selected icon to be larger and it now makes it a lot easier to browse a big collection. The interface also seems to be more responsive or it might just have to do with the improved lirc-configuration, but anyway I am very pleased with the new interface.


Go here to see the complete changelog and list of new features. At the moment the download host seems to be down, but there is a working link to the source code few postings down.

Weather-plugin is old news, it was already included in previous versions,
but it now integrates to right hand corner notification area
(as demonstrated in the audio browser screenshot) with clock.

Quickie: Setting NTP

If you are doing DVB recordings via timers like I am, then you probably want to set up ntp-client to update the systemtime in startup.

During this past month that I have been slowly building my HTPC, systemtime is already ahead by five minutes. That means basically that every timer now starts recording 5 minutes too early and ends early also. I have, of course, setup extra time before and after every timer, in fact VDR does this by default. But if the time gets ahead by more than 10 minutes, my recordings start to end prematurely, which of course is not very nice.


So ntp-client to the rescue! Here is short how to, in which we are using NTP-server in Finland, please find one closer to your location.

echo "net-misc/ntp caps" >> /etc/portage/package.use
emerge -a ntp

Edit /etc/ntp.conf and change the server to one close to your location. For us Finns it is "server fi.pool.ntp.org".

Edit /etc/conf.d/ntp-client and change the NTPCLIENT_OPTS to following (again this server is in Finland)

NTPCLIENT_OPTS="-s -b -u fi.pool.ntp.org"

Test the ntp-client by issuing date commands before and after starting the ntp-client. System time should be updated. If it works, set it to run every boot by adding it to default runlevel.

date /etc/init.d/ntp-client start date rc-update add ntp-client default

If running time sync every boot is not enough for you, then you can set up the ntpd. But do not setup the ntpdate to be run by cron, here is why.

Installing HTPC software (part 3)

In this entry we will have a look at how to install VDR (Video Disk Recorder) on Gentoo.

This time we are not going to utilise Gentoos excellent emerge-tool, but we are going to do it manually. Main reason for compiling VDR by ourselves, is that this way we get the version we want, with patches we want. For the first time it may be easier to install VDR via emerge, but later on, if you want to play more with VDRs plugins, I suggest that you compile it by hand.

Patching VDR can be little bit tricky, so I am going to use my old pre-patched version of 1.4.7 VDR source code. As with MMS there is also unstable release of VDR 1.5, this time however I am going to install the old and proven-to-be-stable version. You should probably download vanilla source code, that way you can patch it yourself if you need to and if not, you can keep the changes to minimum.

First thing is to take care of the dependencies, emerge libcap and jpeg. Those two are required for VDR to compile. Now download the source code (I am using root to run VDR, bad policy, I know) in to /root, then extract it. Now I have following directory /root/vdr-1.4.7 and I will create following symlinks, these are also needed for compilation. After that you can just issue "make && make install" in VDR directory.

ln -s /usr/src/linux /root/DVB
ln -s /root/vdr-1.4.7 /root/VDR

Compiling the plugins is a bit more involved. You must extract your plugins into /root/VDR/PLUGINS/src directory. That is not enough though, you also need to create a symlink for every plugin, removing the version numbering. For example this is what you would do for subtitles-plugin

cd /path/to/vdr/PLUGINS/src
wget http://virtanen.org/vdr/subtitles/files/vdr-subtitles-0.5.0.tgz
tar xvfmz vdr-subtitles-0.5.0.tgz
ln -s subtitles-0.5.0 subtitles

Do the same for all your plugins and modify the process accordingly, then just run "make plugins" in /root/VDR . If everything compiles correctly you need to copy the resulting files in /root/VDR/PLUGINS/lib to /vdr/lib, create the latter if it is not there already. (Notice that if you are compiling xineliboutput-plugin then you need to run "make install" in xineliboutputs own directory.)

Now you are set to go, meaning you can start the hard part, configuring VDR and plugins.

In next entry, configuring VDR and adding the needed scripts.

Installing HTPC software (part 2)

OK kids, today we are learning how to install the MMS (that is short for My Media System kids) the ´Hard way´.

Nice screenshot showcasing new feature, movie thumbnailing.

Kidding aside, installing the latest, unstable release 1.1.0, is not that hard either. Thanks to nice people in Gentoo (and MMS) community we already have mms-1.1.0-rc1.ebuild available. Using the ebuild the installation is pretty easy, although it has few more steps to it, than the basic stable installation I talked about in part 1 of "Installing HTPC software".

First thing we need to do is create a local portage. Make sure you have /usr/local/portage directory available and create it if not. After that add the following line in to your /etc/make.conf, if it is not already there.

PORTDIR_OVERLAY="/usr/local/portage"

Then create the following directory structure

/usr/local/portage/media-video/mms

In the directory mentioned above, run following command to download the ebuild. (Sorry about the small print, but the blogger.com loses few letters from it if it is any bigger.)

wget ftp://83.142.84.214/gentoo-ebuild/mms/mms-1.1.0_rc1.ebuild

Then you need to make few changes to package.keywords and package.use files. The first one adds CPU-type detection for MMS package installation and another one tells emerge to use latest available version of MMS.

echo "media-video/mms cpudetection" >> /etc/portage/package.use
echo "media-video/mms ~*" >> /etc/portage/package.keywords

Next we "digest" the ebuild to create correct Manifest, then we are going to emerge the MMS.

ebuild mms-1.1.0_rc1.ebuild digest
emerge -av =media-video/mms-1.1.0_rc1

Now emerge should run its course and you should have functioning MMS as a result. Configuration files reside in /etc/mms directory, so start from there, some configuration is needed in several files. You should be able to figure out the needed changes, which of course, depend on your setup.

Before we can actually run the MMS we need to install X server, all we need is the most basic of graphical mode. I for example have no need for a desktop environment like KDE or Gnome. Installing basic GUI is as easy as typing "emerge xorg-server" after that you need to configure the graphics settings in /etc/X11/xorg.conf, but I will leave that to you, since every hardware configuration has it´s own settings. However here is a sample xorg.conf for Mirai DTL-642E500 television with Nvidia graphics adapter.

Once the xorg.conf is ready, easiest way to start MMS with X is to edit /etc/xinitrc file this file gets executed when X is started. So add a line like one below to start MMS automatically on X startup.

/usr/local/bin/mms

More on my MMS configuration in next entry (or one after that), you should be able to go on by yourself, configuring your /etc/mms settings to way you like.


Disclaimer: Thanks for the ebuild and advices on how to get it to work goes to Vitalogy in MMS forums.

Trying to evolve

We Finns have proverb about learning by "kantapään kautta", basically that means that we learn from our mistakes. This must be the one of most painful methods of education, but it sure is effective. If you are wondering what I am babbling about, then please read my first blog entry, because this entry is about BACKUPS.


An easy and pretty fast way to make a backup of Gentoo installation is by using script called Stage4. This archives your root partition either into single or multiple gzip's or bz2's. The script is easy to use and does not need much configuration. I am currently running it with default settings. During my first backup my root partition used 2.7GB and was compressed to 552MB. I am storing the backup-archive on my workstation, because even I cant wipe out two computers at once.

You can download the stage4-script from here. I advice you to read through it, the configuration is in the beginning, it is very well documented and is not that long. But basically you just drop the file to /usr/local/bin, chmod it to have execute rights and then execute the script. Script reports problems if there are some and the output is pretty self explanatory. Hopefully you don't have to learn the restoration process, but if you do, it is documented in the wiki.

Once my installation is ready, or at least "more complete", I will be writing a script that makes stage4 archive every week or so, and then transfers it to ftp-server. I also am planning on cloning the whole partition with a tool like dd or partimage, latter being a better (and more complicated) solution, because it does not backup the empty space on a partition. Current situation with my mirrored root drive gives some protection against a hardware malfunction, but offers only little against user mistakes.

So for the time being I am making stage4 archives from my root partition as backups and if you are using Gentoo, I advice you to do at least the same.

Installing HTPC software (part 1)

I'll start this one of with a quick and easy way to get samba file sharing working. If there are no interest in having such service, then you can skip over the Samba part. After the Samba installation I will install software that is needed for the HTPC functionality.

But first, here is a make.conf file that I am using. You may want to edit it to better suit your needs. It has basicly USE-flags enabled that "sound" like they might be useful in HTPC installation.


Samba
Before emerging samba, there is again a file in need of an extra line. This time easiest way to add it, is by running following command

echo "net-fs/samba acl -cups -doc -kerberos -ldap -oav pam -python -quotas readline -selinux winbind" >> /etc/portage/package.use

This will ensure samba will be emerged for just basic file sharing, with nothing too fancy like cups or ldap. Modify as needed. After the above line is added, just emerge samba. After all that is done, it is time to edit /etc/samba/smb.conf to suit your file sharing needs.

Here is a little example of how my smb.conf looks like

[netlogon]
comment = Network Logon Service
path = /var/lib/samba/netlogon
write list = root

[global]
add machine script = /usr/sbin/useradd -c Machine -d /var/lib/nobody -s /bin/false %m$
domain logons = Yes
domain master = Yes
local master = Yes
os level = 65
preferred master = Yes
security = user
workgroup = Workgroup Name
map to guest = Bad User
include = /etc/samba/dhcp.conf
logon path = \\%L\profiles\.msprofile
logon home = \\%L\%U\.9xprofile
logon drive = P:
netbios name = Server name
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes

[profiles]
comment = Network Profiles Service
path = %H
read only = No
store dos attributes = Yes
create mask = 0600
directory mask = 0700

[yoursharehere]
comment = Yoursharehere is a shared directory
inherit acls = No
path = /path2yoursharehere
read only = No
browseable = Yes
writable = Yes
force user = yourusername


After the smb.conf is set up correctly, only thing left is to add samba users and their passwords. I use the following command to create first the user, then adding samba password to that user. Note that the username password should be preferable the same ones you use in your clients. That way a windows user can mount the share as a network drive by just using his own username and password. After adding the users, you are ready to start the samba service and add it to some runlevel, so it will start automatically on boot.

useradd yourusername && smbpasswd -a yourusername


Enabling the Gentoo Portage Multimedia Overlay
Multimedia overlay is used to get the latest (and greatest) svn/cvs -versions of the multimedia programs like mplayer, which is used as the default video player in MMS. The reason we need the latest version is that it has multi-thread support enabled. This helps a lot when playing content that is encoded with H.264 with high definition resolutions. Playing 1080p material with h.264 encoding used to be pretty impossible with modest 3ghz dual core like mine. Nowadays with multithreading, it handles it pretty nicely.

This is how the overlay is enabled

mkdir /usr/local/portage
cd /usr/local/portage
rsync -rptvW rsync://kpex.no-ip.org/kpex-media
echo "PORTDIR_OVERLAY=\"/usr/local/portage\"" >> /etc/make.conf


HTPC software
Xineliboutput is a plugin to VDR, it is used to output the DVB-material via graphics card, either in X (vdr-sxfe) or in frame buffer (vdr-fbfe). Normal use case with VDR is that user has full feature DVB-tuner which has some kind of TV-out, but with high resolution flat panels this is hardly optimal. Instead we are going to use the DVI output on the graphics card to transfer the picture in HD resolution to TV´s HDMI input. Xineliboutput enables us to do just that. The plugin itself is loaded when VDR starts and the client (vdr-sxfe) is started whenever we select "Watch TV" from MMS main menu. And finally to the point: Xineliboutput requires xine-lib to be installed.


Besides xine library and mplayer we also need some codecs. W32codecs contains Microsofts decoders for codecs used in wmv-container (for example). We also should install x264-svn package. And lastly you can install MMS right here, without any extra steps, IF you are content (and smart) to use the stable version. I however, am going to go the extra mile and try out the new release candidate 1.1.0 version.

All of the above are installed by simply (mmsv2 installs at the time of this writing the stable 1.0.8.5 version of the MMS) emerging the following

emerge xine-lib mplayer win32codecs x264-svn mmsv2


Disclaimer: While the Xineliboutput enables showing the DVB material in high resolution X environment, it does not actually do anything to make the picture better. You are still faced with the limitations of the source material, which is often low resolution and encoded with low bitrate. The upside is though that the DVB viewing with VDR can now be integrated as part of the X (the graphical interface) and more importantly part of the MMS.

The Masterplan™

Now that all my hardware seems to be working correctly, it is time to move to installing needed software. But before that I guess I should let you in on my plan for the final HTPC.

I am going to use the My Media System (MMS in short) as a Media Center software. I had the stable version 1.0.8.5 installed on my previous installation. I liked it very much, it seems to be stable, feature rich and simple at the same time. Also the new version 1.1.0, which is currently in release candidate -state, seems to have many nice new features. I think I am at least going to try the new bleeding edge version first and if it is still too unstable, change back to the 1.0.8.5 version.
While MMS takes care of the things like picture viewing, music and video playing and providing nice user interface for these, I am going to use VDR for the DVB side. I've been using VDR to watch DVB-television for some time now (over a year at least) and I am very happy with it.
Other option would have been MythTv which bundles all of this together and provides even more functionality. But MythTv has been more viable on the analog TV side, not so good in DVB. So basically VDR with X frontend does the watching and recording of television and MMS does the rest.

Here is a little demonstration of how both VDR and MMS work together. Keep in mind that the video is old and for example exiting the VDR frontend (the AVP movie) is more seamless nowadays.

Beyond of being "just" a Media Center for my living room, I am also going to utilize it as a file server. My home network includes two workstations and one laptop and because of the WLAN is available, I also have some visitors when friends come over with their WLAN enabled devices, like laptops or mobiles.

I will be using Samba for file sharing between the OpenSuse workstations, laptop and mostly Windows-based visitors. Samba-server is pretty easy to setup and making connections by clients is also very easy. Mirrored hard drives (RAID1) on my HTPC slash file server will take care of the content I don't want to lose due to hard drive crash. I will also probably make a little script that makes backups from HTPC/File server to the net.


OK, so maybe in my next entry I will be actually getting to the software installation phase, which I already promised few entries back.

Getting the hardware to work

Fortunately one thing I was able to rescue from earlier, corrupted, installation is the kernel config. So I don't need to spend much time with kernel configuration. I however decided to go bleeding edge with the just released 2.6.24 kernel. Preserving the old configuration is pretty easy, just copy the old .config-file to the new kernel sources (/usr/src/linux) and run "make oldconfig" the command asks user to answer few questions and voilá, kernel configured.

Installing DVB drivers is also pretty easy and there are few different ways to do it. I used to install them just by using emerge, ie. "emerge mercurial && emerge v4l-dvb-hg" but currently there seems to be compilation problem. Because of the problem with emerge I had to install the drivers by hand. Here is a short overview of the process, the guide is for different tuner card than mine, but one can see the outline of the process easily enough. After dropping the firmware file to correct location (/lib/firmware) and rebooting, udev should load the modules for the card automatically. One can verify that the tuner is working by making sure there is device called /dev/dvb/adapter0 .

Lirc installation is also pretty straightforward, although it can also be problematic for the first timers. For emerge to work correctly a line needs to be added to /etc/make.conf, with my Silverstone (iMon) IR-receiver the line looks like this
LIRC_DEVICES="lirc_imon imon_pad2keys imon imon_pad imon_rsc"

After that "emerge lirc" and reboot the /dev/lirc device should be available. With serial port IR-receiver some tweaking with "setserial" command might be required. Correct lircd.conf is of course needed in /etc directory. For most remotes you don't need to look any further than here. I however made my own for the Harmony remote by using "irrecord" command. Testing that lirc is working command "irw" can be run, while running it should detect key presses from the remote.

If you want to use the remote that came with the case (the one with a "mouse pad" on it) you should have look at this page.

Once the lirc is working, emergin lcdprog makes the VFD available for using. Testing that it is working is easy, just "echo whatever > /dev/lcd0".


So what else is there?

• Proprietary Nvidia graphics drivers can be installed by issuing "emerge x11-drivers/nvidia-drivers" command and then using "modprobe nvidia" to load the module, although I think udev also loads automatically after reboot. Remember to include the nvidia as a driver to your /etc/X11/xorg.conf if you are using X.
• My integrated sound card is recognized automatically and my sound is working after "emerge alsa-utils" and setting the volume with "alsamixer", using "alsactl" one can store the volume settings as defaults.
• Integrated network was correctly detected during install and if it is not and you can not get it to work by hand, installing Gentoo becomes pretty hard. Luckily ethernet works correctly at full gigabit speed.
• Flat panel TV can sometimes be a real pain to get working correctly due to the TV itself giving wrong information about its capabilities. Example xorg.conf file for Mirai DTL-642E500 can be found here. I personally had some problems with finding correct settings and hopefully the linked file is useful to others.

Disclaimer: To get the devices to work, the kernel needs to be configured correctly. But that information is (mainly) available at Gentoo wiki, so there is no need for me to rewrite it all here. Check the Howto Index, Tips & Tricks and Hardware part of the wiki.

Basic Gentoo Installation

As I mentioned before, even the basic Gentoo installation can be a chore. But after installing Gentoo several times, I am starting to see the beauty behind it. Everything is done on command line but in a sense, I have full control of how my system is going to be. Nothing unnecessary gets installed or setup, which is great help when you´re trying to minimize the amount of software.
For example boot process slows considerably by adding more services,
so one should be aware what is actually needed and what can be left out. Also less software means less disk usage, less security issues and most importantly more performance.

Installation process is pretty easy if one follows the Gentoo Handbook and does not deviate much. I decided install to RAID1 array right away to steer clear of the pitfalls of doing it later.
Installation itself was pretty uneventful, but I would not recommend it to any one without proper Linux experience.

There are easier distros to install, or rather, almost all other distros are easier to install. One could go with Ubuntus minimal server installation and make a pretty decent HTPC. Not many however offer you so much control over your OS as Gentoo.


I decided to keep the partiotioning of the disks as simple as possible.

• /dev/md0 10GB for / (root) partition
• /dev/md1 290GB for /safe partition, this is used to store important data
• /dev/md2 4GB for swap partition, yes I know that is pretty much, but I want to be able to Suspend my RAM to disk if need arises.

So far I've emerged (installed) the following additional software.

• syslog-ng and logrotate
• vixie-cron
• grub
• mdadm
• xfsprogs (this is needed for my 500GB DVB Recording XFS partition)
• alsautils
  

And I have enabled following additional services.

• sshd
• syslog
• eth0
• mdadm
• syslog-ng
• vixie-cron
• alsasound
  

I am planning to start the not-critical services in their own runlevel or at least after the HTPC services that are needed to make the Media Center start as fast as possible. Currently the mentioned services still reside on default-runlevel.

Next time, installing Lirc remote control software, Mplayer video player, X the graphical server and Samba for network file sharing and maybe few others.