From Kerrighed

Contents

Presentation

The main idea consists in using a NFS server which exports the Kerrighed environment. Client nodes do not need to have any hard disk devices attached to take part in a cluster session. All kinds of data (e.g. modules, special files, configuration files, etc.) needed at runtime are being provided by NFS. A NFS serves a local '/' directory that can be mounted by the remote (NFS) clients.

In this setup, the NFS server must NOT be part of the cluster. Thus it does not need to run a Kerrighed kernel.. This also means that it's possible to use a 32 bits processor machine as the NFS server (since the version 2.6.30 requires 64 bits processors): some additional details must be considered when using such setup but if you have an restriction regarding available 64 bits nodes this method can be handy.

If you're going to use the Debian GNU Linux distribution for setup, beware to use the Lenny distribution

Nfsroot overview.png

Always check the README file when installing Kerrighed for last minute instruction and details for installation.

Pre-requisites

On the nodes

Nodes must be able to boot with PXE. PXE support is usually implemented in the NIC BIOS, and must be enabled in the BIOS setup. If the nodes cannot use PXE (by a limitation of the NIC or the BIOS), it is also possible to use one the boot loaders available in the project Etherboot/gPXE (http://etherboot.org/wiki/download). A great number of network card's are supported by the project.

It's usually a good idea to use the same hardware configuration for all nodes, if possible. This will help avoiding errors from configuring different options in the Linux kernel configuration and making the setup process to be easier.

On the server

  • DHCP server
  • TFTP server
  • NFS server
  • Syslinux
  • Optionally, a dedicated tool to create a basic Linux system (eg. debootstrap on Debian)

On Debian: 

apt-get install dhcp3-server nfs-kernel-server tftpd-hpa syslinux debootstrap

Configure the server

Configure the TFTP server

If you installed tftpd-hpa on a Debian system, you may edit /etc/default/tftpd-hpa: 

# Default for tftpd-hpa
RUN_DAEMON = "yes"
OPTIONS = '-l -s /srv/tftp'

where /srv/tftp is the place you store kernel, ramdisk and Syslinux configuration files for clients.

Configure the DHCP server 

### PART 1
# General options
option dhcp-max-message-size 2048;
use-host-decl-names on;
deny bootp;

### PART 2
option domain-name "mycluster.home";
option domain-name-servers 123.123.123.123, 124.124.124.124;
option ntp-servers ntp.network.net;

### PART 3
subnet 192.168.0.0 netmask 255.255.255.0 {
 option routers 192.168.0.1;
 option broadcast-address 192.168.0.255;
 # Define the first and last IP address to be authorized
 range <range start> <range end>;
 # This set up the node name to « krgnodeXX » with XX the id of the node (ip-address based).
 send host-name = concat("krgnode", binary-to-ascii(10, 8, ".", substring(leased-address, 3, 1)));
 # If you want to limit which boxes are using kerrighed, use something like this :
 # host ssi1 { fixed-address 192.168.0.101; hardware ethernet xx:xx:xx:xx:xx:xx; 
}

### PART 4 
group {
 filename "pxelinux.0";
 server-name "192.168.0.2";
 option root-path "/NFSROOT/kerrighed";
}

In this example:

  • The first part is generic and does not need any change;
  • In part 2, you need to adapt to your network configuration: domain name, dns server, ntp server, etc.;
  • Part 3 prevents your DHCP server from answering to networks other than your local network;
  • Part 4 defines which files are given to clients when booting.
    • filename point to the syslinux PXE bootloader. Path is relative to the tftp daemon root (as set in /etc/default/tftpd-hpa).
    • Lines beginning with host associate an IP address to a MAC address. IP numbers can be replaced by names if in they are available to the naming service (/etc/hosts, DNS, etc.).

Configure the NFS server

  • Edit /etc/exports: 
/NFSROOT/kerrighed *(rw,sync,no_root_squash,no_subtree_check)

Reload the services

On Debian: 

/etc/init.d/dhcp3-server restart
/etc/init.d/tftpd-hpa restart
/etc/init.d/nfs-kernel-server restart

Create the base system

In this chapter, we describe the creation of a NFSROOT base system, without Kerrighed.

  • Create a dedicated directory where you are going to put the Kerrighed system. Let's say /NFSROOT. 
mkdir /NFSROOT
  • Create a base Linux system inside this directory.
    • With Debian, you may use: 
debootstrap lenny /NFSROOT/kerrighed http://ftp.debian.org/debian

The URL is just an example. Usually is a good idea to locate the nearest mirror of your location.

debootstrap will generate the following output: 

I: Retrieving Release
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://ftp.debian.org/debian
I: Retrieving libac11
I: Validating libacl1
[..]
I: Configuring tasksel-data...
I: Configuring tasksel...
I: Base system installed successfully.

Beware that you will need to execute this command in a 64 bits machine to be able to create valid images for the nodes. See the debootstrap manpage for details.

The debootstrap takes a long time to execute depending of your Internet conection speed. Usually is a good idea to create a tarball of the directory after the install is done so it's possible to recover the setup if necessary. Be sure to use the -p command option of tar to preserve the permissions.

  • Change to your root directory: 
chroot /NFSROOT/kerrighed
  • Change root passwd 
passwd
  • Add an default user to be used during program executions on the nodes of the cluster 
adduser <username>

Where <username> is the login name of your choice.

  • Mount the proc filesystem 
mount -t proc none /proc
  • It's possible that you get errors from the Perl when installing packages in the image. To suppress those errors, type in the console: 
export LC_ALL=C

It's also a good idea to add such command in the .bashrc of root user.

  • Install all needed packages for the client: gcc, autotools, vi (or emacs !), ssh, libncurses, etc.
  • Install DHCP, NFS and SSH support for the nodes.
    • With Debian:

Edit the /etc/apt/sources.list to add sources for apt: 

nano /etc/apt/sources.list

Include the URL's as necessary: 

deb http://ftp.debian.org/debian/ lenny main non-free contrib
deb http://security.debian.org/ lenny/updates main contrib

And execute apt-get for the following packages 

apt-get install dhcp3-common nfs-common nfsbooted openssh-server

nfsbooted can assign different hostnames to each node of your cluster by resolving the IP address assigned to your node.

  • If you plan to build an initial ramdisk, install an initramfs builder.
    • With Debian: 
apt-get install initramfs-tools
  • The ramdisk image must be created with network support. Please have a look at /etc/initramfs-tools/initramfs.conf.
  • Edit /etc/hosts to add all cluster nodes: 
127.0.0.1 localhost.localdomain localhost

192.168.0.1    server.mycluster.home server
192.168.0.101  ssi1.mycluster.home ssi1
192.168.0.102  ssi2.mycluster.home ssi2
192.168.0.103  ssi3.mycluster.home ssi3
192.168.0.104  ssi4.mycluster.home ssi4
  • Your /etc/mtab should point to /proc/mounts: 
rm /etc/mtab
ln -s /proc/mounts /etc/mtab
  • There are mainly two ways for mounting the NFS root:
    • You can mount the root read-only, and some specific directories read-write. This gives to each system only rights it needs.
    • The alternative is mounting the NFS root read-write, which is easier to setup, but provides less control.
    • For both ways, please remember to execute the command 
exportfs -avr

As root so the NFS server will know how to deal with requests for the export.

NFS root read-only

  • Edit fstab to define the mount points (we assume 192.168.0.1 is your NFS server): 
# A swap partition
/dev/hda    none     swap       sw        0 0
none        /proc    proc       defaults  0 0
none        /sys     sysfs      defaults  0 0
none        /config  configfs   defaults  0 0
#
# NFSROOT
# Following partitions are mounted in rw mode, for the moment, I have 
# some troubles with lockd daemon that's why I added the 'nolock' params
192.168.0.1:/NFSROOT/kerrighed/var  /var  nfs rw,hard,nolock 0 0
192.168.0.1:/NFSROOT/kerrighed/tmp  /tmp  nfs rw,hard,nolock 0 0
192.168.0.1:/NFSROOT/kerrighed/root /root nfs rw,hard,nolock 0 0
192.168.0.1:/NFSROOT/kerrighed/home /root nfs rw,hard,nolock 0 0
#
# TMPFS
none                             /var/run tmpfs defaults     0 0
  • Edit your /etc/default/rcS to make your nfs drives really mounted: 
ASYNCMOUNTNFS=no

NFS root read-write

  • For NFS root read-write, the fstab just contains the minimum: 
# A swap partition
/dev/hda    none     swap       sw        0 0
none        /proc    proc       defaults  0 0
none        /sys     sysfs      defaults  0 0
none        /config  configfs   defaults  0 0
none        /var/run tmpfs      defaults  0 0

Configure nodes boot

Once your server is configured and your base system is ready, you can set up your nodes to boot with the base system.

Default configuration for all nodes

  • Copy the kernel and ramdisk (if any) in /srv/tftp from your base system: 
cp /NFSROOT/kerrighed/boot/vmlinuz-xxx /NFSROOT/kerrighed/boot/initrd-xxx \
   /srv/tftp
  • Create a directory in your tftp space to put Syslinux configuration: 
mkdir /srv/tftp/pxelinux.cfg
  • Edit the file /srv/tftp/pxelinux.cfg/default and put in it: 
DEFAULT kerrighed
LABEL kerrighed
    kernel boot/vmlinux-2.6.30-krg3.0.0
    append ip=dhcp root=/dev/nfs session_id=1 autonodeid=1

Specific configuration for nodes

It is possible to create an specific configuration file for one or mode nodes. This usually is required if you're using nodes with different hardware configuration and some nodes requires specific configurations to boot.

To do that, just create an text file inside the directory /srv/tftp/pxelinux.cfg (or whatever directory you may have selected to hold PXE configuration) with the filename equal to the fixed IP address configured in the DHCP server, converted to hexadecimal base. For example, the IP address 10.11.12.101 should be converted to 0A0B0C65 and this value used as the filename for the node configuration with such IP address.

There are several programs that may help converting such values (like gbase).

Install Kerrighed

Validate your environment without Kerrighed, booting your nodes with it. If it's ok, walk a step ahead and install Kerrighed on your system. This will avoid mixing errors of environment configuration and kernel options used during kernel compilation.

Usually the nodes will depend on the network card drivers be available on the Kernel (not as modules) or as modules (usage of initrd is implicity in this case).

  • When building and installing Kerrighed, you should whether chroot to your NFS root: 
chroot /NFSROOT/kerrighed/
  • Or use /NFSROOT/kerrighed/ as your destination directory when making install: 
make install DESTDIR=/NFSROOT/kerrighed/

Please check the README file in the Kerrighed tarball for more details of the installation process. The following instructions are for Kerrighed 3.0.0 on Debian GNU Linux (Lenny).

Installation process of Kerrighed 3.0.0

  1. Check out the procedures in UserDoc#Build_phase.
  2. Configure the nodes regarding the capabilities and start doing some tests with tem. Be sure to check out the UserDoc#Quick_usage_guide for more information about that.
Retrieved from "http://www.kerrighed.org/wiki/index.php/Kerrighed_on_NFSROOT"