在ubuntu8.04上安装kerrighed-2.3.0cluster.doc

在ubuntu 8.04上安装kerrighed 2.3.0 cluster参考原文：Setting Up A Diskless-boot Kerrighed 2.4.1 Cluster in Ubuntu 8.04本实验中用了三台机器，一台作为server（运行Ubuntu 8.04，内核任意），另外两台作为client。Server不需要安装Kerrighed内核，他只作为服务器，为Client节点提供网络以及文件系统等支持。Server通过DHCP、TFTP服务为Client进行网络安装操作系统，并通过NFS为Client提供全局文件系统支持。Server有两个网卡，一个对外，一个对内。本实验中对外的网卡为eth1，其IP与本实验无关；对内的网卡为eth0，其IP为，子网掩码为。这个安装向导分为两个部分：第一部分是配置服务器，让节点可以使用当前的kernel无盘启动；第二部分是安装与配置Kerrighed 2.3.0。为了简单，本文大多内容直接原文拷贝，所以是英文，个别地方做了修改，或采用中文说明。另外，重要的补充说明用红字体标示。Part 1: 建立无盘启动的Ubuntu serverThis will be the basis of our cluster. In order to get a working diskless boot server, there are four main components youll need to install: a DHCP server to assign IP addresses to each node, a TFTP server to boot the kernel for each node over the network, an NFS server to allow the nodes to share a filesystem, and a minimal Ubuntu 8.04 installation for them to share. These server components will all run on one box; this box will be your head node or the controller for the cluster. Lets get started!1.1: Setting up the DHCP serverDHCP is what will allow the nodes to get IP addresses from the server. We want to set up and configure a DHCP daemon to run on the server and give IP addresses only to nodes it recognises, so we will tell the daemon their MAC addresses. First install the DHCP server package with aptitude or apt-get, as root:# aptitude install dhcp3-server补充说明：从此处开始，后续会有很多操作涉及aptitude install或者apt-get install，这些都需要联网操作。那么，在无法上网的情况下，如何安装这些软件呢？一种相对简单的方式是：一次性的把所有的软件安装在另一台上网机上，然后拷贝这些软件到非上网机。此次实验采取的即是这种方法，详细说明见附录一。当然，也可以在本地局域网配置一个软件源，只要添加这个源即可，但源非常庞大，下载较麻烦，暂时未使用该方法。修改内网卡的IP为，掩码为，网关为，DNS为。 Check that the DHCP server configuration file, /etc/default/dhcp3-server, contains the correct Ethernet card to listen to DHCP requests from the nodes: this will be the card that does not connect to the Internet. If you put the wrong card into this file, bad things will happen: your server will start listening to DHCP requests on the LAN and denying them all, since they dont come from nodes. Make sure you know which is the right card! = In this case it is eth0, so make the configuration file look like this:# /etc/default/dhcp3-server #interfaces=eth0 Now you need to configure the DHCP daemon to issue addresses only to nodes, and tell it which addresses to give them. Make sure you have the MAC addresses of your nodes Ethernet cards handy - you can get them by issuing the command ifconfig as root from a live DVD or USB stick, for instance, or they may be written on the nodes casing or on stickers stuck to the cards. You can also just boot the cluster and watch the error messages, picking out the MACs from the denied requests. Edit /etc/dhcp3/dhcpd.conf, DHCPs daemon configuration file, so it looks like the following. (Dont worry that the file references certain things you havent actually installed yet, like a PXE bootloader; youll put these things in later.)# /etc/dhcp3/dhcpd.conf # General optionsoption dhcp-max-message-size 2048;use-host-decl-names on;deny unknown-clients; # This will stop any non-node machines from appearing on the cluster network.deny bootp;# DNS settingsoption domain-name kerrighed; # Just an example name - call it whatever you want.option domain-name-servers ; # The servers IP address, manually configured earlier.# Information about the network setupsubnet netmask option routers ; # Server IP as above.option broadcast-address 55; # Broadcast address for your network.# Declaring IP addresses for nodes and PXE infogroup filename pxelinux.0; # PXE bootloader. Path is relative to /var/lib/tftpbootoption root-path :/nfsroot/kerrighed; # Location of the bootable filesystem on NFS serverhost kerrighednode1 fixed-address 01; # IP address for the first node, kerrighednode1 for example.hardware ethernet 01:2D:61:C7:17:86; # MAC address of the nodes ethernet adapterhost kerrighednode2 fixed-address 02;hardware ethernet 01:2D:61:C7:17:87;server-name kerrighedserver; # Name of the server. Call it whatever you like.next-server ; # Server IP, as above.Now youre done configuring DHCP, so your nodes will be able to get IPs. Its time to add the functionality that will allow the server to transfer a kernel to them afterwards.1.2: Setting up the TFTP server and PXE bootloaderTFTP is the fileserver that will be used by the bootloader to transfer the kernel to your nodes during a PXE boot. We need to install a TFTP server and get a PXE bootloader as part of the syslinux package, so that our nodes will be able to get their operating systems via the cluster server. As root, install the TFTP server package, tftp-hpa, with aptitude or apt-get:# aptitude install tftpd-hpa Open its configuration file, /etc/default/tftpd-hpa, and make sure it uses the following settings. It can be run as a daemon, but it is normally started by the service daemon inetd, and it uses the /var/lib/tftpboot directory to get files from:# /etc/default/tftp-hpa #Defaults for tftp-hpaRUN_DAEMON=yesOPTIONS=-l -s /var/lib/tftpboot Now we need to configure inetd to run the tftp server. Open its configuration file, /etc/inetd.conf, and change the tftp line to the following. If there is no tftp line, add this:tftp dgram udp wait root /usr/sbin/in.tftpd /usr/sbin/in.tftpd -s /var/lib/tftpboot As root, install syslinux, which is the system required for you to be able to PXE boot the nodes, and copy the PXE bootloader code from it to the TFTP server directory. This is the bootloader you told the DHCP daemon about in its configuration file earlier.# aptitude install syslinux# cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot Still as root, create a directory to store the default configuration for all the nodes. They will search in this directory for configuration files during the PXE boot process.# mkdir /var/lib/tftpboot/pxelinux.cfg Still as root, copy your current kernel and initrd from /boot to /var/lib/tftpboot/ in order to test the diskless-boot system. If need, replace uname -r with whatever you are using.# cp /boot/vmlinuz-uname -r /boot/initrd.img-uname -r /var/lib/tftpboot/ Create the file /var/lib/tftpboot/pxelinux.cfg/default. This will be the fallback configuration file that the nodes use to PXE boot when they cant find a file specific to their own IP address. Make the file look like this:LABEL linuxDEFAULT vmlinuz-uname -r console=tty1 root=/dev/nfs initrd=initrd.img-uname -r nfsroot=:/nfsroot/kerrighed ip=dhcp rwYoure done setting up the TFTP and PXE components of the cluster server! Your nodes will now be able to get a kernel and filesystem from the server after theyre given IP addresses. Now you need to add NFS capability.1.3: Setting up the NFS serverThis capability allows for the bootable filesystem that the nodes will download over TFTP to be accessed and shared over the network, so that the cluster uses one filesystem. First well install and set up the server that will do this. As root, install the packages nfs-kernel-server and nfs-common, which comprise the NFS server program. Keep your root authorisation until youre done working with the NFS server.# apt-get install nfs-kernel-server nfs-common Make a directory to store the bootable filesystem in:# mkdir /nfsroot# mkdir /nfsroot/kerrighed Edit /etc/exports, which configures NFS file transfers. Add the following in order to make NFS export the filesystem that will be stored in the directory you just made:# /etc/exports #/nfsroot/kerrighed /(rw,no_subtree_check,async,no_root_squash) Re-export the file systems, since you just changed how this is done:# exportfs -avrYour NFS server should be up and running. Now you can add a filesystem for this server to work with.1.4: Setting up the bootable filesystem This isnt as simple as just copying the OS files into another directory - youll need the debootstrap package to install the bootablefilesystem, so install this first (you should still be root.) Once its installed, use debootstrap to itself install a minimal Ubuntu Hardy system tothe bootable filesystem directory:# aptitude install debootstrap# debootstrap -arch i386 hardy /nfsroot/kerrighed /ubuntu/补充说明：此处需要联网下载一个完整的ubuntu8根系统，对于不能联网的机器，解决方法如下。在联网的机器上安装ubuntu8.04的虚拟机，建立/nfsroot/kerrighed目录后执行以上命令，成功后该目录下将会有一个完整的根目录，拷贝这个目录（最好先压缩再拷贝，防止数据丢失）到非联网机器的相应目录即可。在本实验中，由于后续还要apt-get install一些软件，这些也需要联网，所以应当在联网机上执行完这些所有需要联网的操作之后，再拷贝整个目录到非联网机上。当然，此时在非联网机上就可以掠过这些步骤了，这些需要在联网机上操作的后续步骤如下：Edit /etc/apt/sources.list.# aptitude update$ apt-get install dhcp3-common nfs-common nfsbooted openssh-server$ apt-get install automake autoconf libtool pkg-config gawk rsync bzip2 gcc-3.3 libncurses5 libncurses5-dev wget lsb-release xmlto patchutils xutils-dev build-essential openssh-server ntp Change the current root of the file system to the bootable filesystem directory (stay chrooted until the guide tells you otherwise.) This is so that you can work with the bootable filesystem directly, as if it were a separate machine, while we make some adjustments to it.# chroot /nfsroot/kerrighed Set the root password for the bootable filesystem. You can use a program called apg, the automated password generator, to create a good one.# passwd Mount the /proc directory of the current machine as the bootable systems /proc directory, so you can use programs on the bootable FS:# mount -t proc none /proc Edit /etc/apt/sources.list. We want to add access to some extra Ubuntu repositories in order to be able to download the necessary packages for the FS. Add these lines:deb /ubuntu hardy partnerdeb /ubuntu/ hardy main universe restricted multiversedeb /ubuntu/ hardy-security universe main multiverse restricteddeb /ubuntu/ hardy-updates universe main multiverse restricteddeb-src /ubuntu/ hardy main universe restricted multiversedeb-src /ubuntu/ hardy-security universe main multiverse restricteddeb-src /ubuntu/ hardy-updates universe main multiverse restricted Update the current package listing from the repositories you just added so youll be able to install things from them:# aptitude update Install some packages that our nodes need for using NFS and DHCP:$ apt-get install dhcp3-common nfs-common nfsbooted openssh-server Now we need to make it work with NFS. Edit /etc/fstab, the filesystem index, of the bootable filesystem to look like this:# /etc/fstab# proc /proc proc defaults 0 0/dev/nfs / nfs defaults 0 0 Edit /etc/hosts. This is so that the DHCP server will know which of the PXE-booted nodes get which IP and hostname. Add all your cluster nodes and the server to it. In our example case, it looks like this:# /etc/hosts # localhost kerrighedserver01 kerrighednode102 kerrighednode2 Do the following to create a symbolic link which will automount the bootable filesystem as /dev/nfs on the server, when it starts up. This should not collide with other existing services in the directory (e.g. anything that looks like /etc/rcS.d/S34xxxxxxx), so check carefully before you create the link. If theres a service with a similar name, disable it before you do anything else.$ ln -sf /etc/network/if-up.d/mountnfs /etc/rcS.d/S34mountnfs Edit /etc/network/interfaces and add a line to stop Network Manager from managing the nodes Ethernet cards, as this can cause issues with NFS. Ours looks like the following:# .# The loopback interface:auto loiface lo inet loopback# The primary network interface, manually configured to protect NFS:iface eth0 inet manual Create a user for the bootable system. Replace with whatever you want to call her. adduser will ask you for her real name andother details, so play along.# adduser Ensure that the new user is in the /etc/sudoers file so she can run root commands on the cluster:# /etc/sudoers #User privilege specificationroot ALL=(ALL) ALL ALL=(ALL) ALL Exit the root shell, and then exit from the chrooted bootable filesystem. Youre done configuring the bootable FS and can now test it with yourcommon-or-garden kernel.# exit# exit1.5: Testing the diskless boot system sans Kerrighed Restart the servers on the head node, because youve been messing with them. You need to be root to do this:# /etc/init.d/tftpd-hpa restart# /etc/init.d/dhcp3-server restart# /etc/init.d/nfs-kernel-server restart Configure the BIOS of each node to have the following boot order: your primary boot device should be PXE, which will usually be described as network boot or LAN boot. In certain cases you may need to enable the network cards as boot devices in the BIOS, reboot, and then set the boot priority. Remember also to disable halt on all errors, since this can mess up your PXE booting. Boot each of the nodes to see if it works. If so, you should be presented with a login prompt, where you can log-in using the username you defined earlier. When it all works, youre ready to try with a Kerrighed kernel.Part 2: 建立Kerrighed集群Now that weve got a diskless boot system setup to use as a server, we only need to build the Kerrighed kernel for the nodes to use, put it in the bootable FS, and configure the Kerrighed settings properly in order to have a working SSI (Single System Image) cluster.The first thing to do is build the new kernel itself.2.1: Building the Kerrighed kernel Shutdown the nodes. On the server, chroot back into the bootable filesystem, again as root:# chroot /nfsroot/kerrighed Install some basic necessary packages into the bootable filesystem. These are the dependencies that well need to have on the system in order to successfully compile Kerrighed.$ apt-get install automake autoconf libtool pkg-config gawk rsync bzip2 gcc-3.3 libncurses5 libncurses5-dev wget lsb-release xmlto patchutils xutils-dev build-essential openssh-server ntp Get the Kerrighed source ball from INRIAs GForge, and then get a copy of the vanilla 2.6 kernel. This is the kernel that were going to be modifying into a Kerrighed one.# wget -O /usr/src/kerrighed-2.3.0.tar.gz http:/gforge.inria.fr/frs/download.php/23356/kerrighed-2.3.0.tar.gz# wget -O /usr/src/linux-2.6.20.tar.bz2 /pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2补充说明：此处需要联网操作，对于非联网机器，可在联网机器上下载后拷贝过来。其中kerrighed在上找，linux在上找。 Change to the /usr/src directory, where the downloaded source tarballs are, and decompress them:# cd /usr/src# tar zxf kerrighed-2.3.0.tar.gz# tar jxf linux-2.6.20.tar.bz2 Look inside the decompressed Kerrighed sources. Theres a bug here (at time of writing) in the modules directory Makefile: open it up with your favourite editor (were using Vim here) and take a look. If the bug is present, it will use the sh shell as the value of SHELL, right at the top. Under Ubuntu and Debian, that wont work - itll call /bin/dash, which isnt useful here. Change it to /bin/bash. The other Makefiles are fine.# mv /bin/sh /bin/sh.old# ln -s /bin/bash /bin/sh(Ubuntu default sh is link to dash) Move back up to the kerrighed source directory, configure the sources to work with the kernel version we just downloaded and decompressed, then change to the kernel directory (which is your unpatched Linux kernel). Here, change the kernel configuration file with make menuconfig. This will seem like a mysterious process if youve never done it before: menuconfig is a pseudo-graphical menu that lets you adjust the capabilities of the kernel to a fine degree; it has control instructions at the top and looks a bit like an old-school DOS program. First, go to Device Drivers - Network device support and check that the drivers for your cards have an asterisk * and not a module symbol M next to them. You dont want anything to have the M symbol right now, as were not using a system that will support kernel modules. Next go back to the main menuconfig menu and make sure NFS is enabled by going to File systems - Network File Systems and enabling NFS file system support, NFS server support, and Root file system on NFS. Make sure that the NFSv3 options are also enabled, and again, make sure they are part of the kernel and not loadable modules (asterisks and not Ms). Once this is done, exit by pressing Escape twice. # cd kerrighed-2.3.0# ./configure -with-kernel=/usr/src/linux-2.6.20# make patch# make defconfig# make C /usr/src/linux-2.6.20 menuconfig Now you can make and install the Kerrighed-enabled Linux kernel, and the Kerrighed tools. Then run ldconfig to link up the new Kerrighedlibrary files just in case something breaks:# make kernel# make# make kernel-install# make install# ldconfig补充说明：make的时候会出现path error错误，这是linux-2.6.20内核本身的bug导致。修正bug的方法为在linux-2.6.20/scripts/mod/sumversion.c的最开头一行添加#include 。 If everything has been correctly installed, you should have the following in the bootable filesystem. Check for them with cat or ls (list fromthe Kerrighed 2.3.0 install guide):/boot/vmlinuz-2.6.20-krg Kerrighed kernel/boot/System.map Kerrighed kernel symbol table/lib/modules/2.6.20-krg Kerrighed modules/etc/init.d/kerrighed Kerrighed service script */etc/default/kerrighed Service configuration/usr/local/share/man Manpages/usr/local/bin/krgadm Cluster