集成Windows计算集群服务器进入一个Linux环境通过平台LSF.doc

Integrating Windows Compute Cluster Server into a Linux Environment throughPlatform LSFWhite PaperPublished: October 2007, Updated: January 2008AbstractThis white paper gives an overview of the integration of Windows Compute Cluster Server (WCCS) into a Linux environment using a third-party scheduler (Platform LSF) to manage the transfer of work. The white paper focuses on a typical scenario in which jobs submitted to a specified queue in the Platform LSF scheduler (which is deployed on a Linux compute cluster) are seamlessly integrated through Platform LSF to run in the Windows Compute Cluster Server environment.Getting Started3Overview3System Architecture4System Requirements and Considerations5Preliminary Setup7Install and Configure a Dedicated Active Directory Server7Install Windows Compute Cluster Server8Configure WCCS Using the Compute Cluster Administrator8Set Up the Network10Configure Linux for File Sharing10Install Platform LSF on Windows23Preparatory Steps23Run the Installer File25Integrate WCCS with Platform LSF Running on Linux29Requirements29Create Users in Active Directory and in Windows29Install WCCS and Platform LSF Integration Package31Register the LSF Passwords34Configure the Linux LSF Environment35Operating WCCS from Platform LSF on Linux37Before You Begin37Submit Jobs to WCCS from the LSF Linux Cluster37bsub Command37Executable Residing on WCCS38Executable Residing on Linux38Monitor Jobs39Software Maintenance41Update Release Schedule41Windows Server Update Services41Summary42Appendix 1: Set Up Active Directory43Appendix 2: Remote Installation Services47Appendix 3: Windows Server Update Services48Appendix 4: Samba Configuration51Appendix 5: Related Links53Getting StartedThis white paper describes how to integrate a Microsoft Windows Compute Cluster Server (WCCS) into an existing Linux cluster environment running the Platform LSF job scheduler. This integration enables Linux users to submit jobs to a Linux-based Platform LSF scheduler; the jobs are ultimately executed on WCCS.Because high-performance computing (HPC) clusters represent a significant investment of resources, maximizing the benefits of your existing investment can result in huge savings. The interoperability of WCCS with third-party job schedulers like Platform LSF can ensure seamless integration into heterogeneous clusters. If you have an existing Linux cluster and are familiar with Platform LSF, you can continue to use it while adding the power and ease of WCCS to your environment. WCCS enables you to accomplish more, in less time and with reduced effort, by taking advantage of your users existing skills and integrating with tools they are already using.The main steps for the installation and configuration of WCCS and for the integration of WCCS and the Linux cluster are described in this white paper. Details of the procedures, however, are beyond the scope of this document. For step-by-step instructions, refer to the links in Appendix 4.The integration procedures described in this paper have been verified with Platform LSF version 6.2 HPC, SUSE Linux Enterprise Server 10, and WCCS. The WCCS cluster was deployed with a recommended architecture consisting of a head node with two network interfaces for management and a standalone Active Directory server system.Overview The integration of WCCS into the Linux SUSE environment enables users to submit jobs to the Platform LSF scheduler for execution on WCCS.When a job is submitted to the Platform LSF schedulers WCCS queue on Linux, it is transferred to the Platform LSF installation on Windows Compute Cluster Server. The Platform LSF installation running on WCCS then authenticates the user and runs the job using that users credentials. The following figure shows the procedure.Figure 1 Job SubmissionIf any files that are specified by the job reside in the Linux environment, Platform LSF will make a Samba connection to the Linux environment with the credentials of the user submitting the job. The specified files are then transferred to a local directory, and the job is submitted to the WCCS Job Scheduler and executed. Once the job has been executed, specified files are transferred back to the Linux environment.System ArchitectureA typical WCCS configuration includes a system dedicated as an Active Directory server, a dedicated compute cluster head node, and one or more compute nodes. The Active Directory server and WCCS cluster head node are set up on the existing Linux cluster network. The head node uses an interior private network for node management and communication. Message Passing Interface (MPI) traffic can be routed over the private network or via a dedicated interconnect. Although not mandatory, the private cluster network enables use of the Remote Installation Services (RIS) automated deployment system.Figure 2 Windows Compute Cluster Server ArchitectureSystem Requirements and ConsiderationsThe table below shows the system requirements. Table 1 Windows Compute Cluster Server RequirementsAll Systems64-bit CPU 512 MB RAM minimumActive Directory Server Second hard drive recommended for increased Active Directory performance and recoverabilityWCCS Head NodeTwo network interfaces MPI interconnect optionalSeparate disk partition for use with RISCompute NodesPre-boot Execution Environment (PXE) boot enabled for RIS system managementIntegration with Linux requires consideration of several additional factors. These are described in the table below.Table 2 Considerations for Linux IntegrationNetwork RangeThe recommended Windows Compute Cluster Server implementation requires an IP address range of 192.168.0.x on its internal private cluster network. The external Linux cluster network will need to be set to a different range. It is possible for the Windows Compute Cluster to use a different address range if configured using the manually configured Routing and Remote Access Service (RRAS).SambaSamba is required for data to be imported and exported for job execution. It also enables Platform LSF to be managed in both environments from a single shared directory.User ManagementSamba, as of version 3, cannot act as an Active Directory domain controller. In order for Linux users to be able to authenticate, duplicate accounts will need to be created as Active Directory users with identical user names and passwords.Preliminary Setup In order to integrate WCCS with Platform LSF running on Linux as described in this white paper, you will need a working Linux cluster and a working WCCS cluster.Following is a summary of the prerequisite steps for installation and configuration of WCCS. For detailed installation steps, see the WCCS documentation at: /WindowsServer/en/library/dce5123f-8af4-47c2-9192-9075998e24c71033.mspx?mfr=true.Install and Configure a Dedicated Active Directory Server Begin by installing and configuring a dedicated Active Directory server. Use the following checklist for your Active Directory server installation and configuration. More detailed steps are given in Appendix 1.Table 3 Active Directory ChecklistIn most cases, the default options requested during installation are appropriate for WCCS and should be used.Use of an NTFS-formatted file system is recommended.When given the option to join a workgroup or domain, choose workgroup. The name of the workgroup is not important, as later during configuration you will create and join a new domain.IP address may be acquired by Dynamic Host Configuration Protocol (DHCP), or may be set to static at this stage. (See instructions below.) Later, you will assign the server a static IP address.When presented with the option to upgrade the system, choose Express for a shortcut to security updates. Once security updates are installed and the system reboots, select Update this server again, then choose Custom to locate any hardware driver updates.Internet Security Level is set to High by default. You may need to restart the update process for appropriate update applications to be installed and executed. Security level can be changed in Start Control Panel Internet Options Security Internet Custom Level Reset custom settings to desired level.WCCS Activation may require an Internet connection to Microsoft. Activation takes a few seconds and does not require registration. (Note: Depending on the license key used, a direct Internet connection may not be needed.)If there is no CD drive permanently installed, it may be advantageous to copy the WCCS installation disk to the hard drive for easy access to Windows Server applications. Choose Start My Computer. Right-click on CD icon. Choose Explore Create new folder on the server, and drag the installation disk contents to it. Set the static IP address, unless this was completed during installation. Set the DNS server address to an available DNS server for your Linux cluster network. This address will be integrated into the DNS server. The DNS address in this control panel will then be changed automatically to the Active Directory localhost at .Install Windows Compute Cluster ServerAfter installing Active Directory and DNS, use the following checklist to install WCCS.Table 4 Windows Compute Cluster Server ChecklistConfigure the head node. Set the static IP address on interface connected to Linux cluster network.Set the Active Directory server as the primary DNS.On the head node, join the Active Directory domain. Choose Start My Computer, then right click Properties, select Computer Name Change, and then type the domain name.On the head node, install the Microsoft Compute Cluster Pack (CCP). Install with the default options. Particularly note that the default option to create a new compute cluster should be chosen.Install WCCS on the compute nodes using RIS or an alternative remote deployment method. For more information about RIS, see Appendix 2.Configure WCCS Using the Compute Cluster AdministratorNext, configure WCCS with the Compute Cluster Administrator. This is the main control interface for WCCS, and is shown in the following figure. The Compute Cluster Administrator can be found in: Start All Programs Microsoft Compute Cluster Pack Compute Cluster Administrator. (Do not confuse this with a separate application in Administrative Tools that is labeled Cluster Administrator.)Figure 3 Compute Cluster AdministratorThe Compute Cluster Administrator provides access to the following:Table 5 Compute Cluster AdministratorTo Do ListUse the To Do List to complete configuration of the WCCS cluster.Node ManagementSelect Node Management from the left pane of the Compute Cluster Administrator. The center pane will now display each node. A node must be approved before joining the cluster. Select each node and then Approve from the right pane of the window. The nodes status will change to Paused. Next, select Resume from the right pane to fully activate the node. Remote Desktop SessionsSelect Launch Remote Desktop Connection from the right pane and a desktop logon for that node will be shown in the center pane. User logon is authenticated according to the users and policies of the Active Directory domain.WCCS ActivationEach installation of Windows Compute Cluster Server requires activation within 14 days of installation. On the desktop of each node, select the keys icon in the bottom right corner of the screen. An activation dialog box will be initiated. After 14 days without activation, the node will not be accessible.(Note: There are variations depending on the key and the operating system version used.)Compute Cluster Job SchedulerSelect Launch Compute Cluster Job Scheduler from the right pane. The Job Queue window will open to show the jobs submitted, running, and completed by the cluster. Jobs can also be submitted and managed from here.Set Up the NetworkUse the following checklist for your network setup. Table 6 Network SetupSelect the setup type. Note that the compute nodes are isolated on a private network.Configure the public network. Choose Local Area Connection with an external-facing address as previously set.Configure the private network. Choose Local Area Connection connected to the WCCS compute nodes.Enable the Network Address Translation (NAT) using Internet Connection Sharing (ICS). Choose Enable Internet Connection Sharing.Configure Linux for File SharingWhile the procedure that follows was tested on RedHat 4 systems, it is believed that this procedure will work on any Linux system that support Samba 3.x.To establish a shared file system accessible by the WCCS, you must have Kerberos 5, Winbind and Samba installed and properly configured. In addition, you must configure PAM and nsswitch.conf to authenticate against the Active Directory server.Install and Configure KerberosDepending on the installation options selected, Kerberos 5 may have been installed when the operating system was installed. To find out if Kerberos is installed run (on RedHat Linux):# rpm -qa | grep krbkrb5-workstation-1.3.4-17krb5-auth-dialog-0.2-1krb5-libs-1.3.4-17krbafs-1.2.2-6krb5-devel-1.3.4-17pam_krb5-2.1.8-1krbafs-devel-1.2.2-6At a minimum, you will krb5-workstation, krb5-libs, and pam_krb5.If Kerberos is not installed the source can be obtained from /Kerberos. Alternatively, many Linux vendors provide Kerberos packages that you can download and install. The RedHat distribution (and presumably other vendor supplied packages) of Kerberos is a standard compile of the Kerberos distribution by MIT packaged in rpm format.Once Kerberos is installed, it needs to be configured. Kerberos is configured by editing the /etc/krb5.conf file. In the example below the Active Directory server AD.WCCS.ZOHALLT.COM is the domain controller for the domain WCCS.ZOHALLT.COM.logging default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.loglibdefaults default_realm = WCCS.ZOHALLT.COM dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24h forwardable = yesrealms WCCS.ZOHALLT.COM = kdc = admin_server = default_domain = domain_realm .kerberos.server = WCCS.ZOHALLT.COM . = WCCS.ZOHALLT.COMkdc profile = /var/kerberos/krb5kdc/kdc.confappdefaults pam = debug = false ticket_lifetime = 36000 renew_lifetime = 36000 forwardable = true krb4_convert = false Make sure that the /etc/krb5.conf file is writeable by root:# chmod 644 /etc/krb5.confConfigure /etc/hosts to resolve WCCS hosts names (optional)Test name resolution by pinging the Active Directory server and the Windows Compute Cluster Server head node from the Linux machine. In this case, the Active Directory server is “” and the Windows Compute Cluster Server is “”. If the WCCS server names cannot be resolved, add them to DNS or to the Linux machines host file as shown below:# cat /etc/hosts# Do not remove the following line, or various programs# that require network functionality will fail.2 quasar0 ad1 winccs localhost.localdomain localhostYou should now be able to obtain a Kerberos ticket from the Windows Active Directory server as shown below:# kinit -f administratorWCCS.ZOHALLT.COMPassword for administratorWCCS.ZOHALLT.COM:Verify that you have a ticket:# klistTicket cache: FILE:/tmp/krb5cc_0Default principal: administratorWCCS.ZOHALLT.COMValid starting Expires Service principal01/16/08 13:38:43 01/16/08 23:40:09 krbtgt/WCCS.ZOHALLT.COMWCCS.ZOHALLT.COM renew until 01/17/08 13:38:43Kerberos 4 ticket cache: /tmp/tkt0klist: You have no tickets cachedInstall and Configure SambaBinary packages of Samba are included in almost all Linux and UNIX distributions and may have been installed when the operating system was installed. To find out if Samba is installed run (on RedHat Linux):# rpm -qa | grep sambasamba-common-3.0.10-1.4E.2samba-3.0.10-1.4E.2system-config-samba-1.2.21-1samba-client-3.0.10-1.4E.2samba-swat-3.0.10-1.4E.2If your Linux system does not have Samba, you can probably obtain binary packages from your vendors website or from the Samba home page. Alternatively, you can obtain the Samba source code from the Samba website. Please read How to Compile Samba (/samba/docs/man/Samba-HOWTO-Collection/compiling.html#id450289). Once Samba is installed, configure it by editing the /etc/samba/smb.conf file as shown below:global workgroup = WCCS realm = WCCS.ZOHALLT.COM preferred master = no server string = Linux File Server username map = /etc/samba/smbusers security = ADS encrypt passwords = yes log level = 3 log file = /var/log/samba/%m max log size = 50 printcap name = cups printing = cups winbind enum users = Yes winbind enum groups = Yes winbind use default domain = Yes winbind nested groups = Yes winbind separator = + idmap uid = 2000-3000 idmap gid = 2000-3000 template primary group = Domain Users template shell