installing CentOS 6 on a cluster node

installing CentOS 6 on computing nodes (cluster 10/11)

Warning: CentOS 6 has reached end-of-life (EOL). This guide is preserved for legacy cluster maintenance only.

This document records the procedure for installing CentOS 6 on a computing node in cluster 10/11, including network configuration, InfiniBand firmware, and queue system setup.

1. Prepare Installation Media (macOS example)

Follow the official CentOS guide for creating a bootable USB drive.https://docs.centos.org/en-US/centos/install-guide/Making_Media_USB_Mac

diskutil list
diskutil unmountDisk /dev/diskN      # replace N with disk number
sudo dd if=/path/to/image.iso of=/dev/rdiskN bs=1m
# Press Ctrl + t to check progress
diskutil eject /dev/diskN

Note: /dev/rdiskN is faster than /dev/diskN.

2. Install CentOS

Choose Software Development or Server installation.

Recommended disk partitioning (300 GB in total):

• /boot: 500 MB (ext3)
• swap: 64 GB
• /: remaining space

3. Disable SELinux

In file /etc/selinux/config, change SELINUX=enforcing to SELINUX=disabled.

4. Configure Network

Edit the following files and compare with other nodes:

/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-em1
/etc/sysconfig/network-scripts/ifcfg-ib0
/etc/hosts
/etc/hosts.equiv

Additional steps to clear SSH cache:

1. Delete /root/.ssh/known_hosts on the management node.
2. Copy /root/.ssh from the management node to the compute node.
3. Restart network: service network restart or /etc/init.d/network restart.

5. Create Local YUM Repository from ISO

Mount the ISO:

mount -o loop,ro centos6.iso /mnt/centos6-iso

Create /etc/yum.repos.d/os.repo:

[os]
name=CentOS6-OS
baseurl=file:///mnt/centos6-iso
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

Test:

yum list

6. Install InfiniBand Firmware (Mellanox ConnectX-3)

Please check the following blogs first:

1. https://blog.csdn.net/sinat_36458870/article/details/116596019
2. https://blog.csdn.net/force_eagle/article/details/46728665

6.1 Install dependencies

yum install tcl-devel tk-devel libmnl-devel

Install Mellanox Firmware Tools (MFT) from the official website.https://network.nvidia.com/products/adapter-software/firmware-tools In particular, we need mft-4.14.2.

6.2 Identify HCA firmware

Here we follow the Nvidia support page.https://www.mellanox.com/support/firmware/identification

> [root@node3_ib ~]# lspci | grep Mell
> 03:00.0 Network controller [0207]: Mellanox Technologies MT27500 Family [ConnectX-3]
> [root@node3_ib ~]# flint -d 03:00.0 q
> Image type: FS2
> FW Version: 2.40.5048
> FW Release Date: 5.3.2017
> Product Version: 02.40.50.48
> Rom Info: type=PXE version=3.4.746 devid=4099
> Device ID: 4099
> …
> PSID: DEL0A20000018

Record the PSID value, that is, DEL0A20000018.

6.3 Find firmware on Mellanox website

Since the PSID is from Dell, we need to download the firmware from https://www.mellanox.com/support/firmware/dell?mtag=oem_firmware_download. A complete list can be found in the Mellanox website.https://www.mellanox.com/support/firmware/firmware-downloads

6.4 Start MST

mst start
mst status

6.5 Burn firmware

flint -d /dev/mst/mt4099_pciconf0 -i fw-ConnectX3-xxxxx.bin burn

Reboot afterward.

6.6 Install OFED driver

The driver can be downloaded from the Mellanox website.https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed Specifically, we need 4.1-1.0.2.0 for CentOS 6.5 and install it:

MLNX_OFED_LINUX-4.1-1.0.2.0-rhel6.5-x86_64/mlnxofedinstall

Restart OpenIB:

/etc/init.d/openibd restart

Verify:

ofed_info | head
ibstat

7. Configure SGE (Sun Grid Engine)

All binaries are available on the IO node. Only /home and /backup need to be mounted.

• Create /backup
• Modify /etc/fstab
• Mount, run, and copy settings:

mount -a
export SGE_ROOT=/home/jointforce
cd /home/jointforce
./install_execd
cp /home/jointforce/default/common/settings* /etc/profile.d/

8. Disable iptables

service iptables save
service iptables stop
chkconfig iptables off

9. Configure NIS

Edit:

/etc/sysconfig/network
/etc/yp.conf
/etc/nsswitch.conf

Follow internal documentation or existing cluster configuration.This blog may be useful: http://cn.linux.vbird.org/linux_server/0430nis.php

10. Install EasyCluster

• Copy /opt/easycluster from an existing node.
• Add /opt/easycluster/background/easy_c in /etc/rc.local.
• Start EasyCluster.

11. Configure the system-wide environment

Copy necessary configuration files to /etc/profile.d/.

---

Final Remarks

This setup reflects legacy cluster configuration practices. For new deployments, consider using a modern distribution (e.g., Rocky Linux) and contemporary workload managers such as SLURM.

Cluster configuration should be documented carefully and updated whenever changes are made.