COMEX:

Server type-1 (32 units)

• Dell PowerEdge 1950 [Ref]
• 2 x Intel® Xeon® CPU 5160 [Ref]
• 64 GB. DDR2 Memory
• Mellanox Technologies MT27500 Family [RDMA-Enabled NIC] [Ref]
• CentOS release 6.9

Server type-2 (12 units)

• Dell PowerEdge 1950 [Ref]
• 2 x Intel® Xeon® CPU 5160 [Ref]
• 32 GB. DDR2 Memory
• Mellanox Technologies MT27500 Family [RDMA-Enabled NIC] [Ref]
• CentOS release 6.9

RDMA-enabled switch type-1 (1 unit)

• Mellanox SX1012X Open Ethernet Switch [Ref]
• 12 QSFP ports 40/56GbE
• Lowest latency: 220nsec for 40GbE

RDMA-enabled switch type-2 (1 unit)

• Mellanox MSX1024B-1BFS-RF SwitchX-2 Open Ethernet Switch [Ref]
• 48 SFP+ ports 10GbE
• 12 QSFP ports 40/56GbE
• Lowest latency: 270nsec for 10GbE
• Lowest latency: 220nsec for 40GbE

This description comprises the software required to reproduce experiments presented for COMEX. It describes the software dependencies and specific features. We also explain how to compile, install, and run our software set in fine detail. All results were collected from real hardware runs and subject to be reproducible under the given system environment.

Architecture Simulation Tools

• The gem5 Simulator, a modular platform for computer-system architecture research.

Software Dependencies

• GCC 4.4.7, the GNU Compiler Collection
• Mellanox OpenFabrics Enterprise Distribution 3.3-1.0.4.0 *** important ***
  [Driver Compatibility Matrix]

Benchmarks Suits

• NAS Parallel Benchmarks (NPB)
• Problem Based Benchmark Suite (PBBS)

Testbed Preparation

1. Prepare testbed with CentOS 6.9 operating system, available at CentOS archive page.
2. Download Linux kernel mainline 3.10.87 from Linux foundation archive page.
   wget https://mirrors.edge.kernel.org/pub/linux/kernel/v3.x/linux-3.10.87.tar.gz
3. Extract the kernel source code package.
tar -zxf linux-3.10.87.tar.gz
4. Patch the Linux kernel source code with COMEX Kernel Extension patch. *** Stable version is currently unavailable ***
5. We recommend the default kernel configuration file for kernel installation, which comes with CentOS distribution in /boot/ directory. Copy the configuration file to the extracted kernel source directory as ".config".
   Example:
   cp /boot/config-2.6.32-696.13.2.el6.x86_64 linux-3.10.87/.config
6. Compile and install Linux kernel.
   make clean
make -j 4
make -j 4 modules
make headers_install
make modules_install
make install
7. For options that the system may ask during the installation process, we recommend default values (simply press “enter” to accept).
8. After installation, reboot the system once.
9. Check the grub bootloader configuration in /boot/grub/. Make sure that the default kernel (usually at Line 10) matches our newly installed kernel entry (i.e., title CentOS 3.10.87). Please note that entries start from 0.
   vi /boot/grub/grub.conf
10. Add “transparent_hugepage=never” into the kernel option entry to simply disable transparent hugepage feature since COMEX prototype does not support this feature.
11. The bootloader entry may looks this.
    # grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (3.10.87)
root (hd0,0)
kernel /vmlinuz-3.10.87 ro root=UUID=546aaf77-a60b-46f3-a117-1507805dcc93 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet no_console_suspend=1 log_buf_len=4M transparent_hugepage=never
12. Download the Mellanox OFED driver 3.3-1.0.4.0 for RHEL/CentOS from the Mellanox official page
13. Extract the downloaded package and compile the RDMA OFED driver.
    tar xf MLNX_OFED_LINUX-3.3-1.0.4.0-rhel6.8-x86_64.tgz
cd MLNX_OFED_LINUX-3.3-1.0.4.0-rhel6.8-x86_64
./mlnx_add_kernel_support.sh --yes --distro rhel6.9 -m .
14. Extract the newly created package (in /tmp/ directory) and install RDMA OFED Driver. Accept all default configuration with option "--force".
    tar xf /tmp/MLNX_OFED_LINUX-3.3-1.0.4.0-rhel6.9-x86_64-ext.tgz
cd MLNX_OFED_LINUX-3.3-1.0.4.0-rhel6.9-x86_64-ext
./mlnxofedinstall --distro rhel6.9 --force
15. Reboot the system.
16. Set network interface card address and verify network connectivity. For example, ping command.
17. The testbed is ready for COMEX.