HPMC User Guide v 1.00
© 2022 Bassem W. Jamaleddine

The HPMC is equipped with a Xeon Phi MIC card, called a MIC coprocessor, that is plugged in its bus via the PCIe slot. This coprocessor is memory mapped device, and the motherboard is capable to manage such a memory mapped device on MMIO above 4GB.

Unlike legacy math coprocessor, the MIC is a device that runs its own Linux operating system and can be viewed as an embedded system that is plugged on the bus of the HPMC computer. The MIC uses the ring topology. Refer to LAN/WAN Network Programming with C for more information on the ring topology.

To manage the MIC the following package contains the required modules:
mpss-modules-3.10.0-693.17.1.el7.x86_64-3.8.4-1.x86_64. This package includes the following:

/etc/modprobe.d/mic.conf
/etc/sysconfig/modules/mic.modules
/etc/udev/rules.d/50-udev-mic.rules
/lib/modules/3.10.0-693.17.1.el7.x86_64/extra/mic.ko

/etc/modprobe.d/mic.conf
/etc/sysconfig/modules/mic.modules
/etc/udev/rules.d/50-udev-mic.rules
/lib/modules/3.10.0-693.17.1.el7.x86_64/extra/mic.ko

Since the MIC card is a coprocessor that is made available on a host computer as an embedded Linux system, it needs to be booted as a computer. For this to happen:

the MIC device driver must be loaded, hence to communicate and control the MIC hardware layer: lsmod | grep mic

the MIC is recognized as a device to the host computer: ls /dev/mic0

the MPSS software layer must be available on the host computer. If it has been installed then: which micinfo

To find information about your HPMC computer motherboard:

#  dmidecode -t 2

#  dmidecode -t memory

#  cat /proc/iomem

93000000-b5ffffff : PCI Bus 0000:16
  b5e00000-b5efffff : PCI Bus 0000:17
    b5e00000-b5e1ffff : 0000:17:00.0
      b5e00000-b5e1ffff : mic 

#  lspci | grep -i Phi

17:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor SE10/7120 series (rev 20) 

#  lspci -s 17:00.0 -t

-+-[0000:17]---00.0
 \-[0000:00]-

#  cat /proc/iomem

93000000-b5ffffff : PCI Bus 0000:16
  b5e00000-b5efffff : PCI Bus 0000:17
    b5e00000-b5e1ffff : 0000:17:00.0
      b5e00000-b5e1ffff : mic 

#  lspci -s 17:00.0 -v

17:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor SE10/7120 series (rev 20) 
        Subsystem: Intel Corporation Device 7d99
        Flags: bus master, fast devsel, latency 0, IRQ 41, NUMA node 0
        Memory at 381c00000000 (64-bit, prefetchable) [size=16G]
        Memory at b5e00000 (64-bit, non-prefetchable) [size=128K]
        Capabilities: [44] Power Management version 3
        Capabilities: [4c] Express Endpoint, MSI 00
        Capabilities: [88] MSI: Enable- Count=1/16 Maskable- 64bit+
        Capabilities: [98] MSI-X: Enable+ Count=16 Masked-
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: mic 

#  cat /proc/vmallocinfo | grep '381c00000000'

0xffffc90040000000-0xffffc90440001000 17179873280 ioremap_work+0x24/0xd0 [mic] phys=381c00000000 ioremap

#  cat /proc/vmallocinfo | grep 'b5e00000'

0xffffc90006a80000-0xffffc90006aa1000  135168 adapter_init_device+0x202/0x4b0 [mic] phys=b5e00000 ioremap

■ Adding the MIC to the Host Network Services

Configure your system network layers so that the Xeon Phi MIC devices can be daisy chained together and be part of your system. The MIC card is a high performance computer by itself that can be booted, configured, and made available for offloading math intensive computations. To have the MIC configured on your system, create a bridge to combine the multiple network adapters: your MIC cards and the different adapters can then talk to each others as if directly connected to a normal network switch.

In the directory /etc/sysconfig/network-scripts edit the file ifcfg-br0:

DEVICE=br0
TYPE=Bridge
ONBOOT=yes
DELAY=0
NM_CONTROLLED="no"
BOOTPROTO=static
IPADDR=192.168.0.110
NETMASK=255.255.255.0
GATEWAY=192.168.0.1
DNS1=209.18.47.61
MTU=64512

Find all your network devices and have them conneted to that bridge: Issue the command:

#  ifconfig

you will see which devices are available, then for each device have an entry in the /etc/sysconfig/network-scripts.

For example, if ifconfig reports eno1 and enp2s0, then edit both files:

ifcfg-eno1:
-----------
DEVICE=eno1
NAME=eno1
NM_CONTROLLED=no
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
BRIDGE=br0

The mic0 device is bridged to br0

#  vi /etc/sysconfig/network-scripts/ifcfg-mic0

ifcfg-mic0:
-----------
DEVICE=mic0
ONBOOT=yes
NM_CONTROLLED="no"
BRIDGE=br0
MTU=64512
MACADDR=0a:c9:28:ae:ba:4f

Each MIC has a configuration file, it is located in the /etc/mpss directory. We will configure the MIC0 Edit the /etc/mpss/mic0.cfg

 

■ Configuring the MIC for Bootstrapping

#  vi /etc/mpss/mic0.conf

Version 1 1

# Include configuration common to all MIC cards
Include default.conf

# Include all additional functionality configuration files by default
Include "conf.d/*.conf"

# Base filesystem for embedded Linux file system
Base CPIO /usr/share/mpss/boot/initramfs-knightscorner.cpio.gz

# Family type of MIC card
Family x100

# MPSS version
MPSSVersion 3.x

# Root device for MIC card
RootDevice StaticRamFS /var/mpss/mic0.image.cpio.gz

# Unique per card files for embedded Linux file system
### MicDir /var/mpss/mic0 /var/mpss/mic0.filelist  # The /var/mpss/mic0.filelist filelist argument deprecated.
MicDir /var/mpss/mic0

# Hostname to assign to MIC card
HostName mic0.local

# MAC address configuration
MacAddrs 0a:c9:28:ae:ba:4f 8e:d9:75:b8:3e:60

Network class=StaticBridge bridge=br0 micip=192.168.0.31 modhost=yes modcard=yes

# MIC OS Verbose messages to console
VerboseLogging Enabled

# MIC OS image
OSimage /usr/share/mpss/boot/bzImage-knightscorner /usr/share/mpss/boot/System.map-knightscorner

# Boot MIC card when MPSS stack is started
BootOnStart Enabled

# Control card power state setting
PowerManagement cpufreq_on;corec6_off;pc3_on;pc6_off

Cgroup memory=disabled

Overlay Simple /pubovl/tools /tmp On

 

■ Configuring the MIC KnightsCorner for Bootstrapping

If your HPMC has the KnightsCorner MIC the following mic0.conf is used.

#  vi /etc/mpss/mic0.conf

# MIC OS image
OSimage /usr/share/mpss/boot/bzImage-knightscorner /usr/share/mpss/boot/System.map-knightscorner

 

■ Booting the MIC

If your HPMC computer has a Xeon Phi CPU 7250 processor then there is no need for MPSS as the cores are available like native CPU cores.

If your HPMC computer has a MIC processor Xeon Phi coprocessor 71xx then MPSS is required and a network drivers must be started from the kernel during kernel startup.

If your HPMC computer has a MIC processor Xeon Phi coprocessor 72xx then MPSS is required and the network card is started as a service after kernel startup.

 

■ Adding a Swap Space to the MIC

On the HPMC

#  mkdir /ramfs

#  mount -t ramfs ramfs /ramfs/

#  ls /ramfs

#  dd bs=512M if=/dev/zero of=/ramfs/ram1 count=48

#  ls -l /sys/class/mic/mic0/virtblk_file

#  cat /sys/class/mic/mic0/virtblk_file

#  echo /ramfs/ram1 > /sys/class/mic/mic0/virtblk_file

#  cat /sys/class/mic/mic0/virtblk_file

#  df -a | grep ramfs

To add the swap to the MIC configuration, edit the default.conf within the MIC and add the following two lines:

#  vim /etc/mpss/default.conf

Add the following two lines:

ExtraCommandLine "vfs_read_optimization=on"
ExtraCommandLine "vfs_write_optimization=on"

#  service mpss stop

#  micctrl --resetconfig

#  service mpss start

ssh to the MIC

#  ssh mic0

On the MIC

#  /sbin/modprobe mic_virtblk

#  /sbin/modprobe -l | grep virt

#  /sbin/mkswap /dev/vda

#  /sbin/swapon /dev/vda

#  cat /proc/swaps

#  free -m

 

■ Adding Users to the MIC

Each HPMC computer has its own MIC with mic0 as a hostname. When you login to your HPMC, you can ssh to the mic0: ssh mic0.

mic0 is only accessible as a Linux system from the board itself, this means you need to set up ssh-keygen, then you need to bring the mic0 interface up.



Here we will add the user "elsa" to the MIC of hpmc7. First add the user to hpmc7

#  adduser elsa

#  passwd elsa

#  usermod -a -G hpcusr elsa
     needed to access /dev/mic/*

#  gpasswd -d hpcusr elsa
     to remove elsa from group hpcusr

ssh-keygen for user elsa, and copy the key from /home/elsa/.ssh:

#  cat /elsa/.ssh/id_rsa.pub > /var/mpss/mic0/elsa/.ssh/authorized_keys

If your MIC is up then you need to reset it:

#  micctrl --resetconfig -v

check MIC if is in ready state

#  micctrl -s

If MIC is not in ready state then reset it

#  micctrl -rw

check MIC if is in ready state

#  micctrl -s

If MIC cannot be in ready state then you need to flash it (micflash -update) and reboot the host, and reset the MIC (micctrl -rw) until it is in ready state (mictrl -s) once the MIC is in ready state then bring up MPSS:

#  systemctl start mpss

for physical condition and information about the MIC:

#  micinfo

#  micctrl --adduser=elsa
     do it after creating ssh-keygen on host

This will copy the .ssh/id_rsa.pub and copy the same to .ssh/authorize d_keys on elsa's mic0 home, it is the same as on /var/mpss/mic0/home/elsa/.ssh/

Login as elsa and ssh to mic0:

#  su - elsa

#  ssh mic0
     now user elsa can ssh to the MIC

 

■ MIC hostname

Each HPMC computer has its own MIC with mic0 as a hostname. When you login to your HPMC, you can ssh to the mic0: ssh mic0.

#  ssh mic0

The HPMC mic0 is also visible to any neighboring HPMCs on the network, however its hostname must be prefixed with the HPMC hostname. For example, the HPMC computer whose hostname is hpmc7 will have the MIC whose hostname is hpmc7mic0. Computer on the same network can ssh to hpmc7's MIC using this command:

#  ssh hpmc7mic0

Before you ssh to a MIC you need to make sure the mpss is up on the hostname where the MIC card is inserted.

#  systemctl status mpss
     make sure mpss service is started

 

■ Mount NFS to the MIC

For example, the host mm03 is an HPMC computer that has a MIC. The following procedure shows how to mount the directory /mm03fs over NFS to the MIC.

On host mm03 have NFS started at boot:

#  systemctl start nfs-server

#  systemctl enable nfs-server
     (enable to start after each reboot)

Add the NFS to the MIC mic0 using the following command

#  micctrl --addnfs=mm03:/mm03fs -d /mm03fs mic0 -vv

   [Info] mic0: [Parse] /etc/mpss/mic0.conf
   [Info] mic0: [Parse] Configuration version 1.1
   [Info] mic0: [Parse] /etc/mpss/default.conf
[Warning] mic0: Server mm03 may not be reachable if the internal bridge br0 is not routed out of the host
[Warning] Modified existing NFS entry for MIC card path '/mm03fs'
[Filesys] mic0: Update /var/mpss/mic0/etc/fstab remove NFS /mm03fs entry
[Filesys] mic0: Update /var/mpss/mic0/etc/fstab with NFS mount mm03:/mm03fs at /mm03fs

Reset the MIC

#  micctrl -r

Check that the reset is successful

#  dmesg

  
[54291.465126] mic0: Transition from state resetting to ready
 18 ...
 19 [54290.463181] mic0: Resetting (Post Code 10)
 20 [54291.465114] mic0: Resetting (Post Code 12)
 21 [54291.465126] mic0: Transition from state resetting to ready

Restart the MIC

#  systemctl restart mpss

#  micctrl --rootdev=RamFs --target=/etc/mpss/lg3.cpio.gz -vv

   [Info] mic0: [Parse] /etc/mpss/mic0.conf
   [Info] mic0: [Parse] Configuration version 1.1
   [Info] mic0: [Parse] /etc/mpss/default.conf
[Filesys] mic0: Update RootDevice in /etc/mpss/mic0.conf

 

■ Enable the MIC mpss Service at Boot

MPSS must be installed on the host computer. The host computer is the system where the MIC card has been installed. Make sure MPSS is installed on the HPMC. On the HPMC the following directories and configuration files are evidence that the MPSS has been installed on the HPMC:

 /opt/mpss
 /etc/mpss/mic0.conf
 /var/mpss/mic0

To enable MPSS service so that the MPSS is up after each reboot. Assuming you won't make changes to the MIC kernel image:

#  systemctl enable mpss