Rocm on power9

This page documents the process that has been used to attempt to build/compile the ROCm toolchain for Power9 based hardware.

Development testbed parameters

The testbed that is being used to prepare this system is | Pike. This is currently our only Power9 machine that has an AMD GPU in it. It is running Ubuntu 20-LTS.

The desired parameters of the installation are:

• Installation root: /packages/rocm/git (part of the ppc64le packages tree)

Compilation process

I am attempting to follow the Experimental_ROC collection of scripts, which are the closest I can find to a human-readable series of instructions, at [|| github]

1 - Environmental setup

TLD="/mnt/tmp/scratch/erik-k"
BASE_DIR="${TLD}/Experimental_ROC"
SOURCE_DIR="${TLD}/ROCK-Kernel-Driver"

2 - ROCK KERNEL DRIVER

Acquire code:

cd $(TLD)
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver.git
git clone https://github.com/RadeonOpenCompute/Experimental_ROC.git

These elements of the Experimental_ROC package are apparently necessary:

cd ROCK-Kernel-Driver
mkdir -p install_files
cp -RL ${BASE_DIR}/distro_install_scripts/Ubuntu/common/component_scripts/* ${SOURCE_DIR}/install_files

The critical configuration item in .config:

CONFIG_HSA_AMD=y

As of kernel 5.6.0 on Power9, this compile will fail building the KFD. This is because of an (outdated? unmaintained?) if/else #define inside the code, which copies from the fixed string "INT" instead of from another defined structure; Evidently the length of this string is insufficient, and padding it to "INT " causes the build error to resolve.

This is resolved by the following patch:

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 59557e3e2..e1ba64c44 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -978,7 +978,7 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
        }
 #else
        crat_table->oem_revision = 0;
-       memcpy(crat_table->oem_id, "INV", CRAT_OEMID_LENGTH);
+       memcpy(crat_table->oem_id, "INV          ", CRAT_OEMID_LENGTH);
        memcpy(crat_table->oem_table_id, "UNAVAIL", CRAT_OEMTABLEID_LENGTH);
 #endif
        crat_table->total_entries = 0;

3 - ROCT_Thunk-Interface

Acquire the code and compile:

cd ${TLD}
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROCT-Thunk-Interface.git
cd ROCT-Thunk-Interface
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git ../
make -j12
make install

This little requirement was fun to find out:

make install-dev

4 - ROCR-RUNTIME

Acquire code and compile:

cd ${TLD}
git clone -b rocm-3.5.x https://github.com/RadeonOpenCompute/ROCR-Runtime.git
cd ROCR-Runtime
mkdir build
cd build
#cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSAKMT_INC_PATH=/packages/rocm/git/include -DHSAKMT_LIB_PATH=/packages/rocm/git/lib ../src
make -j12
make install

5 - ROC-SMI

Acquire code and install:

cd ${TLD}
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROC-smi.git
cd ROC-smi
mkdir /packages/rocm/git/bin
cp rocm_smi.py /packages/rocm/git/bin

On our system it complains volubly about 'is' vs '==' in the python code; The lines can be edited to fix this.

6 - ROC-cmake

Fetch code and install:

cd ${TLD}
git clone https://github.com/RadeonOpenCompute/rocm-cmake.git
cd rocm-cmake
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git ../
make
make install

7 - rocinfo

This item has proven remarkably difficult to build and I still do not know what is wrong here:

CMake Error at CMakeLists.txt:60 (find_package):
  By not providing "Findhsa-runtime64.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "hsa-runtime64", but CMake did not find one.

  Could not find a package configuration file provided by "hsa-runtime64"
  (requested version 1.0) with any of the following names:

    hsa-runtime64Config.cmake
    hsa-runtime64-config.cmake

  Add the installation prefix of "hsa-runtime64" to CMAKE_PREFIX_PATH or set
  "hsa-runtime64_DIR" to a directory containing one of the above files.  If
  "hsa-runtime64" provides a separate development package or SDK, be sure it
  has been installed.

I can find nothing about this on Google, only complains of it finding *too many* HSAs.

The following garbage hack was enough to compile it:

• mkdir build; cd build; cmake ../
• Manually set the hsa-runtime64 location using ccmake to /packages/rocm/git and do configure inside cmake - it works (?)
• export C_INCLUDE_PATH="/packages/rocm/git/include/hsa:/packages/rocm/git/include:$C_INCLUDE_PATH"
• export CPLUS_INCLUDE_PATH="/packages/rocm/git/include/hsa:/packages/rocm/git/include:$CPLUS_INCLUDE_PATH"
• make

The final compile line idiotically seeks the wrong hsa-runtime64 library name entirely. The compile is trivial so I just fixed it myself,

/usr/bin/c++ -L/packages/rocm/git/lib -std=c++11 -fexceptions -fno-rtti -fno-math-errno -fno-threadsafe-statics -fmerge-all-constants -fms-extensions -Werror -Wall -m64 -ggdb -O0 -g CMakeFiles/rocminfo.dir/rocminfo.cc.o -o rocminfo -lhsa-runtime64

8 - Install HIP runtime

Prerequisites:

• Recommend to install Z3 theorem prover (LLVM and by extension HCC want it).
• apt install libxml2 libxml2-dev

cd z3
mkdir -p build-ppc64le; cd build-ppc64le
cmake ../ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/z3/git
make -j24
make install

Z3 module for /packages/modulefiles/z3/git:

#%Module########################################
##
## Z3 theorem prover
##

set _module_name  [module-info name]

module-whatis   "The Z3 theorem prover from MS Research"

proc ModulesHelp { } {
global _module_name
puts stderr "The $_module_name modulefile defines the default system paths"
puts stderr "and environment variables needed to use Z3."
}

##module-verbosity on

set _module_name  [module-info name]
set is_module_rm  [module-info mode remove]
set sys           [uname sysname]
set os            [uname release]
set mach          [uname machine]

set     SWTOP   /packages
set     ZTHREE  $SWTOP/z3/git

prepend-path    PATH $ZTHREE/bin

prepend-path    LD_LIBRARY_PATH $ZTHREE/lib

prepend-path    C_INCLUDE_PATH  $ZTHREE/include
prepend-path    CPLUS_INCLUDE_PATH  $ZTHREE/include

Note that /lib differs from path in x86_64 path which is /lib64.

8.1 - install HCC

Based upon instruction at:

https://rocmdocs.amd.com/en/latest/Installation_Guide/HCC-Compiler.html

Fetch the code. Find a nice paper to read, this isn't going to take just one hot minute...

cd $TLD
git clone --recursive -b clang_tot_upgrade https://github.com/RadeonOpenCompute/hcc.git
mkdir -p build; cd build

HCC configurator is unsmart, tell it things:

cmake ../ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/hcc/git -DHSA_HEADER=/packages/rocm/git/include/hsa/hsa.h -DHSA_LIBRARY=/packages/rocm/git/lib/libhsa-runtime64.so -DROCM_INSTALL_PATH=/packages/rocm/git -DROCM_ROOT=/packages/rocm/git</code>

There are a few more edits that are evidently required:

<code>build/include/kalmar_runtime.h:
#define _Float16 __fp16 </code>

from: https://reviews.llvm.org/D57188

<code> /packages/rocm/git/hsa/include/hsa/hsa.h: insert
#define LITTLEENDIAN_CPU
</code>

Without these compilation will fail riiiight at the veeeeery end.

<code>make -j8
make install

Note that building this includes building LLVM, which has some mind-blowingly massive link stages (as in using 10GB of ram each massive), so if the machine does not have a gigantic amount of memory, constraining -j may be recommended.

8.2 - Install HIP

Download code and set install options:

cd $TLD
git clone https://github.com/ROCm-Developer-Tools/HIP.git
cd HIP
mkdir -p build; cd build
CC=clang CXX=clang++ cmake -DLLVM_DIR=/packages/hcc/git -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/hip/git -DHSA_PATH=/packages/rocm/git/hsa -DROCM_PATH=/packages/git/rocm ..
make -j1

There is a reason for -j1. It barfs because of the _Float16 problem in the installed header. Fix it:

vi /packages/hcc/git/include/kalmar_math.h 
#define _Float16 __fp16 

Oh, this is nice:

fatal error: 'hip/hcc_detail/hip_prof_str.h' file not found

https://github.com/ROCm-Developer-Tools/HIP/issues/999

Further additions to the cmake configure line:

CC=clang CXX=clang++ cmake -DLLVM_DIR=/packages/hcc/git -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/hip/git -DHSA_PATH=/packages/rocm/git/hsa -DROCM_PATH=/packages/git/rocm -DUSE_PROF_API=1 -DHIP_PLATFORM=hcc ..

9 - roctracer

Fetch source and configure:

cd $TLD
git clone https://github.com/ROCm-Developer-Tools/roctracer.git
cd roctracer
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSA_RUNTIME_LIB=/packages/rocm/git/lib/libhsa-runtime64.so ../

This currently rails, requires (probably among other things) hip_runtime_api.h

• need to install HIP before this?

10 - rocprofiler

Fetch the source:

cd $TLD
git clone  https://github.com/ROCmSoftwarePlatform/rocprofiler.git
cd rocprofiler
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSA_RUNTIME_LIB=/packages/rocm/git/lib/libhsa-runtime64.so ../

This currently fails, requires roctracer header (see 8)