Rocm on power9

This page documents the process that has been used to attempt to build/compile the ROCm toolchain for Power9 based hardware.

Development testbed parameters

The testbed that is being used to prepare this system is | Pike. This is currently our only Power9 machine that has an AMD GPU in it. It is running Ubuntu 20-LTS.

The desired parameters of the installation are:

• Installation root: /packages/rocm/git (part of the ppc64le packages tree)

Compilation process

I am attempting to follow the Experimental_ROC collection of scripts, which are the closest I can find to a human-readable series of instructions, at [|| github]

1 - Environmental setup

TLD="/mnt/tmp/scratch/erik-k"
BASE_DIR="${TLD}/Experimental_ROC"
SOURCE_DIR="${TLD}/ROCK-Kernel-Driver"

2 - ROCK KERNEL DRIVER

Acquire code:

cd $(TLD)
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver.git
git clone https://github.com/RadeonOpenCompute/Experimental_ROC.git

These elements of the Experimental_ROC package are apparently necessary:

cd ROCK-Kernel-Driver
mkdir -p install_files
cp -RL ${BASE_DIR}/distro_install_scripts/Ubuntu/common/component_scripts/* ${SOURCE_DIR}/install_files

The critical configuration item in .config:

CONFIG_HSA_AMD=y

As of kernel 5.6.0 on Power9, this compile will fail building the KFD. This is because of an (outdated? unmaintained?) if/else #define inside the code, which copies from the fixed string "INT" instead of from another defined structure; Evidently the length of this string is insufficient, and padding it to "INT " causes the build error to resolve.

This is resolved by the following patch:

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 59557e3e2..e1ba64c44 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -978,7 +978,7 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
        }
 #else
        crat_table->oem_revision = 0;
-       memcpy(crat_table->oem_id, "INV", CRAT_OEMID_LENGTH);
+       memcpy(crat_table->oem_id, "INV          ", CRAT_OEMID_LENGTH);
        memcpy(crat_table->oem_table_id, "UNAVAIL", CRAT_OEMTABLEID_LENGTH);
 #endif
        crat_table->total_entries = 0;

3 - ROCT_Thunk-Interface

Acquire the code and compile:

cd ${TLD}
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROCT-Thunk-Interface.git
cd ROCT-Thunk-Interface
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git ../
make -j12
make install

This little requirement was fun to find out:

make install-dev

4 - ROCR-RUNTIME

Acquire code and compile:

cd ${TLD}
git clone -b rocm-3.5.x https://github.com/RadeonOpenCompute/ROCR-Runtime.git
cd ROCR-Runtime
mkdir build
cd build
#cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSAKMT_INC_PATH=/packages/rocm/git/include -DHSAKMT_LIB_PATH=/packages/rocm/git/lib ../src
make -j12
make install

5 - ROC-SMI

Acquire code and install:

cd ${TLD}
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROC-smi.git
cd ROC-smi
mkdir /packages/rocm/git/bin
cp rocm_smi.py /packages/rocm/git/bin

On our system it complains volubly about 'is' vs '==' in the python code; The lines can be edited to fix this.

6 - ROC-cmake

Fetch code and install:

cd ${TLD}
git clone https://github.com/RadeonOpenCompute/rocm-cmake.git
cd rocm-cmake
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git ../
make
make install

7 - rocinfo

This item has proven remarkably difficult to build and I still do not know what is wrong here:

CMake Error at CMakeLists.txt:60 (find_package):
  By not providing "Findhsa-runtime64.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "hsa-runtime64", but CMake did not find one.

  Could not find a package configuration file provided by "hsa-runtime64"
  (requested version 1.0) with any of the following names:

    hsa-runtime64Config.cmake
    hsa-runtime64-config.cmake

  Add the installation prefix of "hsa-runtime64" to CMAKE_PREFIX_PATH or set
  "hsa-runtime64_DIR" to a directory containing one of the above files.  If
  "hsa-runtime64" provides a separate development package or SDK, be sure it
  has been installed.

I can find nothing about this on Google, only complains of it finding *too many* HSAs.

The following garbage hack was enough to compile it:

• mkdir build; cd build; cmake ../
• Manually set the hsa-runtime64 location using ccmake to /packages/rocm/git and do configure inside cmake - it works (?)
• export C_INCLUDE_PATH="/packages/rocm/git/include/hsa:/packages/rocm/git/include:$C_INCLUDE_PATH"
• export CPLUS_INCLUDE_PATH="/packages/rocm/git/include/hsa:/packages/rocm/git/include:$CPLUS_INCLUDE_PATH"
• make

The final compile line idiotically seeks the wrong hsa-runtime64 library name entirely. The compile is trivial so I just fixed it myself,

/usr/bin/c++ -L/packages/rocm/git/lib -std=c++11 -fexceptions -fno-rtti -fno-math-errno -fno-threadsafe-statics -fmerge-all-constants -fms-extensions -Werror -Wall -m64 -ggdb -O0 -g CMakeFiles/rocminfo.dir/rocminfo.cc.o -o rocminfo -lhsa-runtime64

9 - roctracer

Fetch source and configure:

cd $TLD
git clone https://github.com/ROCm-Developer-Tools/roctracer.git
cd roctracer
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSA_RUNTIME_LIB=/packages/rocm/git/lib/libhsa-runtime64.so ../

This currently rails, requires (probably among other things) hip_runtime_api.h

• need to install HIP before this?

10 - rocprofiler

Fetch the source:

cd $TLD
git clone  https://github.com/ROCmSoftwarePlatform/rocprofiler.git
cd rocprofiler
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSA_RUNTIME_LIB=/packages/rocm/git/lib/libhsa-runtime64.so ../

This currently fails, requires roctracer header (see 8)