Rocm on power9

This page documents the process that has been used to attempt to build/compile the ROCm toolchain for Power9 based hardware.

Development testbed parameters

The testbed that is being used to prepare this system is Pike. This is currently our only Power9 machine that has an AMD GPU in it. It is running Ubuntu 20-LTS.

The desired parameters of the installation are:

• Installation root: /packages/rocm/git (part of the ppc64le packages tree)

Currently, progress has hit a hard stall: rocminfo reports NO agents available,

Loading hcc/git_3.1
  Loading requirement: z3/git
Loaded llvm/10, rocm and hcc
Currently Loaded Modulefiles:
 1) llvm/10_ubuntu   2) rocm/git   3) z3/git   4) hcc/git_3.1  
erik-k@pike ~ $ rocminfo
ROCk module is loaded
Able to open /dev/kfd read-write
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
System Timestamp Freq.:  0.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             

==========               
HSA Agents               
==========               
*** Done ***

This occurs in the code when hsa_enumerate_agents is called. GDB is unable to proceed into the HSA code itself: Debug compile links some proprietary .so which is only available as an x86 .so and will not run (in gdb or by itself).

Compilation process

I am attempting to follow the Experimental_ROC collection of scripts, which are the closest I can find to a human-readable series of instructions, at [|| github]

1 - Environmental setup

TLD="/mnt/tmp/scratch/erik-k"
BASE_DIR="${TLD}/Experimental_ROC"
SOURCE_DIR="${TLD}/ROCK-Kernel-Driver"

2 - ROCK KERNEL DRIVER

Acquire code:

cd $(TLD)
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver.git
git clone https://github.com/RadeonOpenCompute/Experimental_ROC.git

These elements of the Experimental_ROC package are apparently necessary:

cd ROCK-Kernel-Driver
mkdir -p install_files
cp -RL ${BASE_DIR}/distro_install_scripts/Ubuntu/common/component_scripts/* ${SOURCE_DIR}/install_files

The critical configuration item in .config:

CONFIG_HSA_AMD=y

As of kernel 5.6.0 on Power9, this compile will fail building the KFD. This is because of an (outdated? unmaintained?) if/else #define inside the code, which copies from the fixed string "INT" instead of from another defined structure; Evidently the length of this string is insufficient, and padding it to "INT " causes the build error to resolve.

This is resolved by the following patch:

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 59557e3e2..e1ba64c44 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -978,7 +978,7 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
        }
 #else
        crat_table->oem_revision = 0;
-       memcpy(crat_table->oem_id, "INV", CRAT_OEMID_LENGTH);
+       memcpy(crat_table->oem_id, "INV          ", CRAT_OEMID_LENGTH);
        memcpy(crat_table->oem_table_id, "UNAVAIL", CRAT_OEMTABLEID_LENGTH);
 #endif
        crat_table->total_entries = 0;

3 - ROCT_Thunk-Interface

Acquire the code and compile:

cd ${TLD}
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROCT-Thunk-Interface.git
cd ROCT-Thunk-Interface
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git ../
make -j12
make install

This little requirement was fun to find out:

make install-dev

4 - ROCR-RUNTIME

Acquire code and compile:

cd ${TLD}
git clone -b rocm-3.5.x https://github.com/RadeonOpenCompute/ROCR-Runtime.git
cd ROCR-Runtime
mkdir build
cd build
#cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSAKMT_INC_PATH=/packages/rocm/git/include -DHSAKMT_LIB_PATH=/packages/rocm/git/lib ../src
make -j12
make install

5 - ROC-SMI

Acquire code and install:

cd ${TLD}
git clone -b roc-3.5.x https://github.com/RadeonOpenCompute/ROC-smi.git
cd ROC-smi
mkdir /packages/rocm/git/bin
cp rocm_smi.py /packages/rocm/git/bin

On our system it complains volubly about 'is' vs '==' in the python code; The lines can be edited to fix this.

6 - ROC-cmake

Fetch code and install:

cd ${TLD}
git clone https://github.com/RadeonOpenCompute/rocm-cmake.git
cd rocm-cmake
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git ../
make
make install

7 - rocinfo

This item has proven remarkably difficult to build and I still do not know what is wrong here:

CMake Error at CMakeLists.txt:60 (find_package):
  By not providing "Findhsa-runtime64.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "hsa-runtime64", but CMake did not find one.

  Could not find a package configuration file provided by "hsa-runtime64"
  (requested version 1.0) with any of the following names:

    hsa-runtime64Config.cmake
    hsa-runtime64-config.cmake

  Add the installation prefix of "hsa-runtime64" to CMAKE_PREFIX_PATH or set
  "hsa-runtime64_DIR" to a directory containing one of the above files.  If
  "hsa-runtime64" provides a separate development package or SDK, be sure it
  has been installed.

I can find nothing about this on Google, only complains of it finding *too many* HSAs.

The following garbage hack was enough to compile it:

• mkdir build; cd build; cmake ../
• Manually set the hsa-runtime64 location using ccmake to /packages/rocm/git and do configure inside cmake - it works (?)
• export C_INCLUDE_PATH="/packages/rocm/git/include/hsa:/packages/rocm/git/include:$C_INCLUDE_PATH"
• export CPLUS_INCLUDE_PATH="/packages/rocm/git/include/hsa:/packages/rocm/git/include:$CPLUS_INCLUDE_PATH"
• make

The final compile line idiotically seeks the wrong hsa-runtime64 library name entirely. The compile is trivial so I just fixed it myself,

/usr/bin/c++ -L/packages/rocm/git/lib -std=c++11 -fexceptions -fno-rtti -fno-math-errno -fno-threadsafe-statics -fmerge-all-constants -fms-extensions -Werror -Wall -m64 -ggdb -O0 -g CMakeFiles/rocminfo.dir/rocminfo.cc.o -o rocminfo -lhsa-runtime64

8 - Install HIP runtime

Prerequisites:

• Recommend to install Z3 theorem prover (LLVM and by extension HCC want it).
• apt install libxml2 libxml2-dev

cd z3
mkdir -p build-ppc64le; cd build-ppc64le
cmake ../ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/z3/git
make -j24
make install

Z3 module for /packages/modulefiles/z3/git:

#%Module########################################
##
## Z3 theorem prover
##

set _module_name  [module-info name]

module-whatis   "The Z3 theorem prover from MS Research"

proc ModulesHelp { } {
global _module_name
puts stderr "The $_module_name modulefile defines the default system paths"
puts stderr "and environment variables needed to use Z3."
}

##module-verbosity on

set _module_name  [module-info name]
set is_module_rm  [module-info mode remove]
set sys           [uname sysname]
set os            [uname release]
set mach          [uname machine]

set     SWTOP   /packages
set     ZTHREE  $SWTOP/z3/git

prepend-path    PATH $ZTHREE/bin

prepend-path    LD_LIBRARY_PATH $ZTHREE/lib

prepend-path    C_INCLUDE_PATH  $ZTHREE/include
prepend-path    CPLUS_INCLUDE_PATH  $ZTHREE/include

Note that /lib differs from path in x86_64 path which is /lib64.

8.1 - install HCC

Based upon instruction at:

https://rocmdocs.amd.com/en/latest/Installation_Guide/HCC-Compiler.html

Fetch the code. Find a nice paper to read, this isn't going to take just one hot minute...

cd $TLD
git clone --recursive -b clang_tot_upgrade https://github.com/RadeonOpenCompute/hcc.git
mkdir -p build; cd build

HCC configurator is unsmart, tell it things:

cmake ../ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/hcc/git -DHSA_HEADER=/packages/rocm/git/include/hsa/hsa.h -DHSA_LIBRARY=/packages/rocm/git/lib/libhsa-runtime64.so -DROCM_INSTALL_PATH=/packages/rocm/git -DROCM_ROOT=/packages/rocm/git

The idiotcmake is hardcoded to look for /usr/bin/cc and will ignore the result of 'which gcc' and $CC (same c++). Thus on some platforms (glares at RHEL-7) it will be necessary to add

-DCMAKE_C_COMPILER=$(which gcc) -DCMAKE_CXX_COMPILER=$(which g++)

to the above.

There are a few more edits that are evidently required:

build/include/kalmar_runtime.h:
#define _Float16 __fp16 

from: https://reviews.llvm.org/D57188

 /packages/rocm/git/hsa/include/hsa/hsa.h: insert
#define LITTLEENDIAN_CPU

Without these compilation will fail riiiight at the veeeeery end.

make -j8
make install

Note that building this includes building LLVM, which has some mind-blowingly massive link stages (as in using 10GB of ram each massive), so if the machine does not have a gigantic amount of memory, constraining -j may be recommended.

8.2 - Install HIP

spoiler - skip to end to bypass my stream of consciousness and get the build instructions

Download code and set install options:

cd $TLD
git clone https://github.com/ROCm-Developer-Tools/HIP.git
cd HIP
mkdir -p build; cd build
CC=clang CXX=clang++ cmake -DLLVM_DIR=/packages/hcc/git -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/hip/git -DHSA_PATH=/packages/rocm/git/hsa -DROCM_PATH=/packages/git/rocm ..
make -j1

There is a reason for -j1. It barfs because of the _Float16 problem in the installed header. Fix it:

vi /packages/hcc/git/include/kalmar_math.h 
#define _Float16 __fp16 

Oh, this is nice:

fatal error: 'hip/hcc_detail/hip_prof_str.h' file not found

https://github.com/ROCm-Developer-Tools/HIP/issues/999

Further additions to the cmake configure line:

CC=clang CXX=clang++ cmake -DLLVM_DIR=/packages/hcc/git -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/hip/git -DHSA_PATH=/packages/rocm/git/hsa -DROCM_PATH=/packages/git/rocm -DUSE_PROF_API=1 -DHIP_PLATFORM=hcc ..

This is pissy:

CMake Warning at CMakeLists.txt:238 (MESSAGE):
  Profiling API header not found.  Disabling roctracer integration.  Use
  -DPROF_API_HEADER_PATH=<path to prof_protocol.h header>

In file included from /mnt/tmp/scratch/erik-k/HIP/src/hip_hcc_internal.h:33:
/mnt/tmp/scratch/erik-k/HIP/src/hip_prof_api.h:9:10: fatal error: 'hip/hcc_detail/hip_prof_str.h' file not found
#include "hip/hcc_detail/hip_prof_str.h"

Appparently this may have to be manually generated???

https://github.com/ROCm-Developer-Tools/HIP/issues/999

Relevant parts of the cmakefile :eyeroll: :

if(HIP_PLATFORM STREQUAL "hcc")
if(USE_PROF_API EQUAL 1)
set(PROF_API_STR "${CMAKE_CURRENT_SOURCE_DIR}/include/hip/hcc_detail/hip_prof_str.h")
set(PROF_API_HDR "${CMAKE_CURRENT_SOURCE_DIR}/include/hip/hcc_detail/hip_runtime_api.h")
set(PROF_API_SRC "${CMAKE_CURRENT_SOURCE_DIR}/src")
set(PROF_API_GEN "${CMAKE_CURRENT_SOURCE_DIR}/hip_prof_gen.py")
set(PROF_API_LOG "${PROJECT_BINARY_DIR}/hip_prof_gen.log.txt")
set(PROF_API_CMD "${PROF_API_GEN} -v ${OPT_PROF_API} ${PROF_API_HDR} ${PROF_API_SRC} ${PROF_API_STR} >${PROF_API_LOG}")
MESSAGE(STATUS "Generating profiling promitives: ${PROF_API_STR}")
execute_process(COMMAND sh -c "rm -f ${PROF_API_STR}; ${PROF_API_CMD}")
set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS ${PROF_API_GEN} ${PROF_API_HDR} ${PROF_API_STR})

Relevant discovery after manual execution: File is hardcoded to '/usr/bin/python' and ubuntu20 has python3 binary.

Changed ../hip_prof_gen.py interpreter line and retrying cmake.

SUCCESS!

Script to build/install:

cd $TLD
git clone https://github.com/ROCm-Developer-Tools/HIP.git
cd HIP

Edit hip_prof_gen.py line 1 to set interpreter to '/usr/bin/python3'

mkdir -p build; cd build
CC=clang CXX=clang++ cmake -DLLVM_DIR=/packages/hcc/git -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/hip/git -DHSA_PATH=/packages/rocm/git/hsa -DROCM_PATH=/packages/git/rocm -DUSE_PROF_API=1 -DHIP_PLATFORM=hcc ..
make -j8
make install

This directory is likely to be important:

/packages/hip/git/cmake

HIP modulefile:

#%Module########################################
##
## HIP system
##

set _module_name  [module-info name]

module-whatis   "The AMD HIP compiler via Git"

proc ModulesHelp { } {
global _module_name
puts stderr "The $_module_name modulefile defines the default system paths"
puts stderr "and environment variables needed to use HIP."
}

#module-verbosity on

set _module_name  [module-info name]
set is_module_rm  [module-info mode remove]
set sys           [uname sysname]
set os            [uname release]
set mach          [uname machine]

if {[is-loaded gcc] && !$is_module_rm } { module unload gcc }

module load rocm/git
module load z3/git
module load hcc/git_3.1

set     SWTOP   /packages/hip/git

prepend-path    PATH $SWTOP/bin

prepend-path    LD_LIBRARY_PATH $SWTOP/lib

prepend-path    C_INCLUDE_PATH  $SWTOP/include
prepend-path    CPLUS_INCLUDE_PATH  $SWTOP/include

9 - roctracer

Fetch source and configure:

cd $TLD
git clone https://github.com/ROCm-Developer-Tools/roctracer.git
cd roctracer
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSA_RUNTIME_LIB=/packages/rocm/git/lib/libhsa-runtime64.so ../

This currently rails, requires (probably among other things) hip_runtime_api.h

• need to install HIP before this?

10 - rocprofiler

Fetch the source:

cd $TLD
git clone  https://github.com/ROCmSoftwarePlatform/rocprofiler.git
cd rocprofiler
mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/packages/rocm/git -DHSA_RUNTIME_LIB=/packages/rocm/git/lib/libhsa-runtime64.so ../

This currently fails, requires roctracer header (see 8)