HMC AIX Setup
This page will document the installation procedure for a system controlled by an IBM (v)HMC.
The installation steps are in summary,
- Install VIOS
- Create virtual network bridges
- Create virtual disks and assign
- Install AIX
- Setup AIX
Hardware prerequisites assumed: - At least 1 storage device on SATA/SAS - Network port connected to HMC - At least 2 external network ports connected, 1 to public & 1 to private network
VIOS install
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use
chfs -a size=+10G /
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as
PwrVMVIOSBIV3.1.3.10Fls92021.iso
References:
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios
Top -> System -> 'create vio server'
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch & that the switch has that port on the right vlan.
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?
The finishing of the install procedure is unreliable. It may or may not report "done". As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'
Next step:
Server -> vio servers -> [click server] -> vios actions -> console -> open terminal
After a bit, a godawfully barely-functional console will ask to run.
Enter a password
Accept license.
Run oem_setup_env to get prompted again, and accept license.
Okay, new blank slate VIOS is installed.
Partitions
At this point we also want to create OS partitions because we will need them available when we setup virtual disks
System -> top 'create partition' -> name it
VIOS configuration
Virtual networking configuration: https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge
System -> powervm -> virtual networks -> add virtual network
name: brpriv Bridged: yes, tagged no, pvid 172, use default switch, next
Enable jumbo frames [critical for private network!] and LSO, next
Assign backing device from list. Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems. If this is chosen wrong, the only apparent option is delete the whole virbr and start over.
It will be created and use the default 802.3 virtual switch.
Now go back and greate the public bridge:
system -> powervm -> virtual networks -> add virtual network
name: brpub Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next
ok, create
Virtual install library setup
Update: See this for installing new things on v10.2 hmc,
https://www.ibm.com/docs/en/power10/000V-HMC?topic=images-manage-virtual-io-server
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).
SSH to the vios, username 'padmin'
oem_setup-env mkdir -p /Maingroup/images cd /Maingroup/images chfs -a size=+15G / # to enlarge storage sufficiently
using scp, ISO images are available on erik-k's downloads:
aix_7200-05-03-2136_flash_092021.iso rhel-8.2-ppc64le-dvd.iso ubuntu-20.04.1-live-server-ppc64el.iso
SCP these to to /Maingroup/images
Now go to system -> virtual storage -> vio server -> manage
optical devices -> create virtual library -> 25GB
optical devices -> action -> add media -> from existing file
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.
Partition configuration
systems -> powervm -> virtual storage
select vio server -> action -> manage
click storage pools -> Create a storage pool & assign the NVMEs to it
click virtual disks -> create
name: 'aixroot' or something pool: nvmepool size: 500G e.g. assign to partition: aix-part
create & assign adapter on partition -> yes please
Now click system -> partitions -> [partition] -> virtual networks -> attach virtual network
[X] show and attaach new adapters [X] check brpub and brpriv to connect to both networks
Click system -> virtual storage -> [vios] -> manage
virtual optical devices -> [select image] -> modify assignment to partition we're installing -> ok
AIX install
After assigning virtual networks, virtual disk drive & aix-7.2 install media,
system -> partitions -> aix partition -> start
system -> partitions -> aix partition -> system actions -> console -> open
AIX INSTALL CRITICAL: Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.
Look at the devices and confirm you have a scsi disk and a cd drive
AIX configuration
Upon firstboot, smitty comes up.
Set timezone to use/los angeles Use passwd to set password
Run
df -sm
to see what an infinitesimal size of disk has been assigned to a bunch of partitions. Use
chfs -a size=+xG /filesystem
to grow them. Remember we have 500G of virtual disk!
mkdir /root
cd /etc, open passwd in vi,
edit root's homedir to /root
Ethernet configuration
lsdev | grep -i eth -> should print en0 and en1
cd /etc vi dhcpcd.ini
Page down to the bottom, append,
interface en0 {
option 12 "cirrus-aix"
}
interface en1 {
option 12 "cirrus-aix"
}
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.
Use system -> virtual networking diagram to find which interface is connected to which network.
Use ifconfig -a to get MAC addresses or
entstat -d enX | grep 'Hard'
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.
Once this is done,
startsrc -s dhcpcd
to connect and get IP addresses.
Now we have to configure jumbo frames on the host,
ifconfig -a
will reveal which adapter is mated to the private network, N
See: https://developer.ibm.com/articles/au-aix-largesend-jumboframes/
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,
chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0
Check
lsattr -El (device) | grep -e mtu -e jumbo
Check afterwards from another oaciss machine (because aix ping command is stupid) too:
orthus# ping -s 9000 cumulus-aix.stor
SSH
Go to /etc/ssh and edit sshd_config
Change ListenAddress to the private interface address for now.
stopsrc -s sshd startsrc -s sshd
This will temporarily secure the system by restricting ssh to the private network
Care and feeding package
scp erik-k@orthus:~/ibmset.tar /root
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.
Yum
cd yum; rpm -ivh *
Once this is installed,
yum install -y bash wget sudo vim tar gcc-gfortran gcc-c++ emacs blas freetype2-devel lapack libpng-devel python3-devel xz 0mq-devel binutils-devel coreutils blas-devel openblas-devel matplotlib libjpeg-devel openblas-devel ImageMagick-devel vim-X11 lua-devel tcl-devel tk-devel findutils gdb cmake
Bash is now available, yay
Edit /root/.bashrc:
PATH="/opt/freeware/bin:$PATH"
In aix 7.3, this occurred,
# /opt/freeware/bin/yum There was a problem importing one of the Python modules required to run yum. The error leading to this problem was: No module named rpm Please install a package which provides this module, or verify that the module is installed correctly. It's possible that the above module doesn't match the current version of Python, which is: 2.7.10 (default, Jun 22 2016, 05:57:59) [C] If you cannot solve this problem yourself, please go to the yum faq at: http://yum.baseurl.org/wiki/Faq
Because yum is broken on aix 7.3. See https://community.ibm.com/community/user/power/blogs/sangamesh-mallayya1/2021/05/28/dnf-is-now-available-on-aix-toolbox?CommunityKey=10c1d831-47ee-4d92-a138-b03f7896f7c9&tab=recentcommunityblogsdashboard
Download and run dnf_aixtoolbox.sh with -y.
A reduced set can be installed with aix 7.3,
yum install -y bash wget sudo tar gcc-gfortran gcc-c++ emacs blas freetype2-devel lapack libpng-devel python3-devel xz binutils-devel coreutils blas-devel openblas-devel libjpeg-devel openblas-devel ImageMagick-devel lua-devel tcl-devel tk-devel findutils gdb cmake
which unfortunately does not include vim (!).
LDAP
cd /root/ibmset uncompress 20151204_GSKit8_8_0_50_44.tar.Z tar -xf 20151204_GSKit8_8_0_50_44.tar cd 20151204_GSKit8_8_0_50_44
use smitty to install ALL FOUR packages, not just the 64 bit ones.
The following are the confirmed working installations on Cirrus:
bash-5.1# lslpp -L | grep -i gskit
GSKit8.gskcrypt32.ppc.rte
8.0.50.44 C F IBM GSKit Cryptography Runtime
GSKit8.gskcrypt64.ppc.rte
8.0.50.44 C F IBM GSKit Cryptography Runtime
GSKit8.gskssl32.ppc.rte 8.0.50.44 C F IBM GSKit SSL Runtime With
GSKit8.gskssl64.ppc.rte 8.0.50.44 C F IBM GSKit SSL Runtime With
gpfs.gskit 8.0.55.19 C F GPFS GSKit Cryptography
Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html [root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name "CA Signed" (enter 'Password' for password on key)
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb
End special instructions
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.
Next per https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client
we need to run idsLicense.
cd /root/ibmset/license tar -xf idslic.tar ./idsLicense
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see
bash-5.1# lslpp -L | grep -i idsl idsldap.clt32bit64.rte 6.4.0.23 C F Directory Server - 32 bit idsldap.clt64bit64.rte 6.4.0.23 C F Directory Server - 64 bit idsldap.clt_max_crypto32bit64.rte idsldap.clt_max_crypto64bit64.rte idsldap.cltbase64.adt 6.4.0.23 C F Directory Server - Base Client idsldap.cltbase64.rte 6.4.0.23 C F Directory Server - Base Client idsldap.license64.rte 6.4.0.23 C F Directory Server - License
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM
- lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k
should print
erik-k:
id=15382
pgrp=nic
groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin
home=/home/users/erik-k
shell=/bin/bash
SYSTEM=compat
registry=LDAP
and
- lsgroup -f nic
should print
nic:
id=3000
users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika
registry=LDAP
This confirms that LDAP authentication is working. Horray! \o/
chsec -f /etc/security/user -s default -a "SYSTEM=compat or LDAP"
because we have to have another step.
Sudo setup
visudo
Enter
User_Alias SUDO = erik-k,sameer,gansys,sivashan,nchaimov
for the user list and
SUDO ALL=(ALL) ALL
by root near the bottom
NFS
chnfsdom .stor startsrc -s nfsrgyd nfso -p -o nfs_use_reserved_ports=1
The file system mount creation commands:
mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp
At this point
startsrc -s nfs
should work
This will _massively_ simplify moving data back and forth!
It is also necessary for beegfs compat access,
mkdir -p /mnt/users/beegfs mknfsmnt -f /mnt/beegfs/users -d / -h bee2nfs.stor -M 'sys' -B -A -t rw -w bg -K 4 -k tcp
Spectrum Scale GPFS
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.
Create a directory and extract it (because it is a tarbomb!).
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys
Check that root@ems1 can passwordless ssh to host.stor...
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept
mmstartup -N HOSTNAME
mmgetstate -N HOSTNAME
Modules setup
The /packages directory is already mounted by the nfs setup step, but environment modules are not setup.
In /etc/profile:
export PATH=/opt/freeware/bin/:$PATH export PATH=/opt/IBM/openxlf/default/bin:$PATH export PATH=/opt/IBM/xlC/default/bin:$PATH source /usr/local/packages/modules-5.0.1/init/bash
This will avoid the need for everyone to edit their bashrc