When processor clock speeds flatlined in 2004, after more than fifteen years of exponential increases, the era of near automatic performance improvements that the HPC application community had previously enjoyed came to an abrupt end. To develop software that will perform well on petascale and exascale systems with thousands of nodes and millions of cores, the list of major challenges that must now be confronted is formidable:

1. dramatic escalation in the costs of intrasystem communication between processors and/or levels of memory hierarchy;
2. increased heterogeneity of the processing units (mixing CPUs, GPUs, etc. in varying and unexpected design combinations);
3. high levels of parallelism and more complex constraints means that cooperating processes must be dynamically and unpredictably scheduled for asynchronous execution;
4. software will not run at scale without much better resilience to faults and far more robustness; and
5. new levels of self-adaptivity will be required to enable software to modulate process speed in order to satisfy limited energy budgets.

The MORSE associate team will tackle the first three challenges in a orchestrating work between research groups respectively specialized in sparse linear algebra, dense linear algebra and runtime systems. The overall objective is to develop robust linear algebra libraries relying on innovative runtime systems that can fully benefit from the potential of those future large-scale complex machines. Challenges 4) and 5) will also be investigated by the different teams in the context of other partnerships, but they will not be the main focus of the associate team as they are much more prospective.

The overall goal of the MORSE associate team is to enable advanced numerical algorithms to be executed on a scalable unified runtime system for exploiting the full potential of future exascale machines. We expect advances in three directions based first on strong and closed interactions between the runtime and numerical linear algebra communities. This initial activity will then naturally expand to more focused but still joint research in both fields.

Research papers about MORSE can be found here.

The main purpose is to address the performance shortcomings of the LAPACK and ScaLAPACK libraries on multicore processors and multi-socket systems of multicore processors and their inability to efficiently utilize accelerators such as Graphics Processing Units (GPUs).

Chameleon is a framework written in C which provides routines to solve dense general systems of linear equations, symmetric positive definite systems of linear equations and linear least squares problems, using LU, Cholesky, QR and LQ factorizations. Real arithmetic and complex arithmetic are supported in both single precision and double precision. It supports Linux and Mac OS/X machines (mainly tested on Intel x86-64 and IBM Power architectures).

Chameleon is based on the PLASMA source code but is not limited to shared-memory environment and can exploit multiple GPUs. Chameleon is interfaced in a generic way with StarPU, PaRSEC, OpenMP, QUARK runtime systems. This feature allows to analyze in a unified framework how sequential task-based algorithms behave regarding different runtime systems implementations. Using Chameleon with StarPU runtime system allows to exploit GPUs through kernels provided by cuBLAS and clusters of interconnected nodes with distributed memory (using MPI). Computation of very large systems with dense matrices on a cluster of nodes is still being experimented and stabilized. It is not expected to get stable performances with the current version using MPI.

Chameleon is originally based on PLASMA so that design principles are very similar. The content of this section PLASMA's design principles has been copied from the Design principles section of the PLASMA User's Guide.

• we need the lapacke interface to tmg routines and symbol like LAPACKE_dlatms_work should be defined in the lapacke library. Make sure the Debian packages libopenblas-dev and liblapacke-dev (no problem with Intel MKL) do provide the tmg interface. If not you can possibly update your distribution or install the lapacke interface library in another way, by yourself from source or with Spack, or with Guix-HPC,…

You can optionally activate some options at cmake configure (like CUDA, MPI, …) invoking cmake path/to/your/CMakeLists.txt -DOPTION1= -DOPTION2= ...

cmake /home/jdoe/chameleon/ -DCMAKE_BUILD_TYPE=Debug \
			    -DCMAKE_INSTALL_PREFIX=/home/jdoe/install/ \
			    -DCHAMELEON_USE_CUDA=ON \
			    -DCHAMELEON_USE_MPI=ON \
			    -DBLA_VENDOR=Intel10_64lp_seq

You can get the full list of options with -L[A][H] options of cmake command

cmake -LH /home/jdoe/chameleon/

You can also set the options thanks to the ccmake interface.

You have different choices to detect dependencies on your system, either by setting some environment variables containing paths to the libs and headers or by specifying them directly at cmake configure. In any case, if the dependencies are installed in non standard directories, do not forget to use the PKG_CONFIG_PATH environment variable and the CMAKE_PREFIX_PATH environment (or CMake) variable. Different cases:

1. detection of dependencies through environment variables:

   • LD_LIBRARYPATH (DYLD_LIBRARYPATH on Mac OSX) should contain the list of paths where to find the libraries:

     export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:install/path/to/your/lib
     

   • INCLUDE (or CPATH, or C_INCLUDEPATH)should contain the list of paths where to find the header files of libraries

     export INCLUDE=$INCLUDE:install/path/to/your/headers
     

2. detection with user's given paths:

   • you can specify the path at cmake configure by invoking

     cmake path/to/your/CMakeLists.txt -DLIB_DIR=path/to/your/lib
     

     where LIB stands for the name of the lib to look for, e.g.

     cmake path/to/your/CMakeLists.txt -DQUARK_DIR=path/to/quarkdir \
     				  -DCBLAS_DIR= ...
     

     it is also possible to specify headers and library directories separately

     cmake path/to/your/CMakeLists.txt \
           -DQUARK_INCDIR=path/to/quark/include \
           -DQUARK_LIBDIR=path/to/quark/lib
     

3. detection with custom environment variables: all variables like _DIR, _INCDIR, _LIBDIR can be set as environment variables instead of CMake options, there will be read
4. using pkg-config for libraries that provide .pc files
   • update your PKG_CONFIGPATH to the paths where to find .pc files of installed external libraries like hwloc, starpu, some blas/lapack, etc
5. using CMAKE_PREFIXPATH for libraries that provide some CMake config files containing targets definitions (e.g. fooConfig.cmake).

Note that PaRSEC and StarPU are only detected with pkg-config mechanism because it is always provided and this avoids errors. The CMAKE_PREFIXPATH can be used to indicate where dependencies are installed.

Guix requires a running GNU/Linux system, GNU tar and Xz. Follow the installation instructions

cd /tmp
wget https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
chmod +x guix-install.sh
sudo ./guix-install.sh

or on Debian

sudo apt install guix

The Chameleon packages are not official Guix packages. It is then necessary to add a channel to get additional packages. Create a ~/.config/guix/channels.scm file with the following snippet:

(cons (channel
    (name 'guix-hpc-non-free)
    (url "https://gitlab.inria.fr/guix-hpc/guix-hpc-non-free.git"))
  %default-channels)

Update guix package definition

guix pull

Update new guix in the path

PATH="$HOME/.config/guix/current/bin${PATH:+:}$PATH"
hash guix

For further shell sessions, add this to the ~/.bash_profile file

export PATH="$HOME/.config/guix/current/bin${PATH:+:}$PATH"
export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale"

Chameleon packages are now available

guix search ^chameleon

Refer to the official documentation of Guix to learn the basic commands.

Standard Chameleon, last release

guix install chameleon
# or use guix shell to get a new shell (isolated from the native environment) with chameleon available in the PATH
guix shell --pure chameleon -- /bin/bash --norc

Notice that there exist several build variants

• chameleon (default) : with starpu - with mpi - with OpenBlas
• chameleon-mkl-mt : default version but with Intel MKL multithreaded to replace OpenBlas
• chameleon-mkl-mt-wompi : with Intel MKL multithreaded and without mpi
• chameleon-cuda : with starpu - with mpi - with cuda
• chameleon-cuda-wompi : with starpu - without mpi - with cuda
• chameleon-cuda-mkl-mt : with starpu - with mpi - with cuda - with Intel MKL multithreaded to replace OpenBlas
• chameleon-cuda-mkl-mt-wompi : with starpu - without mpi - with cuda - with Intel MKL multithreaded to replace OpenBlas
• chameleon-simgrid : with starpu - with mpi - with simgrid
• chameleon-openmp : with openmp - without mpi
• chameleon-parsec : with parsec - without mpi
• chameleon-quark : with quark - without mpi

Change the version

guix install chameleon --with-branch=chameleon=master
guix install chameleon --with-commit=chameleon=b31d7575fb7d9c0e1ba2d8ec633e16cb83778e8b
guix install chameleon --with-git-url=chameleon=https://gitlab.inria.fr/fpruvost/chameleon.git
guix install chameleon --with-source=chameleon=$HOME/git/chameleon

Notice also that default mpi is OpenMPI and default blas/lapack is Openblas. This can be changed with a transformation option.

Change some dependencies

# install chameleon with intel mkl to replace openblas, nmad to replace openmpi and starpu with fxt
guix install chameleon --with-input=openblas=mkl --with-input=openmpi=nmad --with-input=starpu=starpu-fxt

To install Chameleon and its dependencies within a docker image (OpenMPI stack)

docker_chameleon=`guix pack -f docker chameleon chameleon --with-branch=chameleon=master --with-input=openblas=mkl mkl starpu hwloc openmpi openssh slurm bash coreutils inetutils util-linux procps git grep tar sed gzip which gawk perl emacs-minimal vim gcc-toolchain make cmake pkg-config -S /bin=bin --entry-point=/bin/bash`
# Load the generated tarball as a docker image
docker_chameleon_tag=`docker load --input $docker_chameleon | grep "Loaded image: " | cut -d " " -f 3-`
# Change tag name, see the existing image name with "docker images" command, then change to a more simple name
docker tag $docker_chameleon_tag guix/chameleon-tmp

Create a Dockerfile inheriting from the image (renamed guix/chameleon here):

FROM guix/chameleon-tmp

# Create a directory for user 1000
RUN mkdir -p /builds
RUN chown -R 1000 /builds

ENTRYPOINT ["/bin/bash", "-l"]

# Enter the image as user 1000 in /builds
USER 1000
WORKDIR /builds
ENV HOME /builds

Then create the final docker image from this docker file.

docker build -t guix/chameleon .

Test the image

docker run -it guix/chameleon
# test starpu
STARPU=`pkg-config --variable=prefix libstarpu`
mpiexec -np 4 $STARPU/lib/starpu/mpi/comm
# test chameleon
CHAMELEON=`pkg-config --variable=prefix chameleon`
mpiexec -np 2 $CHAMELEON/bin/chameleon_stesting -H -o gemm -P 2 -t 2 -m 2000 -n 2000 -k 2000

To package Chameleon and its dependencies within a singularity image (OpenMPI stack)

# define reproducible guix environment
guix describe -f channels > guix-channels.scm
guix shell --export-manifest chameleon-cuda --with-branch=chameleon=master --with-input=openblas=mkl bash coreutils emacs gawk grep inetutils openmpi openssh procps sed time util-linux vim which > guix-manifests.scm
SINGULARITY_IMAGE=`guix time-machine -C guix-channels.scm -- pack -f squashfs -m guix-manifests.scm -S /bin=bin --entry-point=/bin/bash`
cp $SINGULARITY_IMAGE chameleon-cuda.gz.sif

# copy the singularity image on the supercomputer, e.g. 'supercomputer'
scp chameleon-cuda.gz.sif supercomputer:

On a machine where Singularity is installed Chameleon can then be called as follows

# at least openmpi and singularity are required here, e.g. module add openmpi singularity
module add openmpi singularity
export SINGULARITY_CMD=`which singularity`
export SINGULARITY_IMAGE=$HOME/chameleon-cuda.gz.sif
# use LD_PRELOAD to give the location of the CUDA driver installed on the supercomputer
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcuda.so
# then in your allocation with Slurm or OAR, for example
mpirun $MPI_OPTIONS -x LD_PRELOAD $SINGULARITY_CMD exec --bind /usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/ $SINGULARITY_IMAGE chameleon_stesting -o gemm -n 96000 -b 1600 --nowarmup -g 2

One can generate a tar.gz archive the same way as the singularity image

guix describe -f channels > guix-channels.scm
guix shell --export-manifest chameleon-cuda --with-branch=chameleon=master --with-input=openblas=mkl bash coreutils emacs gawk grep inetutils openmpi openssh procps sed time util-linux vim which > guix-manifests.scm
PACKRR=`guix time-machine --channels=guix-channels.scm -- pack -RR --manifest=guix-manifests.scm -S /bin=bin`
cp $PACKRR chameleon-cuda.tar.gz
# copy the archive on the supercomputer, e.g. 'supercomputer'
scp chameleon-cuda.tar.gz supercomputer:

Then on the supercomputer that has neither Guix nor Singularity one can do the following

mkdir guixrr/
cd guixrr/
tar xvf $HOME/chameleon-cuda.tar.gz
chmod +w .
export GUIX_ROOT=$PWD
# then in your allocation with Slurm or OAR, for example
${GUIX_ROOT}/bin/mpirun --launch-agent ${GUIX_ROOT}/bin/orted -x GUIX_EXECUTION_ENGINE=performance -x LD_PRELOAD="/usr/lib64/libcuda.so" -x STARPU_SILENT=1 ${GUIX_ROOT}/bin/chameleon_stesting -o gemm -n 16000,32000,64000,96000,128000 -b 2000 -g 4 -P 2

We provide a Chameleon Spack package (with StarPU) for Linux or macOS. Please refer to the documentation for installation instructions.

# please read https://spack.readthedocs.io/en/latest/getting_started.html
git clone https://github.com/spack/spack.git
. spack/share/spack/setup-env.sh
cd spack
git checkout v0.19.1

Chameleon is then available

spack info chameleon
spack spec chameleon

Refer to the getting started guide and basic usage guide to learn how to use Spack properly.

Standard Chameleon, last state on the 'master' branch

spack install -v chameleon
# chameleon is installed here:
spack location -i chameleon

Notice that there exist several build variants (see spack info chameleon)

• chameleon (default) : with starpu - with mpi
• tune the build type (CMake) with build_type=RelWithDebInfo|Debug|Release
• enable/disable shared libraries with +/- shared
• enable/disable mpi with +/- mpi
• enable/disable cuda with +/- cuda
• enable/disable fxt with +/- fxt
• enable/disable simgrid with +/- simgrid
• runtime=openmp : with openmp - without starpu

Change the version

spack install -v chameleon@master

Notice also that default mpi is OpenMPI and default blas/lapack is Openblas. This can be changed by adding some constraints on virtual packages.

Change some dependencies

# see lapack providers
spack providers lapack
# see mpi providers
spack providers mpi
# install chameleon with intel mkl to replace openblas
spack install -v chameleon ^intel-mkl # or ^intel-oneapi-mkl

A CHAMELEONConfig.cmake file is provided at installation, stored in <prefix>/lib/cmake/chameleon, so that users in cmake project can use through the variable CHAMELEON_ROOT (set it as environment or CMake variable).

sudo apt-get update
sudo apt-get install -y libopenblas-dev liblapacke-dev libstarpu-dev
git clone --recursive https://gitlab.inria.fr/solverstack/chameleon.git
cd chameleon && mkdir -p build && cd build
CHAMELEON_ROOT=$PWD/install
cmake .. -DCMAKE_INSTALL_PREFIX=$CHAMELEON_ROOT && make -j5 install
# chameleon is installed in $CHAMELEON_ROOT

# if your work in a cmake project you can use the CHAMELEONConfig.cmake file
# installed under <prefix>/lib/cmake/chameleon/ by setting your
# CMAKE_PREFIX_PATH with the path of installation. In your cmake project, use
# find_package(CHAMELEON) and link your libraries and/or executables with the
# library target CHAMELEON::chameleon
cmake . -DCMAKE_PREFIX_PATH=$CHAMELEON_ROOT

The compiler, linker flags that are necessary to build an application using Chameleon are given through the pkg-config mechanism.

sudo apt-get update
sudo apt-get install -y libopenblas-dev liblapacke-dev libstarpu-dev
git clone --recursive https://gitlab.inria.fr/solverstack/chameleon.git
cd chameleon && mkdir -p build && cd build
CHAMELEON_ROOT=$PWD/install
cmake .. -DCMAKE_INSTALL_PREFIX=$CHAMELEON_ROOT && make -j5 install
# chameleon is installed in $CHAMELEON_ROOT

export PKG_CONFIG_PATH=$CHAMELEON_ROOT/lib/pkgconfig:$PKG_CONFIG_PATH
pkg-config --cflags chameleon
pkg-config --libs chameleon
pkg-config --libs --static chameleon

# use it in your configure/make

The .pc files required are located in the sub-directory lib/pkgconfig of your Chameleon install directory.

Lets imagine you have a file main.c that you want to link with Chameleon static libraries. Lets consider /home/yourname/install/chameleon is the install directory of Chameleon containing sub-directories include/ and lib/. Here could be your compilation command with gcc compiler:

gcc -I/home/yourname/install/chameleon/include -o main.o -c main.c

Now if you want to link your application with Chameleon static libraries, you could do:

gcc main.o -o main                                         \
/home/yourname/install/chameleon/lib/libchameleon.a        \
/home/yourname/install/chameleon/lib/libchameleon_starpu.a \
/home/yourname/install/chameleon/lib/libcoreblas.a         \
-lstarpu-1.3 -Wl,--no-as-needed -lmkl_intel_lp64           \
-lmkl_sequential -lmkl_core -lpthread -lm -lrt

As you can see in this example, we also link with some dynamic libraries starpu-1.3, Intel MKL libraries (for BLAS/LAPACK/CBLAS/LAPACKE), pthread, m (math) and rt. These libraries will depend on the configuration of your Chameleon build. You can find these dependencies in .pc files we generate during compilation and that are installed in the sub-directory lib/pkgconfig of your Chameleon install directory. Note also that you could need to specify where to find these libraries with -L option of your compiler/linker.

Before to run your program, make sure that all shared libraries paths your executable depends on are known. Enter ldd main to check. If some shared libraries paths are missing append them in the LD_LIBRARYPATH (for Linux systems) environment variable (DYLD_LIBRARYPATH on Mac).

For dynamic linking (need to build Chameleon with CMake option BUILD_SHAREDLIBS=ON) it is similar to static compilation/link but instead of specifying path to your static libraries you indicate the path to dynamic libraries with -L option and you give the name of libraries with -l option like this:

gcc main.o -o main \
-L/home/yourname/install/chameleon/lib \
-lchameleon -lchameleon_starpu -lcoreblas \
-lstarpu-1.3 -Wl,--no-as-needed -lmkl_intel_lp64 \
-lmkl_sequential -lmkl_core -lpthread -lm -lrt

Note that an update of your environment variable LD_LIBRARYPATH (DYLD_LIBRARYPATH on Mac) with the path of the libraries could be required before executing

export LD_LIBRARY_PATH=path/to/libs:path/to/chameleon/lib

Some parameters of the Chameleon library can be set to some default values through environment variables which are listed below. Note that the code itself can modify these values through calls to `CHAMELEON<sub>Enable</sub>()`, `CHAMELEON<sub>Disable</sub>()`, or `CHAMELEON<sub>Set</sub>()` (see Options)

• CHAMELEON_TILESIZE defines the default tile size value for all algorithms. The default value is 384.
• CHAMELEON_{INNERBLOCKSIZE} defines the default inner blocking size value for algorithms that requires it (mainly QR/LQ algorithms). The default value is 48.
• CHAMELEON_{HOUSEHOLDERMODE} changes the basic QR algorithm from a flat tree (1, ChamFlatHouseholder or Flat) to an Householder tree (2, ChamTreeHouseholder, or Tree ). The default value is ChamFlatTree.
• CHAMELEON_{HOUSEHOLDERSIZE} defines the size of the local housholder trees if the Houselmoder tree mode is set. The default value is 4.
• CHAMELEON_{TRANSLATIONMODE} defines the translation used in the LAPACK API routines. 1, In, or ChamInPlace sets the in-place translation to avoid copies. 2, Out, ChamOutOfPlace sets the out-of-place translation that uses a copy of the matrix. The default is ChamInPlace.
• CHAMELEON_GENERIC, if ON all algorithms using specialized algorithms specific to data distributions are disabled.
• CHAMELEON_AUTOMINMAX, if ON the minimal/maximal limits of tasks that can be submitted to the runtime system are set. These limits are computed per algorithm using the lookahead parameter. (StarPU specific, and currently only available for getrf)
• CHAMELEON_LOOKAHEAD defines the number of steps that will be submitted in advance in algorithms using lookahead techniques. The default is 1.
• CHAMELEON_WARNINGS enables/disables the warning output
• CHAMELEON_{PARALLELKERNEL} enables/disables the use of multi-threaded kernels. Available only for StarPU runtime system.
• CHAMELEON_{GENERATESTATS} enables the profiling information of the kernels (StarPU specific)
• CHAMELEON_PROGRESS enables the progress function to show the percentage of tasks completed.

EZTrace can be used by chameleon to generate traces. Two modules are automatically generated as soon as EZTrace is detected on the system. The first one (which is recommended) is the chameleon_tcore module. It traces all the TCORE_...() functions that are called by the codelets of all the runtime but PaRSEC. The second one is the chameleon_core module which traces the lower level CORE_...() functions. If using PaRSEC, you need to use this module to generate the traces.

To generate traces with EZTrace, you need first to compile with -DBUILD_SHAREDLIBS=ON. EZTrace is using weak symbols to overload function calls with ld_preload and enable trace generation. Then, either you install the libeztrace-*.so files into the EZTrace install directory, or you can add the path of the modules to your environement

export EZTRACE_LIBRARY_PATH=/path/to/your/modules

To check if the modules are available you should have

$ eztrace_avail
1	omp	Module for OpenMP parallel regions
2	pthread	Module for PThread synchronization functions (mutex, semaphore, spinlock, etc.)
3	stdio	Module for stdio functions (read, write, select, poll, etc.)
4	mpi	Module for MPI functions
5	memory	Module for memory functions (malloc, free, etc.)
6	papi	Module for PAPI Performance counters
128	chameleon_core	Module for Chameleon CORE functions
129	chameleon_tcore	Module for Chameleon TCORE functions

Then, you can restrict the modules used during the execution

export EZTRACE_TRACE="mpi chameleon_tcore"

The module mpi is required if you want to run in distributed.

The setup can be checked with eztrace_loaded

$ eztrace_loaded
4	mpi	Module for MPI functions
129	chameleon_tcore	Module for Chameleon TCORE functions

To generate the traces, you need to run your binary through eztrace:

eztrace ./chameleon_dtesting -o gemm -n 1000 -b 200
mpirun -np 4 eztrace ./chameleon_dtesting -o gemm -n 1000 -b 200 -P 2

Convert the binary files into a .trace file, and visualize it.

eztrace_convert <username>_eztrace_log_rank_<[0-9]*>
vite eztrace_output.trace

For more information on EZTrace, you can follow the support page.

StarPU can generate its own trace log files by compiling it with the --with-fxt option at the configure step (you can have to specify the directory where you installed FxT by giving --with-fxt=... instead of --with-fxt alone). In addition, the environment variable STARPU_FXTTRACE must be set to 1.

export STARPU_FXT_TRACE=1

By doing so, traces are generated after each execution of a program which uses StarPU in the directory pointed by the STARPU_FXTPREFIX environment variable (if not set the default path is tmp).

export STARPU_FXT_PREFIX=/home/jdoe/fxt_files/

When executing a ./testing/... Chameleon program, if it has been enabled (StarPU compiled with FxT), the program will generate trace files in the directory $STARPU_FXTPREFIX.

To save only some specific types of events the variable STARPU_FXTEVENTS.

Finally, to generate the trace file which can be opened with Vite program, you can use the starpu_fxttool executable of StarPU. This tool should be in the bin directory of StarPU's installation. You can use it to generate the trace file like this:

path/to/your/install/starpu/bin/starpu_fxt_tool -i prof_filename

There is one file per mpi processus (prof_filename0, prof_filename1 …). To generate a trace of mpi programs you can call it like this:

path/to/your/install/starpu/bin/starpu_fxt_tool -i prof_filename*

The trace file will be named paje.trace (use -o option to specify an output name). Alternatively, for non mpi execution (only one processus and profiling file), you can set the environment variable STARPU_{GENERATETRACE}=1 to automatically generate the paje trace file.

Simulation mode can be activated by setting the cmake option CHAMELEON_SIMULATION to ON. This mode allows you to simulate execution of algorithms with StarPU compiled with SimGrid. To do so, we provide some perfmodels in the simucore/perfmodels/ directory of Chameleon sources. To use these perfmodels, please set your STARPU_HOME environment variable to path/to/your/chameleon_sources/simucore/perfmodels. Finally, you need to set your STARPU_HOSTNAME environment variable to the name of the machine to simulate.

The algorithms available for now: gemm, symm, potrf, potrs, potri, posv, getrf_nopiv, getrs_nopiv, geqrf, geqrf_hqr, gels, gels_hqr, simple and double precisions on PlaFRIM nodes with GPUs. The tile size to use depending on the platform i.e. STARPU_HOSTNAME (choose a size N multiple of the tile size):

• sirocco-k40m: 960
• sirocco-p100: 1240
• sirocco-v100: 1600
• sirocco-a100: 1600
• sirocco-rtx8000: 1600

In addition the potrf algorithm is also available on mirage and sirocco machines for the following tile sizes

• mirage: 320, 960
• sirocco: 80, 440, 960, 1440, 1920

Database of models is subject to change.

export STARPU_HOME=/tmp/chameleon/simucore/perfmodels/
export STARPU_HOSTNAME=sirocco
./testing/chameleon_dtesting -o potrf -t 22 -g 2 -n 14400 -b 1440 --nowarmup
0;dpotrf;22;2;1;1;0;1440;121;14400;14400;1804289383;0.000000e+00;7.867404e-01;1.265261e+03

export STARPU_HOSTNAME=sirocco-k40m
./testing/chameleon_stesting -o gemm -t 38 -g 2 -n 64000 -b 1600 --nowarmup
0;sgemm;38;2;1;1;0;1600;111;111;64000;64000;64000;64000;64000;64000;4.892778e-01;-1.846424e-01;1649760492;596516649;1189641421;0.000000e+00;2.010660e+01;2.607541e+04

export STARPU_HOSTNAME=sirocco-p100
./testing/chameleon_dtesting -o geqrf -g 2 -t 30 -b 1240 -n 39680 --nowarmup
0;dgeqrf;30;2;1;1;0;1240;48;39680;39680;39680;4;1804289383;0.000000e+00;3.893336e+01;2.139677e+03

If the matrix can not fit in the main memory, StarPU can automatically evict tiles to the disk. The following variables need to be set:

• STARPU_DISKSWAP environment variable to a place where to store

evicted tiles, for example: STARPU_DISK_SWAP=/tmp

• STARPU_{DISKSWAPBACKEND} environment variable to the I/O method,

for example: STARPU_DISK_SWAP_BACKEND=unistd_o_direct

• STARPU_LIMITCPUMEM environment variable to the amount of memory

that can be used in MBytes, for example: STARPU_LIMIT_CPU_MEM=1000

The C interface of BLAS and LAPACK, that is, CBLAS and LAPACKE, are used to solve the system. The size of the system (matrix) and the number of right hand-sides can be given as arguments to the executable (be careful not to give huge numbers if you do not have an infinite amount of RAM!). As for every step, the correctness of the solution is checked by calculating the norm \(||Ax-B||/(||A||||x||+||B||)\). The time spent in factorization+solve is recorded and, because we know exactly the number of operations of these algorithms, we deduce the number of operations that have been processed per second (in GFlops/s). The important part of the code that solves the problem is:

/* Cholesky factorization:
 * A is replaced by its factorization L or L^T depending on uplo */
LAPACKE_dpotrf( LAPACK_COL_MAJOR, 'U', N, A, N );
/* Solve:
 * B is stored in X on entry, X contains the result on exit.
 * Forward ...
 */
cblas_dtrsm(
    CblasColMajor,
    CblasLeft,
    CblasUpper,
    CblasConjTrans,
    CblasNonUnit,
    N, NRHS, 1.0, A, N, X, N);
/* ... and back substitution */
cblas_dtrsm(
    CblasColMajor,
    CblasLeft,
    CblasUpper,
    CblasNoTrans,
    CblasNonUnit,
    N, NRHS, 1.0, A, N, X, N);

It introduces the simplest Chameleon interface which is equivalent to CBLAS/LAPACKE. The code is very similar to step0 but instead of calling CBLAS/LAPACKE functions, we call Chameleon equivalent functions. The solving code becomes:

/* Factorization: */
CHAMELEON_dpotrf( UPLO, N, A, N );
/* Solve: */
CHAMELEON_dpotrs(UPLO, N, NRHS, A, N, X, N);

The API is almost the same so that it is easy to use for beginners. It is important to keep in mind that before any call to CHAMELEON routines, CHAMELEON_Init has to be invoked to initialize CHAMELEON and the runtime system. Example:

CHAMELEON_Init( NCPU, NGPU );

After all CHAMELEON calls have been done, a call to CHAMELEON_Finalize is required to free some data and finalize the runtime and/or MPI.

CHAMELEON_Finalize();

We use CHAMELEON routines with the LAPACK interface which means the routines accepts the same matrix format as LAPACK (1-D array column-major). Note that we copy the matrix to get it in our own tile structures, see details about this format here Tile Data Layout. This means you can get an overhead coming from copies.

This program is a copy of step1 but instead of using the LAPACK interface which reads to copy LAPACK matrices inside CHAMELEON routines we use the tile interface. We will still use standard format of matrix but we will see how to give this matrix to create a CHAMELEON descriptor, a structure wrapping data on which we want to apply sequential task-based algorithms. The solving code becomes:

/* Factorization: */
CHAMELEON_dpotrf_Tile( UPLO, descA );
/* Solve: */
CHAMELEON_dpotrs_Tile( UPLO, descA, descX );

To use the tile interface, a specific structure CHAM_desct must be created. This can be achieved from different ways.

1. Use the existing function CHAMELEON_DescCreate: means the matrix data are considered contiguous in memory as it is considered in PLASMA (Tile Data Layout).
2. Use the existing function CHAMELEON_{DescCreateOOC}: means the matrix data is allocated on-demand in memory tile by tile, and possibly pushed to disk if that does not fit memory.
3. Use the existing function CHAMELEON_{DescCreateUser}: it is more flexible than Desc_Create because you can give your own way to access to tile data so that your tiles can be allocated wherever you want in memory, see next paragraph Step3.
4. Create you own function to fill the descriptor. If you understand well the meaning of each item of CHAM_desct, you should be able to fill correctly the structure.

In Step2, we use the first way to create the descriptor:

CHAMELEON_Desc_Create(&descA, NULL, ChamRealDouble,
                  NB, NB, NB*NB, N, N,
                  0, 0, N, N,
                  1, 1);

• descA is the descriptor to create.

• The second argument is a pointer to existing data. The existing data must follow LAPACK/PLASMA matrix layout Tile Data Layout (1-D array column-major) if CHAMELEON_DescCreate is used to create the descriptor. The CHAMELEON_{DescCreateUser} function can be used if you have data organized differently. This is discussed in the next paragraph Step3. Giving a NULL pointer means you let the function allocate memory space. This requires to copy your data in the memory allocated by the *Desc_Create. This can be done with

  CHAMELEON_Lapack_to_Tile(A, N, descA);
  

• Third argument of @code{Desc_Create} is the datatype (used for memory allocation).
• Fourth argument until sixth argument stand for respectively, the number of rows (NB), columns (NB) in each tile, the total number of values in a tile (NB*NB), the number of rows (N), colmumns (N) in the entire matrix.
• Seventh argument until ninth argument stand for respectively, the beginning row (0), column (0) indexes of the submatrix and the number of rows (N), columns (N) in the submatrix. These arguments are specific and used in precise cases. If you do not consider submatrices, just use 0, 0, NROWS, NCOLS.
• Two last arguments are the parameter of the 2-D block-cyclic distribution grid, see ScaLAPACK. To be able to use other data distribution over the nodes, CHAMELEON_{DescCreateUser} function should be used.

This program makes use of the same interface than Step2 (tile interface) but does not allocate LAPACK matrices anymore so that no copy between LAPACK matrix layout and tile matrix layout are necessary to call CHAMELEON routines. To generate random right hand-sides you can use:

/* Allocate memory and initialize descriptor B */
CHAMELEON_Desc_Create(&descB,  NULL, ChamRealDouble,
                  NB, NB,  NB*NB, N, NRHS,
                  0, 0, N, NRHS, 1, 1);
/* generate RHS with random values */
CHAMELEON_dplrnt_Tile( descB, 5673 );

The other important point is that is it possible to create a descriptor, the necessary structure to call CHAMELEON efficiently, by giving your own pointer to tiles if your matrix is not organized as a 1-D array column-major. This can be achieved with the CHAMELEON_{DescCreateUser} routine. Here is an example:

CHAMELEON_Desc_Create_User(&descA, matA, ChamRealDouble,
                       NB, NB, NB*NB, N, N,
                       0, 0, N, N, 1, 1,
                       user_getaddr_arrayofpointers,
                       user_getblkldd_arrayofpointers,
                       user_getrankof_zero, NULL);

Firsts arguments are the same than CHAMELEON_DescCreate routine. Following arguments allows you to give pointer to functions that manage the access to tiles from the structure given as second argument. Here for example, matA is an array containing addresses to tiles, see the function allocate_tilematrix defined in step3.h. If you want the matrix to be allocate by Chameleon, you can use the CHAMELEON_{MATALLOCGLOBAL}, or the CHAMELEON_MATALLOCTILE variables to allocate repectively as a single large allocation, or to allocate tile by tile as late as possible. The three functions you have to define for Desc_CreateUser are:

• a function that returns address of tile \(A(m,n)\), m and n standing for the indexes of the tile in the global matrix. Lets consider a matrix @math{4x4} with tile size 2x2, the matrix contains four tiles of indexes: \(A(m=0,n=0)\), \(A(m=0,n=1)\), \(A(m=1,n=0)\), \(A(m=1,n=1)\)
• a function that returns the leading dimension of tile \(A(m,*)\)
• a function that returns MPI rank of tile \(A(m,n)\)
• a pointer to a structure that these three functions can use to store additional data.

Examples for these functions are visible in step3.h. Note that the way we define these functions is related to the tile matrix format and to the data distribution considered. This example should not be used with MPI since all tiles are affected to processus 0, which means a large amount of data will be potentially transfered between nodes.

This program is a copy of step2 but instead of using the tile interface, it uses the tile async interface. The goal is to exhibit the runtime synchronization barriers. Keep in mind that when the tile interface is called, like CHAMELEON_dpotrfTile, a synchronization function, waiting for the actual execution and termination of all tasks, is called to ensure the proper completion of the algorithm (i.e. data are up-to-date). The code shows how to exploit the async interface to pipeline subsequent algorithms so that less synchronisations are done. The code becomes:

/* Cham structure containing parameters and a structure to interact with
 * the Runtime system */
CHAM_context_t *chamctxt;
/* CHAMELEON sequence uniquely identifies a set of asynchronous function calls
 * sharing common exception handling */
RUNTIME_sequence_t *sequence = NULL;
/* CHAMELEON request uniquely identifies each asynchronous function call */
RUNTIME_request_t request = CHAMELEON_REQUEST_INITIALIZER;
int status;

...

chameleon_sequence_create(chamctxt, &sequence);

/* Factorization: */
CHAMELEON_dpotrf_Tile_Async( UPLO, descA, sequence, &request );

/* Solve: */
CHAMELEON_dpotrs_Tile_Async( UPLO, descA, descX, sequence, &request);

/* Synchronization barrier (the runtime ensures that all submitted tasks
 * have been terminated */
RUNTIME_barrier(chamctxt);
/* Ensure that all data processed on the gpus we are depending on are back
 * in main memory */
RUNTIME_desc_getoncpu(descA);
RUNTIME_desc_getoncpu(descX);

status = sequence->status;

Here the sequence of dpotrf and dpotrs algorithms is processed without synchronization so that some tasks of dpotrf and dpotrs can be concurently executed which could increase performances. The async interface is very similar to the tile one. It is only necessary to give two new objects RUNTIME_sequencet and RUNTIME_requestt used to handle asynchronous function calls.

Step5 shows how to set some important parameters. This program is a copy of Step4 but some additional parameters are given by the user. The parameters that can be set are:

• number of Threads

• number of GPUs

  The number of workers can be given as argument to the executable with --threads= and --gpus= options. It is important to notice that we assign one thread per gpu to optimize data transfer between main memory and devices memory. The number of workers of each type CPU and CUDA must be given at CHAMELEON_Init.

  if ( iparam[IPARAM_THRDNBR] == -1 ) {
      get_thread_count( &(iparam[IPARAM_THRDNBR]) );
      /* reserve one thread par cuda device to optimize memory transfers */
      iparam[IPARAM_THRDNBR] -=iparam[IPARAM_NCUDAS];
  }
  NCPU = iparam[IPARAM_THRDNBR];
  NGPU = iparam[IPARAM_NCUDAS];
  /* initialize CHAMELEON with main parameters */
  CHAMELEON_Init( NCPU, NGPU );
  

• matrix size
• number of right-hand sides

• block (tile) size

  The problem size is given with --n= and --nrhs= options. The tile size is given with option --nb=. These parameters are required to create descriptors. The size tile NB is a key parameter to get performances since it defines the granularity of tasks. If NB is too large compared to N, there are few tasks to schedule. If the number of workers is large this leads to limit parallelism. On the contrary, if NB is too small (i.e. many small tasks), workers could not be correctly fed and the runtime systems operations could represent a substantial overhead. A trade-off has to be found depending on many parameters: problem size, algorithm (drive data dependencies), architecture (number of workers, workers speed, workers uniformity, memory bus speed). By default it is set to 128. Do not hesitate to play with this parameter and compare performances on your machine.

• inner-blocking size

  The inner-blocking size is given with option --ib=. This parameter is used by kernels (optimized algorithms applied on tiles) to perform subsequent operations with data block-size that fits the cache of workers. Parameters NB and IB can be given with CHAMELEON_Set function:

  CHAMELEON_Set(CHAMELEON_TILE_SIZE,        iparam[IPARAM_NB] );
  CHAMELEON_Set(CHAMELEON_INNER_BLOCK_SIZE, iparam[IPARAM_IB] );
  

This program is a copy of Step5 with some additional parameters to be set for the data distribution. To use this program properly CHAMELEON must use StarPU Runtime system and MPI option must be activated at configure. The data distribution used here is 2-D block-cyclic, see for example ScaLAPACK for explanation. The user can enter the parameters of the distribution grid at execution with --p= option. Example using OpenMPI on four nodes with one process per node:

mpirun -np 4 ./step6 --n=10000 --nb=320 --ib=64 --threads=8 --gpus=2 --p=2

In this program we use the tile data layout from PLASMA so that the call

CHAMELEON_Desc_Create(&descA, NULL, ChamRealDouble,
                      NB, NB, NB*NB, N, N,
                      0, 0, N, N,
                      GRID_P, GRID_Q);

is equivalent to the following call

CHAMELEON_Desc_Create_User(&descA, NULL, ChamRealDouble,
                           NB, NB, NB*NB, N, N,
                           0, 0, N, N,
                           GRID_P, GRID_Q,
                           chameleon_getaddr_ccrb,
                           chameleon_getblkldd_ccrb,
                           chameleon_getrankof_2d, NULL);

functions chameleon_getaddrccrb, chameleon_{getblklddccrb}, chameleon_getrankof2d being used in Desc_Create. It is interesting to notice that the code is almost the same as Step5. The only additional information to give is the way tiles are distributed through the third function given to CHAMELEON_{DescCreateUser}. Here, because we have made experiments only with a 2-D block-cyclic distribution, we have parameters P and Q in the interface of Desc_Create but they have sense only for 2-D block-cyclic distribution and then using chameleon_getrankof2d function. Of course it could be used with other distributions, being no more the parameters of a 2-D block-cyclic grid but of another distribution. And the last parameter void* get_rankof_arg of CHAMELEON_{DescCreateUser} can be used to get custom data in the get_rankof function.

This program is a copy of step6 with some additional calls to build a matrix from within chameleon using a function provided by the user. This can be seen as a replacement of the function like CHAMELEON_dplgsyTile() that can be used to fill the matrix with random data, CHAMELEON_{dLapacktoTile}() to fill the matrix with data stored in a lapack-like buffer, or CHAMELEON_{DescCreateUser}() that can be used to describe an arbitrary tile matrix structure. In this example, the build callback function are just wrapper towards CORE_xxx() functions, so the output of the program step7 should be exactly similar to that of step6. The difference is that the function used to fill the tiles is provided by the user, and therefore this approach is much more flexible.

The new function to understand is CHAMELEON_mapTile, e.g.

struct data_pl data_A={(double)N, 51};
CHAMELEON_map_Tile(ChamW, ChamUpperLower, descA, Cham_map_plgsy, (void*)&data_A);

The idea here is to let Chameleon fill the matrix data in a task-based fashion (parallel) by using a function given by the user. First, the user has to give the access mode to the matrix between: ChamR, *ChamW, ChamRW depending on the kind of operations the callback function needs to do on the tiles. In our example here we fill the matrix with random values for the first time so that we use the access mode ChamW. Second, the user should define if all the blocks must be entirelly filled or just the upper/lower part with, e.g. ChamUpperLower. We still relies on the same structure CHAM_desct which must be initialized with the proper parameters, by calling for example CHAMELEON_DescCreate. Then comes the pointer to the user's function. And finally the last parameter is an opaque pointer is used to let the user give some extra data used by his function.

It is possible to provide custom data distributions to Chameleon, to go beyond the 2D block cyclic distributions. A generic interface is provided with the functions chameleon_{getrankofcustominit}, chameleon_{getrankofcustomdestroy} and chameleon_{getrankofcustom}, with the following signatures:

int chameleon_getrankof_custom_init( custom_dist_t **custom_dist,
				     const char     *dist_file );

int chameleon_getrankof_custom_destroy( custom_dist_t **dist );

int chameleon_getrankof_custom( const CHAM_desc_t *desc, int m, int n );

The first function is used to read a custom distribution from an external file, whose name is provided in the dist_file argument. The file format is described below. The second function can be used to destroy the custom_distt pointer when it is no longer useful. The last function should be used as the get_rankof argument to CHAMELEON_{DescCreateUser}, together with the custom distribution obtained from chameleon_{getrankofcustominit}. The typical usage is the following:

custom_dist_t* custom_dist;
chameleon_getrankof_custom_init( &custom_dist, "filename" );
CHAMELEON_Desc_Create_User(&descA, NULL, ChamRealDouble,
			   NB, NB, NB*NB, N, N,
			   0, 0, N, N,
			   CHAMELEON_Comm_size(), 1,
			   chameleon_getaddr_ccrb,
			   chameleon_getblkldd_ccrb,
			   chameleon_getrankof_custom, custom_dist);
/* Use the descriptor */
CHAMELEON_Desc_Destroy(&descA);
chameleon_getrankof_custom_destroy(&custom_dist);

Since we do not use a 2D block-cyclic distribution, the values of P and Q have no importance in CHAMELEON_{DescCreateUser}. However, make sure that the product of P and Q is equal to the number of processes by using the couple (CHAMELEON_Commsize(), 1) as a replacement for (P, Q).

The custom distribution is provided by a pattern that can have any dimension, and which is repeated all over the matrix. The file format expected by chameleon_{getrankofcustominit} is a simple text format, with space-separated integer values. The first two values represent the size of the pattern (number of rows \(m_d\) and number of columns \(n_d\)). Then, the function expects \(m_d * n_d\) values, where each value is the index of the process that should handle this tile. For example, the following file content would result in a 2D block-cyclic distribution with P=2 and Q=3 (it is not necessary to skip lines, but it can make the file more readable):

2 3
0 1 2
3 4 5

We list the linear algebra routines of the form CHAMELEON_name[_Tile[_Async]] (name follows LAPACK naming scheme, see http://www.netlib.org/lapack/lug/node24.html) that can be used with the Chameleon library. For details about these functions please refer to the doxygen documentation. name can be one of the following:

• BLAS 2/3 routines
  • gemm: matrix matrix multiply and addition
  • hemm: gemm with A Hermitian
  • herk: rank k operations with A Hermitian
  • her2k: rank 2k operations with A Hermitian
  • lauum: computes the product U * U' or L' * L, where the triangular factor U or L is stored in the upper or lower triangular part of the array A
  • symm: gemm with A symmetric
  • syrk: rank k operations with A symmetric
  • syr2k: rank 2k with A symmetric
  • trmm: gemm with A triangular
• Triangular solving routines
  • trsm: computes triangular solve
  • trsmpl: performs the forward substitution step of solving a system of linear equations after the tile LU factorization of the matrix
  • trsmrv:
  • trtri: computes the inverse of a complex upper or lower triangular matrix A
• LL' (Cholesky) routines
  • posv: linear systems solving using Cholesky factorization
  • potrf: Cholesky factorization
  • potri: computes the inverse of a complex Hermitian positive definite matrix A using the Cholesky factorization A
  • potrimm:
  • potrs: linear systems solving using existing Cholesky factorization
  • sysv: linear systems solving using Cholesky decomposition with A symmetric
  • sytrf: Cholesky decomposition with A symmetric
  • sytrs: linear systems solving using existing Cholesky decomposition with A symmetric
• LU routines
  • gesv_incpiv: linear systems solving with LU factorization and partial pivoting
  • gesv_nopiv: linear systems solving with LU factorization and without pivoting
  • getrf_incpiv: LU factorization with partial pivoting
  • getrf_nopiv: LU factorization without pivoting
  • getrs_incpiv: linear systems solving using existing LU factorization with partial pivoting
  • getrs_nopiv: linear systems solving using existing LU factorization without pivoting
• QR/LQ routines
  • gelqf: LQ factorization
  • gelqf_param: gelqf with hqr
  • gelqs: computes a minimum-norm solution min || A*X - B || using the LQ factorization
  • gelqs_param: gelqs with hqr
  • gels: Uses QR or LQ factorization to solve a overdetermined or underdetermined linear system with full rank matrix
  • gels_param: gels with hqr
  • geqrf: QR factorization
  • geqrf_param: geqrf with hqr
  • geqrs: computes a minimum-norm solution min || A*X - B || using the RQ factorization
  • hetrd: reduces a complex Hermitian matrix A to real symmetric tridiagonal form S
  • geqrs_param: geqrs with hqr
  • tpgqrt: generates a partial Q matrix formed with a blocked QR factorization of a "triangular-pentagonal" matrix C, which is composed of a unused triangular block and a pentagonal block V, using the compact representation for Q. See tpqrt to generate V
  • tpqrt: computes a blocked QR factorization of a "triangular-pentagonal" matrix C, which is composed of a triangular block A and a pentagonal block B, using the compact representation for Q
  • unglq: generates an M-by-N matrix Q with orthonormal rows, which is defined as the first M rows of a product of the elementary reflectors returned by CHAMELEON_zgelqf
  • unglq_param: unglq with hqr
  • ungqr: generates an M-by-N matrix Q with orthonormal columns, which is defined as the first N columns of a product of the elementary reflectors returned by CHAMELEON_zgeqrf
  • ungqr_param: ungqr with hqr
  • unmlq: overwrites C with Q*C or C*Q or equivalent operations with transposition on conjugate on C (see doxygen documentation)
  • unmlq_param: unmlq with hqr
  • unmqr: similar to unmlq (see doxygen documentation)
  • unmqr_param: unmqr with hqr
• EVD/SVD
  • gesvd: singular value decomposition
  • heevd: eigenvalues/eigenvectors computation with A Hermitian

• Specific Matrix transformation for Data Analysis

  • cesca: centered-scaled matrix transformation, pretreatment algorithm for Principal Component Analysis
  • gram: Gram matrix transformation, pretreatment algorithm for Multidimensional Scaling
  • Extra routines
    • Norms
      • lange: compute norm of a matrix (Max, One, Inf, Frobenius)
      • lanhe: lange with A Hermitian
      • lansy: lange with A symmetric
      • lantr: lange with A triangular
    • Random matrices generation
      • plghe: generate a random Hermitian matrix
      • plgsy: generate a random symmetrix matrix
      • plgtr: generate a random trapezoidal matrix
      • plrnt: generate a random matrix
      • plrnk: generate a random matrix of rank K with K <= min(M,N)
    • Others
      • geadd: general matrix matrix addition
      • lacpy: copy matrix into another
      • lascal: scale a matrix
      • laset: copy the triangular part of a matrix into another, set a value for the diagonal and off-diagonal part
      • tradd: trapezoidal matrices addition
    • Map functions
      • map: apply a user operator on each tile of the matrix

  In addition, all BLAS 3 routines gemm, hemm, her2k, herk, lauum, symm, syr2k, syrk, trmm, trsm and LAPACK lacpy, lange, lanhe, lansy, lantr, laset, posv, potrf, potri, potrs, trtri can be called using an equivalent of the (C)BLAS/LAPACK(E) API. The parameters are the same and the user just has to add CHAMELEON_ to the standard name of the routine. For example, in C

  CHAMELEON_Init(4,0);
  CHAMELEON_cblas_dgemm(CblasColMajor, CblasNoTrans, CblasNoTrans,
  		      N, NRHS, N, 1.0, A, N, X, N, -1.0, B, N);
  CHAMELEON_Finalize();
  

  In Fortran, the function names are for example: CHAMELEON_blas_dgemm instead of DGEMM and CHAMELEON_lapack_dposv instead of DPOSV.

Enable CHAMELEON feature.

int CHAMELEON_Enable  (CHAMELEON_enum option);

Features that can be enabled/disabled:

• CHAMELEON_WARNINGS: printing of warning messages,
• CHAMELEON_AUTOTUNING: autotuning for tile size and inner block size (inactive),
• CHAMELEON_{GENERATETRACE}: enable/start the trace generation
• CHAMELEON_{GENERATESTATS}: enable/start the kernel statistics
• CHAMELEON_PROGRESS: to print a progress status,
• CHAMELEON_GEMM3M: to enable the use of the gemm3m BLAS function.

Disable CHAMELEON feature.

int CHAMELEON_Disable (CHAMELEON_enum option);

Symmetric to CHAMELEON_Enable.

Set CHAMELEON parameter.

int CHAMELEON_Set     (CHAMELEON_enum param, int  value);

Parameters to be set:

• CHAMELEON_TILESIZE: size matrix tile,
• CHAMELEON_{INNERBLOCKSIZE}: size of tile inner block,
• CHAMELEON_{HOUSEHOLDERMODE}: type of householder trees (FLAT or TREE),
• CHAMELEON_{HOUSEHOLDERSIZE}: size of the groups in householder trees,
• CHAMELEON_{TRANSLATIONMODE}: related to the CHAMELEON_LapacktoTile, see ztile.c.

Get value of CHAMELEON parameter.

int CHAMELEON_Get     (CHAMELEON_enum param, int *value);

• Alternatively, Chameleon can also be configured through environment variables.
  • CHAMELEON_GEMMALGO give the possibility to switch among multiple variants of the GEMM algorithms. These variants are GENERIC for the generic variant that should work with any configuration; SUMMA_C that works for 2D block cyclic distribution of the matrices A, B, and C with a C stationnary version; SUMMA_A and SUMMA_B are SUMMA variant of the algorithm that works for any distribution with respectively A, or *B that are stationnary. Note that the last two variants are only available with the StarPU runtime backend.

Reports CHAMELEON version number.

int CHAMELEON_Version        (int *ver_major, int *ver_minor, int *ver_micro);

Initialize CHAMELEON: initialize some parameters, initialize the runtime and/or MPI.

int CHAMELEON_Init           (int nworkers, int ncudas);

Finalyze CHAMELEON: free some data and finalize the runtime and/or MPI.

int CHAMELEON_Finalize       (void);

Suspend CHAMELEON runtime to poll for new tasks, to avoid useless CPU consumption when no tasks have to be executed by CHAMELEON runtime system.

int CHAMELEON_Pause          (void);

Symmetrical call to CHAMELEON_Pause, used to resume the workers polling for new tasks.

int CHAMELEON_Resume         (void);

Return the MPI rank of the calling process.

int CHAMELEON_My_Mpi_Rank    (void);

Return the size of the distributed computation

int CHAMELEON_Comm_size( int *size )

Return the rank of the distributed computation

int CHAMELEON_Comm_rank( int *rank )

Prepare the distributed processes for computation

int CHAMELEON_Distributed_start(void)

Clean the distributed processes after computation

int CHAMELEON_Distributed_stop(void)

Return the number of CPU workers initialized by the runtime

int CHAMELEON_GetThreadNbr()

Conversion from LAPACK layout to tile layout.

int CHAMELEON_Lapack_to_Tile (void *Af77, int LDA, CHAM_desc_t *A);

Conversion from tile layout to LAPACK layout.

int CHAMELEON_Tile_to_Lapack (CHAM_desc_t *A, void *Af77, int LDA);

Create matrix descriptor, internal function.

int CHAMELEON_Desc_Create(CHAM_desc_t **desc, void *mat, cham_flttype_t dtyp,
		      int mb, int nb, int bsiz, int lm, int ln,
		      int i, int j, int m, int n, int p, int q);

Create matrix descriptor, user function.

int CHAMELEON_Desc_Create_User(CHAM_desc_t **desc, void *mat, cham_flttype_t dtyp,
			   int mb, int nb, int bsiz, int lm, int ln,
			   int i, int j, int m, int n, int p, int q,
			   void* (*get_blkaddr)( const CHAM_desc_t*, int, int),
			   int (*get_blkldd)( const CHAM_desc_t*, int ),
			   int (*get_rankof)( const CHAM_desc_t*, int, int ),
			   void* get_rankof_arg);

Create matrix descriptor for tiled matrix which may not fit memory.

int CHAMELEON_Desc_Create_OOC(CHAM_desc_t **descptr, cham_flttype_t dtyp, int mb, int nb, int bsiz,
			  int lm, int ln, int i, int j, int m, int n, int p, int q);

User's function version of CHAMELEON_{DescCreateOOC}.

int CHAMELEON_Desc_Create_OOC_User(CHAM_desc_t **descptr, cham_flttype_t dtyp, int mb, int nb, int bsiz,
			       int lm, int ln, int i, int j, int m, int n, int p, int q,
			       int (*get_rankof)( const CHAM_desc_t*, int, int ),
			       void* get_rankof_arg);

Destroys matrix descriptor.

int CHAMELEON_Desc_Destroy (CHAM_desc_t **desc);

Ensures that all data of the descriptor are up-to-date.

int CHAMELEON_Desc_Acquire (CHAM_desc_t  *desc);

Release the data of the descriptor acquired by the application. Should be called if CHAMELEON_DescAcquire has been called on the descriptor and if you do not need to access to its data anymore.

int CHAMELEON_Desc_Release (CHAM_desc_t  *desc);

Ensure that all data are up-to-date in main memory (even if some tasks have been processed on GPUs).

int CHAMELEON_Desc_Flush(CHAM_desc_t  *desc, RUNTIME_sequence_t *sequence);

Create a sequence.

int CHAMELEON_Sequence_Create  (RUNTIME_sequence_t **sequence);

Destroy a sequence.

int CHAMELEON_Sequence_Destroy (RUNTIME_sequence_t *sequence);

Wait for the completion of a sequence.

int CHAMELEON_Sequence_Wait    (RUNTIME_sequence_t *sequence);

Terminate a sequence.

int CHAMELEON_Sequence_Flush(RUNTIME_sequence_t *sequence, RUNTIME_request_t *request)

• CHAMELEON_PARALLEL_WORKER_LEVEL=hardware-level[:number-of-parallel-workers]

Specify the number of parallel workers per hardware-level. The default value is 1. Note that hardware-level must correspond to an hwloc machine level type (hwloc_objtypet) e.g.: L2, L3, SOCKET, MACHINE.

• CHAMELEON_PARALLEL_WORKER_SHOW : When defined, the parallel workers contents is displayed.

For now, there is still an issue of bad performances with the usage of the lws scheduler with the parallel workers.

In the following examples, STARPU_MAIN_THREAD_BIND is set to 1 to bind the main thread of StarPU to a dedicated CPU, subtracted from the CPU workers. This avoids using a whole parallel worker to make the submission.

The machine has 64 CPUs. One is dedicated to the task submission, Two CPUs are dedicated to run the GPUs.