\( \newcommand{\blu}[1]{{\color{blue}#1}} \newcommand{\red}[1]{{\color{red}#1}} \newcommand{\grn}[1]{{\color{green!50!black}#1}} \newcommand{\local}{_i} \newcommand{\inv}{^{-1}} % Index for interface and interior \newcommand{\G}{\Gamma} \newcommand{\Gi}{\Gamma_i} \newcommand{\I}{{\cal I}} \newcommand{\Ii}{\I_i} % Matrix A \newcommand{\A}{{\cal A}} \newcommand{\Ai}{\A\local} \newcommand{\Aj}{\A_j} \newcommand{\Aib}{\bar{\A}\local} \newcommand{\AII}{\A_{\I\I}} \newcommand{\AIG}{\A_{\I\G}} \newcommand{\AGI}{\A_{\G\I}} \newcommand{\AGG}{\A_{\G\G}} \newcommand{\AIiIi}{\A_{\Ii\Ii}} \newcommand{\AIiGi}{\A_{\Ii\Gi}} \newcommand{\AGiIi}{\A_{\Gi\Ii}} \newcommand{\AGiGi}{\A_{\Gi\Gi}} \newcommand{\AGiGiw} {{\ensuremath{\A_{\Gi\Gi}^w}}} \newcommand{\AIIi}{\AII\local} \newcommand{\AIGi}{\AIG\local} \newcommand{\AGIi}{\AGI\local} \newcommand{\AGGi}{\AGG\local} \newcommand{\Ab}{\bar{\A}} \newcommand{\Ah}{{\widehat{\A}}} \newcommand{\Aih}{{\Ah\local}} \newcommand{\At}{{\widetilde{\A}}} \newcommand{\Ait}{{\At\local}} \newcommand{\Ao}{\A_0} \newcommand{\Aot}{\At_0} \newcommand{\Aob}{\Ab_0} \newcommand{\AiNN}{\Ai^{(NN)}{}} \newcommand{\AitNN}{\Ait^{(NN)}{}} \newcommand{\AiAS}{\Ai^{(AS)}{}} \newcommand{\AitAS}{\Ait^{(AS)}{}} % Matrix S \renewcommand{\S}{{\cal S}} \newcommand{\Si}{\S\local} \newcommand{\Sb}{\bar{\S}} \newcommand{\Sib}{\Sb\local} \newcommand{\Sh}{{\widehat{\S}}} \newcommand{\Sih}{{\Sh\local}} \newcommand{\St}{{\widetilde{\S}}} \newcommand{\Sit}{{\St\local}} \newcommand{\So}{\S_0} \newcommand{\Soi}{\S_0^i} \newcommand{\Sot}{\St_0} \newcommand{\Sob}{\Sb_0} \newcommand{\SiNN}{\Si^{(NN)}{}} \newcommand{\SitNN}{\Sit^{(NN)}{}} \newcommand{\SiAS}{\Si^{(AS)}{}} \newcommand{\SitAS}{\Sit^{(AS)}{}} % Matrix K \newcommand{\K}{{\cal K}} \newcommand{\Ki}{\K\local} \newcommand{\Kb}{\bar{\K}} \newcommand{\Kib}{\Kb\local} \newcommand{\Kh}{{\widehat{\K}}} \newcommand{\Kih}{{\Kh\local}} \newcommand{\Kt}{{\widetilde{\K}}} \newcommand{\Kit}{{\Kt\local}} \newcommand{\Ko}{\K_0} \newcommand{\Kot}{\Kt_0} \newcommand{\Kob}{\Kb_0} \newcommand{\KiNN}{\Ki^{(NN)}{}} \newcommand{\KitNN}{\Kit^{(NN)}{}} \newcommand{\KiAS}{\Ki^{(AS)}{}} \newcommand{\KitAS}{\Kit^{(AS)}{}} \newcommand{\KII}{\K_{\I\I}} \newcommand{\KIG}{\K_{\I\G}} \newcommand{\KGI}{\K_{\G\I}} \newcommand{\KGG}{\K_{\G\G}} \newcommand{\KIiIi}{\K_{\Ii\Ii}} \newcommand{\KIiGi}{\K_{\Ii\Gi}} \newcommand{\KGiIi}{\K_{\Gi\Ii}} \newcommand{\KGiGi}{\K_{\Gi\Gi}} \newcommand{\KIIi}{\KII\local} \newcommand{\KIGi}{\KIG\local} \newcommand{\KGIi}{\KGI\local} \newcommand{\KGGi}{\KGG\local} \newcommand{\KGiGiw} {{\ensuremath{\K_{\Gi\Gi}^w}}} % Matrix B \newcommand{\B}{{\cal B}} \newcommand{\Bi}{\B\local} \newcommand{\Bib}{\widehat{\B}\local} \newcommand{\Bob}{\widehat{\B}_0} % Matrix C \newcommand{\C}{{\cal C}} % Matrix T \newcommand{\T}{{\cal T}} \newcommand{\Ti}{{\T\local}} % Vectors \newcommand{\uI}{u_\I} \newcommand{\uG}{u_\G} \newcommand{\xI}{x_\I} \newcommand{\xG}{x_\G} \newcommand{\bI}{b_\I} \newcommand{\bG}{b_\G} \newcommand{\fI}{f_\I} \newcommand{\fG}{f_\G} \newcommand{\ftG}{\widetilde f_\G} \newcommand{\ftGi}{\widetilde f_\Gi^{(i)}} \newcommand{\ftGj}{\widetilde f_\Gj^{(j)}} \)

Compose MaPHyS++ weekly benchmark on plafrim: experimental setup

Table of Contents

Back to index.

See the Experimental setup

1. Test case description

We consider a stationary heterogeneous diffusion equation (or Darcy equation) in a 3D stratified medium \(\nabla \cdot (k\nabla u) = 1\). The equation is discretized using the Finite Element Method. The matrices and right hand side are generated with genfem.

This benchmark is a weak scaling test with 18, 36, 72 and 144 subdomains. Each subdomain consists of a \(30 \times 30 \times 30\) cube. In the Y-direction, conductivity alternates every 5 elements, with an Heterogeneity of 1000.

Subdomain illustration: 30x30x30 cube

Figure 1: One subdomain, a cube of edge 30, conductivity \(k=1\) in blue and \(k=1000\) in red

For the baton test case, the subdomains are put one against the other in the X-dimension.

Baton test case illustration

Figure 2: Baton test case, with \(6 \times 1 \times 1\) subdomains

For the cuboid test case, the subdomains are put so as to form the "most cubic" shape possible.

Cuboid test case illustration

Figure 3: Cuboid test case, with \(4 \times 3 \times 2\) subdomains

2. Baton test case, on the input matrix K

2.1. openblas

mpp_K_openblas.png

2.2. mkl

mpp_K_mkl.png

3. Baton test case, on the Schur complement matrix S

3.1. openblas

mpp_S_openblas.png

3.2. mkl

mpp_S_mkl.png

4. Experiments in command line form

4.1. Preliminaries

rm -rf build && mkdir build && cd build

4.2. Command (Example)

For 36 subdomains with AS/S preconditionners using openblas 

# creating Makefile
guix shell --pure -D maphys++ slurm eigen armadillo scalapack zlib -- cmake .. -DCMAKE_BUILD_TYPE=Release -DMAPHYSPP_COMPILE_EXAMPLES=ON -DMAPHYSPP_COMPILE_TESTS=OFF -DMAPHYSPP_GCC_WARNINGS=OFF
# compiling binary target
guix shell --pure -D maphys++ slurm eigen armadillo scalapack zlib -- make mpp_driver_cg

cd src/examples/

cat <<EOF > input.in
n_subdomains: 36
max_iter: 500

# Working on the Schur complement
use_schur: 1

# Path to partitions
partitions: /home/gitlab-compose/baton_30_newformat
# Partiton type
partition_type: baton
# Type of preconditioner
precond: AS
# Pastix or Mumps
direct_solver: Pastix

tolerance: 1e-6
EOF

cat <<EOF > batch.batch
#!/bin/bash
#SBATCH --job-name=maphyspp
#SBATCH --nodes=1
#SBATCH --ntasks=36
#SBATCH --ntasks-per-node=36
#SBATCH --cpus-per-task=1
#SBATCH --time=00:10:00
#SBATCH --output=slurm.out
#SBATCH --error=slurm.out
#SBATCH --exclusive
#SBATCH --constraint bora
module load mpi/openmpi/4.1.1
export OMPI_MCA_pml='^ucx'

guix shell --pure --preserve='^SLURM|^OMPI' -D maphys++ slurm eigen armadillo scalapack zlib -- mpiexec -n \${SLURM_NPROCS} ./mpp_driver_cg input.in
EOF

sbatch batch.batch

For 36 subdomains with AS/S preconditionners using mkl, you should add to the guix environment: 

--with-input=mumps-openmpi=mumps-mkl-openmpi --with-input=openblas=mkl

Output is in slurm.out, errors are in slurm.err.

5. Source

Run date: 

sam. 13 janv. 2024 01:45:57 CET

Commit: 

31c44a4b8f2d91674a39bfad010acd0d6e8c92cb

Channels: 

(list (channel
        (name 'guix-hpc)
        (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
        (branch "master")
        (commit
          "3ffe8f79a2f128111785fb1107d5f803e00bf00c"))
      (channel
        (name 'guix-past)
        (url "https://gitlab.inria.fr/guix-hpc/guix-past")
        (branch "master")
        (commit
          "5ffb6fe235b4cddd99f7579afc1fe0e670c1e43e")
        (introduction
          (make-channel-introduction
            "0c119db2ea86a389769f4d2b9c6f5c41c027e336"
            (openpgp-fingerprint
              "3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5"))))
      (channel
        (name 'guix-science-nonfree)
        (url "https://github.com/guix-science/guix-science-nonfree.git")
        (branch "master")
        (commit
          "9a3f3824d8ed289832d706679410edadac1202ae")
        (introduction
          (make-channel-introduction
            "58661b110325fd5d9b40e6f0177cc486a615817e"
            (openpgp-fingerprint
              "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
      (channel
        (name 'guix-hpc-non-free)
        (url "https://gitlab.inria.fr/guix-hpc/guix-hpc-non-free.git")
        (branch "master")
        (commit
          "372c5f471448b32c9204f79c1d46e9b984d03c07"))
      (channel
        (name 'guix)
        (url "https://git.savannah.gnu.org/git/guix.git")
        (branch "master")
        (commit
          "8c294e43eb0b0ea61da7c0570872e0908f64c8ed")
        (introduction
          (make-channel-introduction
            "9edb3f66fd807b096b48283debdcddccfea34bad"
            (openpgp-fingerprint
              "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))

results.csv 

NbDomains,preconditioner,use_schur,blas,DirectSolver,nIter,CoarseEigenSolve,DirectSolve,CoarsePcdSetup,IterativeSolve,LocalPcdSetup,SchurComputation,TotalTime,RunTime,GlobMatOrder,robin_weight,threadspertask
18,0,0,openblas,Mumps,3430,0,0,0,7.00155,0,0,7.00741,7.52967,519918,1.0,1
36,0,0,openblas,Mumps,4758,0,0,0,11.2522,0,0,11.2627,11.8042,1038876,1.0,1
,0,0,openblas,,,,,,,,,,,,1.0,1
,0,0,openblas,,,,,,,,,,,,1.0,1
18,0,0,mkl,Mumps,3430,0,0,0,7.01078,0,0,7.01645,7.75559,519918,1.0,1
36,0,0,mkl,Mumps,4758,0,0,0,11.2916,0,0,11.3024,11.994,1038876,1.0,1
,0,0,mkl,,,,,,,,,,,,1.0,1
,0,0,mkl,,,,,,,,,,,,1.0,1
18,0,1,openblas,Mumps,662,0,0.0357824,0,1.77802,0,1.15401,2.97139,3.47094,519918,1.0,1
36,0,1,openblas,Mumps,901,0,0.0533284,0,5.55604,0,1.65416,7.34062,8.31301,1038876,1.0,1
,0,1,openblas,,,,,,,,,,,,1.0,1
,0,1,openblas,,,,,,,,,,,,1.0,1
18,0,1,mkl,Mumps,666,0,0.028732,0,1.87234,0,1.15864,3.06413,3.72095,519918,1.0,1
36,0,1,mkl,Mumps,882,0,0.041272,0,4.12886,0,1.29136,5.46367,6.14979,1038876,1.0,1
,0,1,mkl,,,,,,,,,,,,1.0,1
,0,1,mkl,,,,,,,,,,,,1.0,1
18,diag,0,openblas,Mumps,1667,0,0,0,3.56792,0.000747302,0,3.5731,4.06387,519918,1.0,1
36,diag,0,openblas,Mumps,2907,0,0,0,7.30875,0.00149598,0,7.32046,7.88171,1038876,1.0,1
,diag,0,openblas,,,,,,,,,,,,1.0,1
,diag,0,openblas,,,,,,,,,,,,1.0,1
18,diag,0,mkl,Mumps,1667,0,0,0,3.55335,0.000781551,0,3.55936,4.17266,519918,1.0,1
36,diag,0,mkl,Mumps,2907,0,0,0,7.2497,0.00112526,0,7.25855,7.89684,1038876,1.0,1
,diag,0,mkl,,,,,,,,,,,,1.0,1
,diag,0,mkl,,,,,,,,,,,,1.0,1
18,diag,1,openblas,Mumps,266,0,0.0363662,0,0.723494,0.000142053,1.15644,1.92005,2.41261,519918,1.0,1
36,diag,1,openblas,Mumps,445,0,0.0526605,0,2.08185,0.000235363,1.40526,3.539,4.06708,1038876,1.0,1
,diag,1,openblas,,,,,,,,,,,,1.0,1
,diag,1,openblas,,,,,,,,,,,,1.0,1
18,diag,1,mkl,Mumps,266,0,0.0267398,0,0.709954,0.000139643,1.07117,1.81035,2.37822,519918,1.0,1
36,diag,1,mkl,Mumps,445,0,0.0402094,0,2.07116,0.000322218,1.28593,3.39964,4.13128,1038876,1.0,1
,diag,1,mkl,,,,,,,,,,,,1.0,1
,diag,1,mkl,,,,,,,,,,,,1.0,1
18,AS,0,openblas,Mumps,214,0,0,0,6.5981,0.0893631,0,6.60304,7.09816,519918,1.0,1
36,AS,0,openblas,Mumps,353,0,0,0,12.6122,0.100284,0,12.62,13.1687,1038876,1.0,1
,AS,0,openblas,,,,,,,,,,,,1.0,1
,AS,0,openblas,,,,,,,,,,,,1.0,1
18,AS,0,mkl,Mumps,214,0,0,0,16.6212,0.133975,0,16.6578,17.2374,519918,1.0,1
36,AS,0,mkl,Mumps,353,0,0,0,13.6171,0.105413,0,13.6956,14.7539,1038876,1.0,1
,AS,0,mkl,,,,,,,,,,,,1.0,1
,AS,0,mkl,,,,,,,,,,,,1.0,1
18,AS,1,openblas,Mumps,34,0,0.0357249,0,0.428878,0.0574972,1.15043,1.61878,2.11335,519918,1.0,1
36,AS,1,openblas,Mumps,59,0,0.0527261,0,0.917067,0.0828637,1.41301,2.38223,2.92677,1038876,1.0,1
,AS,1,openblas,,,,,,,,,,,,1.0,1
,AS,1,openblas,,,,,,,,,,,,1.0,1
18,AS,1,mkl,Mumps,34,0,0.0258663,0,0.312519,0.0580292,1.06763,1.4118,2.01173,519918,1.0,1
36,AS,1,mkl,Mumps,59,0,0.0409987,0,0.760328,0.0827662,1.28972,2.09336,2.7909,1038876,1.0,1
,AS,1,mkl,,,,,,,,,,,,1.0,1
,AS,1,mkl,,,,,,,,,,,,1.0,1
18,RR,0,openblas,Mumps,1175,0,0,0,31.2165,0.0415131,0,31.2224,31.7122,519918,1.0,1
36,RR,0,openblas,Mumps,2027,0,0,0,67.1622,0.0495898,0,67.1731,67.7077,1038876,1.0,1
,RR,0,openblas,,,,,,,,,,,,1.0,1
,RR,0,openblas,,,,,,,,,,,,1.0,1
18,RR,0,mkl,Mumps,1174,0,0,0,84.6381,0.0746731,0,84.6765,85.294,519918,1.0,1
36,RR,0,mkl,Mumps,2030,0,0,0,64.4206,0.0489026,0,64.4315,65.1069,1038876,1.0,1
,RR,0,mkl,,,,,,,,,,,,1.0,1
,RR,0,mkl,,,,,,,,,,,,1.0,1
18,RR,1,openblas,Mumps,69,0,0.0358692,0,0.702815,0.0318874,1.15257,1.89604,2.37967,519918,1.0,1
36,RR,1,openblas,Mumps,96,0,0.0512391,0,1.34625,0.0547555,1.41232,2.81091,3.34478,1038876,1.0,1
,RR,1,openblas,,,,,,,,,,,,1.0,1
,RR,1,openblas,,,,,,,,,,,,1.0,1
18,RR,1,mkl,Mumps,68,0,0.026694,0,0.48284,0.0371663,1.06481,1.57766,2.19229,519918,1.0,1
36,RR,1,mkl,Mumps,97,0,0.0407194,0,1.08893,0.0650637,1.27402,2.40628,3.13853,1038876,1.0,1
,RR,1,mkl,,,,,,,,,,,,1.0,1
,RR,1,mkl,,,,,,,,,,,,1.0,1

Date: 18/01/2024 at 18:51:43

Author: HiePACS HiePACS

Created: 2024-01-18 Thu 18:51

Validate