Benchmark Ampere Altra Q32-17 in HL15 Chassis

[geerlingguy]

I am running an Ampere Altra Q32-17 in a 45Homelab HL15 4U rackmount chassis, and currently I can get the power draw at idle down to around 50W if I limit the system to just the motherboard, CPU, 32 GB of RAM, and a 2 TB NVMe SSD.

With all the stuff I'm using to make it a NAS (6x HDD, 4x SSD, HBA), power draw rises to around 120W. So I'll have to remove all that when doing my final power efficiency calculations.

I'm also running Rocky Linux 9, so I'll need to check if this playbook even runs properly there. We'll see!

Tests to run:

• Full system with HDDs spinning and everything, 64 GB RAM
• Bare system with no HDDs, HBA, etc., 64 GB RAM (4x sticks)
• Bare system with 32 GB RAM (2x sticks)
• Bare system with 128 GB RAM (8x sticks)

[geerlingguy]

Initial test (with all drives in and such) resulted in around 154W total system load for 313.84 Gflops, so 2.03 Gflops/W (NOTE: This is with all HDDs spinning, HBA installed, etc., with 64 GB of RAM).

================================================================================
HPLinpack 2.3  --  High-Performance Linpack benchmark  --   December 2, 2018
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :   70717
NB     :     256
PMAP   : Row-major process mapping
P      :       1
Q      :      32
PFACT  :   Right
NBMIN  :       4
NDIV   :       2
RFACT  :   Crout
BCAST  :  1ringM
DEPTH  :       1
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

--------------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
      ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be               1.110223e-16
- Computational tests pass if scaled residuals are less than                16.0

================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4       70717   256     1    32             751.26             3.1384e+02
HPL_pdgesv() start time Wed Feb 14 23:41:40 2024

HPL_pdgesv() end time   Wed Feb 14 23:54:11 2024

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=   2.27276032e-03 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

[geerlingguy]

With Ps at 4 and Qs at 8, 156W giving me a score of 334.21 Gflops, for 2.14 Gflops/W.

Will pull storage-related stuff out next, and try again.

[geerlingguy]

Pulled all the storage, the HBA, and the high-wattage case fans (just running mobo with M.2 NVMe, 64GB RAM, and CPU + Noctua cooler), and system power is normalizing around 100W.

Final result is 332.07 Gflops, for 3.32 Gflops/W. I think I could get this higher with 128 GB of RAM, but we'll see. I ordered four more sticks of Samsung DDR3200 ECC RAM to match the sticks I have installed already.

================================================================================
HPLinpack 2.3  --  High-Performance Linpack benchmark  --   December 2, 2018
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :   70717
NB     :     256
PMAP   : Row-major process mapping
P      :       4
Q      :       8
PFACT  :   Right
NBMIN  :       4
NDIV   :       2
RFACT  :   Crout
BCAST  :  1ringM
DEPTH  :       1
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

--------------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
      ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be               1.110223e-16
- Computational tests pass if scaled residuals are less than                16.0

================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4       70717   256     4     8             710.01             3.3207e+02
HPL_pdgesv() start time Thu Feb 15 00:24:22 2024

HPL_pdgesv() end time   Thu Feb 15 00:36:12 2024

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=   2.64084523e-03 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

[geerlingguy]

Pulled two sticks of RAM and now it's running at 90W—with a result of 260.41 Gflops. So 2.89 Gflops/W at 32GB RAM:

================================================================================
HPLinpack 2.3  --  High-Performance Linpack benchmark  --   December 2, 2018
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :   48905
NB     :     256
PMAP   : Row-major process mapping
P      :       4
Q      :       8
PFACT  :   Right
NBMIN  :       4
NDIV   :       2
RFACT  :   Crout
BCAST  :  1ringM
DEPTH  :       1
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

--------------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
      ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be               1.110223e-16
- Computational tests pass if scaled residuals are less than                16.0

================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4       48905   256     4     8             299.45             2.6041e+02
HPL_pdgesv() start time Thu Feb 15 00:49:00 2024

HPL_pdgesv() end time   Thu Feb 15 00:54:00 2024

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=   2.67958453e-03 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

[geerlingguy]

With 128 GB of RAM, I got a score of 351.69 Gflops, averaging 119W, so 2.96 Gflops/W

================================================================================
HPLinpack 2.3  --  High-Performance Linpack benchmark  --   December 2, 2018
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :  100551
NB     :     256
PMAP   : Row-major process mapping
P      :       4
Q      :       8
PFACT  :   Right
NBMIN  :       4
NDIV   :       2
RFACT  :   Crout
BCAST  :  1ringM
DEPTH  :       1
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

--------------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
      ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be               1.110223e-16
- Computational tests pass if scaled residuals are less than                16.0

================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4      100551   256     4     8            1927.16             3.5169e+02
HPL_pdgesv() start time Mon Feb 19 17:58:46 2024

HPL_pdgesv() end time   Mon Feb 19 18:30:53 2024

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=   2.36368774e-03 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

[geerlingguy]

Interestingly, with all 8 memory channels filled, the CPU consumes a bit more power, 20W total (for four 16 GB DDR4 3200 sticks, averaging 5W per stick, which lines up with the 128-core CPU when I was testing 2/4/6 sticks).

So at least that's consistent!