Page 1 of 1

AMD Epyc 7H12 Power Readings

Posted: Thu Nov 16, 2023 4:31 pm
by Skillz
I happen to have a really expensive power meter for measuring power use on my hosts. I will be making threads as I test the hosts to see what kind of power they draw at the wall.

7H12 - 128 tasks of Universe - ~305W at the wall - 2.5Ghz

Re: AMD Epyc 7H12 Power Readings

Posted: Sat Nov 18, 2023 8:55 pm
by Skillz
Private GFN full 128 threads - 290W at the wall - 2.2Ghz

Re: AMD Epyc 7H12 Power Readings

Posted: Sun Nov 19, 2023 4:56 am
by StefanR5R
I would expect Private GFN Server to pull more power than Universe@Home. One possible reason could be a cache bottleneck at Private GFN Server.

I had a quick look at some random hosts at Private GFN Server, and they show the following FFT lengths in current workunits:
"LLR2 testing" .......... 120K --> 0.94 MB cache footprint
"PRST Testing" ..... 2x288K --> 4.5 MB cache footprint?
"Sandbox 2" ............ 2x36K --> 0.6 MB cache footprint?
So there should be enough cache to run single-threaded LLR 2 testing and Sandbox 2 on all logical CPUs. But PRST Testing perhaps need to be restricted to 2 tasks at once per CCX (on Zen 2 CPUs, such as the 7H12), and forced with CCX-aligned CPU affinities.

If you want the operating system to retain all threads of an individual task within one and the same CCX for much of the time (rather than setting CPU affinities yourself), you can enable the BIOS option "ACPI SRAT L3 Cache as NUMA Domain". Works well with Linux at least. I don't know if Windows takes this up properly, but it should.

SMT may or may not be beneficial to either of these three applications. If SMT doesn't help, it may be especially unhelpful in Windows, compared to Linux.

--------

On the other hand, the fact that CPU clocks dropped from 2.5 GHz for Universe to 2.2 GHz for PGFN makes it look as if the CPU firmware did in fact encounter some sort of power limit within the cores, to clock them down accordingly.

Re: AMD Epyc 7H12 Power Readings

Posted: Sun Nov 19, 2023 12:37 pm
by Skillz
I was wondering if it was the VRMs that got hot for the project causing the CPU to throttle. The AsRock board has no way to monitor VRM temps and I don't know of a way to check through the OS (Linux) to see VRM temps. I'm assuming the AsRock board doesn't have a temp sensor on the VRM since the IPMI/BMC sensor list doesn't list them like the Supermicro dual board I have does.

Re: AMD Epyc 7H12 Power Readings

Posted: Sun Nov 19, 2023 2:09 pm
by crashtech
Perhaps server boards are expecting the VRMs to be in a wind tunnel and therefore aren't an item of concern.

Re: AMD Epyc 7H12 Power Readings

Posted: Sun Nov 19, 2023 7:59 pm
by mmonnin
Like many PG CPU apps, PRST slows down with too many concurrent tasks on my 5950s so I limit the # of threads and let the other threads run something else.

Sandbox 2?

Re: AMD Epyc 7H12 Power Readings

Posted: Sun Nov 19, 2023 8:09 pm
by Skillz
The only thing I am getting are the LLR2 Testing tasks. I'm not even sure what Sandbox 2 is.

I think I might set a box fan up next to the host to move air across the whole board. See if that helps the cooling and gets it to boost higher.

Re: AMD Epyc 7H12 Power Readings

Posted: Mon Nov 20, 2023 1:52 pm
by mmonnin
Damn, I missed the end of PRST tasks notification on the PGFN main page. Like 4k hours away from my 26k WUProp hours goal. I have like 2/3 of the remaining tasks but it's not enough hours once they timeout. No wonder there have been none.

Re: AMD Epyc 7H12 Power Readings

Posted: Tue Nov 21, 2023 4:26 pm
by StefanR5R
StefanR5R wrote: Sun Nov 19, 2023 4:56 am I had a quick look at some random hosts at Private GFN Server, and they show the following FFT lengths in current workunits:
"LLR2 testing" .......... 120K --> 0.94 MB cache footprint
"PRST Testing" ..... 2x288K --> 4.5 MB cache footprint?
"Sandbox 2" ............ 2x36K --> 0.6 MB cache footprint?
So there should be enough cache to run single-threaded LLR 2 testing and Sandbox 2 on all logical CPUs. But PRST Testing perhaps need to be restricted to 2 tasks at once per CCX (on Zen 2 CPUs, such as the 7H12), and forced with CCX-aligned CPU affinities.
Well, it's too late now due to the end of the "PRST testing" subproject, but for posterity:
The recipe to configure multithreading in PRST was posted by stream in the project's thread,
https://www.primegrid.com/forum_thread. ... rue#159303

Code: Select all

<app_config>
  <app_version>
    <app_name>prst</app_name>
    <cmdline>-t 2</cmdline>
    <avg_ncpus>2</avg_ncpus>
  </app_version>
</app_config>
…for 2-threaded tasks. If you put two concurrent tasks onto each Zen 2 CCX, try 2-threaded and 4-threaded tasks, i.e. MT unused/ SMT used. Either way, you should force affinities because otherwise you still bog the CPU down with memory accesses.