MLC@Home Cuda app is CPU limited

Help with projects and applications, client configuration, system software tweaks.
The pinboard for cheat sheets.
Post Reply
User avatar
biodoc
TAAT Member
Reactions:
Posts: 660
Joined: Sun Sep 15, 2019 3:22 pm
Location: Massachusetts, USA

MLC@Home Cuda app is CPU limited

Post by biodoc » Wed Jan 06, 2021 2:47 pm

I've noticed that the MLC cuda app is CPU limited so it may be a good project for some low powered cards.

Observations with a 2070S on a 3700X running universe on the CPUs:

1) 1 MLC cuda task per GPU with one free CPU thread (~65% shader utilization)
2) 2 MLC cuda tasks per GPU with 2 free CPU threads (~65% shader utilization)
3) 1 MLC cuda task per GPU with 2 free CPU threads (~70% shader utilization)
4) 1 MLC cuda task per GPU with universe suspended (~83% shader utilization)
5) 1 MLC cuda task per GPU with 8 free CPU threads (~80% shader utilization)( 80 watts draw on the GPU)

Here's the app_config.xml I used for test #2.

Code: Select all

<app_config>
	<app>
		<name>mlds-gpu</name>
		<gpu_versions>
			<gpu_usage>0.5</gpu_usage>
			<cpu_usage>0.01</cpu_usage>
		</gpu_versions>		
	</app>
</app_config>

Fardringle
TAAT Member
Reactions:
Posts: 464
Joined: Thu May 07, 2020 2:21 pm

Re: MLC@Home Cuda app is CPU limited

Post by Fardringle » Wed Jan 06, 2021 3:09 pm

I have the GTX 750ti in my daughters' computer (with an i7-4790) running MLC right now. It only gets about 8-10 hours of run time per day since they actually use that computer a lot for school and other stuff, and it's averaging a massive 46K per day over the 6 days I've had it attached to the project. :lol:

I'll have to remember to check the GPU utilization while it is running.

User avatar
10esseetony
TAAT Member
Reactions:
Posts: 399
Joined: Sun Sep 15, 2019 7:02 pm

Re: MLC@Home Cuda app is CPU limited

Post by 10esseetony » Sun Jan 10, 2021 9:24 am

Thanks @biodoc

edit: and @StefanR5R for pointing me here

Fardringle
TAAT Member
Reactions:
Posts: 464
Joined: Thu May 07, 2020 2:21 pm

Re: MLC@Home Cuda app is CPU limited

Post by Fardringle » Sun Jan 10, 2021 1:23 pm

The 750Ti is running at 100% utilization with a single CPU core free on the i7-4790, so this is probably only an issue with fast GPUs.

crashtech
TAAT Member
Reactions:
Posts: 911
Joined: Sun Sep 15, 2019 4:45 pm
Location: Idaho, USA

Re: MLC@Home Cuda app is CPU limited

Post by crashtech » Thu Jan 21, 2021 3:51 pm

So something odd has popped up for me now that I have had time to look at what MLC@Home is doing. It looks like I can get better max GPU utilization in Windows(!) Linux seems to top out around 80% with no CPU going, but Windows can exceed 90%. Can anyone corroborate this?

Fardringle
TAAT Member
Reactions:
Posts: 464
Joined: Thu May 07, 2020 2:21 pm

Re: MLC@Home Cuda app is CPU limited

Post by Fardringle » Thu Jan 21, 2021 11:57 pm

I can't really test in Linux, but I can confirm that at least with my wimpy GPUs (GTX 750ti and GTX 1060) that GPU utilization is near 100% with a single task running and a single CPU core/thread free.

Fardringle
TAAT Member
Reactions:
Posts: 464
Joined: Thu May 07, 2020 2:21 pm

Re: MLC@Home Cuda app is CPU limited

Post by Fardringle » Fri Jan 22, 2021 6:59 pm

Since I have temporary use of a Quadro RTX 3000, I've been playing with it (stress testing ;)) and decided to let it have a go at the MLC GPU app for a bit.

Running a single GPU task with the CPU idle, Task Manager in Windows showed that it was using about 1.33 CPU threads and GPU-Z showed about 25-35% GPU utilization.

I added an app_config.xml file with <gpu_usage> and <cpu_usage> both set to .33 to see what would happen. BOINC did start running three GPU tasks simultaneously, and GPU-Z jumped up to 85-90% GPU utilization. However, it seems that the project is ignoring the <cpu_usage> tag, at least on this system, since each one of the three GPU threads was now using 1.33 CPU threads of its own (4 total threads in use).


That might explain biodoc's results that showed a significant improvement in GPU usage when the CPU is idle, if the app really wants to have that much CPU available for each running task.

After about 15 minutes of running three tasks simultaneously, GPU usage dropped to zero and % progress on the tasks also stopped, even though the Elapsed timer was still counting. I let it sit that way for a while with no change, then rebooted the computer but the tasks still wouldn't proceed any more.

Then, out of curiosity I changed the app_config file to
<gpu_usage>1</gpu_usage>
<cpu_usage>2</cpu_usage>

to see if it would run better with even more CPU dedicated to the single GPU task.

After I loaded the config files, a single MLC task started running normally again, with GPU-Z showing around 45-50% GPU load and Task Manager still showing 1.33 CPU threads active on the task. I'm not sure what the best settings will be and don't have time to do any more testing with this system right now, but I thought I'd share my initial findings to see if they make sense to (or help) anyone else..

StefanR5R
TAAT Member
Reactions:
Posts: 928
Joined: Wed Sep 25, 2019 4:32 pm

Re: MLC@Home Cuda app is CPU limited

Post by StefanR5R » Sat Jan 23, 2021 1:58 am

Originally, both <cpu_usage> and <gpu_usage> are doing the same: They inform boinc-client's task scheduler how the application behaves. But they do not control the application itself.

Only a few applications (vboxwrapper and some other multithreaded applications) have re-purposed <cpu_usage> and use it to modify application behavior. (Edit: Actually, they use app_version::avg_ncpus for this, but definitely not app::gpu_versions::cpu_usage. These applications don't have a GPU version after all.)
Fardringle wrote:
Fri Jan 22, 2021 6:59 pm
After about 15 minutes of running three tasks simultaneously, GPU usage dropped to zero and % progress on the tasks also stopped, even though the Elapsed timer was still counting. I let it sit that way for a while with no change, then rebooted the computer but the tasks still wouldn't proceed any more.
[...]
After I loaded the config files, a single MLC task started running normally again,
Sounds like a bug in the application, or in the CUDA libraries/driver stack.

Fardringle
TAAT Member
Reactions:
Posts: 464
Joined: Thu May 07, 2020 2:21 pm

Re: MLC@Home Cuda app is CPU limited

Post by Fardringle » Sat Jan 23, 2021 1:02 pm

StefanR5R wrote:
Sat Jan 23, 2021 1:58 am
Originally, both <cpu_usage> and <gpu_usage> are doing the same: They inform boinc-client's task scheduler how the application behaves. But they do not control the application itself.

Only a few applications (vboxwrapper and some other multithreaded applications) have re-purposed <cpu_usage> and use it to modify application behavior. (Edit: Actually, they use app_version::avg_ncpus for this, but definitely not app::gpu_versions::cpu_usage. These applications don't have a GPU version after all.)
Yeah. I just wanted to see what would happen with the changes. :)
StefanR5R wrote:
Sat Jan 23, 2021 1:58 am
Fardringle wrote:
Fri Jan 22, 2021 6:59 pm
After about 15 minutes of running three tasks simultaneously, GPU usage dropped to zero and % progress on the tasks also stopped, even though the Elapsed timer was still counting. I let it sit that way for a while with no change, then rebooted the computer but the tasks still wouldn't proceed any more.
[...]
After I loaded the config files, a single MLC task started running normally again,
Sounds like a bug in the application, or in the CUDA libraries/driver stack.
Could be a bug specifically between the app and the Quadro RTX drivers, too...

Post Reply