Multiple GPU build ideas

Hardware upgrades, reuse, repairs, and optimizations. Fire hazards and bleeding edges.
Skillz
Site Admin
Reactions:
Posts: 1367
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz » Sat Sep 11, 2021 11:28 am

Icecold wrote:
Sat Sep 11, 2021 11:26 am
That should be the correct bios page. Maybe that specific board doesn't support it? :( I did get a message from my WTB post on [H] form somebody that said "I have a handful of full-size AsRock Rack server mb's that are socket 2011v3. They all support bifurcation. " so I'm in the process of trying to buy at least one of those to see how it works out.
Details on them? I may be interested as well.

User avatar
pututu
TAAT Member
Reactions:
Posts: 159
Joined: Thu Nov 05, 2020 11:12 pm

Re: Multiple GPU build ideas

Post by pututu » Sat Sep 11, 2021 11:36 am

I have another Asrock X99M Killer Fatal1ty with Haswell xeon. I can check that later.

I'm guessing it is the workstation or the server board that supports the bifurcation.

Icecold
TAAT Member
Reactions:
Posts: 1294
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold » Sat Sep 11, 2021 11:39 am

pututu wrote:
Sat Sep 11, 2021 11:36 am
I have another Asrock X99M Killer Fatal1ty with Haswell xeon. I can check that later.

I'm guessing it is the workstation or the server board that supports the bifurcation.
It seems like a total crapshoot TBH. It seems to be there in some boards you wouldn't expect it to be but isn't in others. I appreciate you checking though
Skillz wrote:
Sat Sep 11, 2021 11:28 am
Details on them? I may be interested as well.
I'm still waiting on more info but I'll let you know.

User avatar
pututu
TAAT Member
Reactions:
Posts: 159
Joined: Thu Nov 05, 2020 11:12 pm

Re: Multiple GPU build ideas

Post by pututu » Sat Sep 11, 2021 12:08 pm

BIOS looks the same as the other one other than the background color.
X99M.png

Skillz
Site Admin
Reactions:
Posts: 1367
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz » Sat Sep 11, 2021 12:49 pm

I wonder if AMD has a better chance of their boards supporting this since AMD isn't strict on their PCI-e lanes available to their CPUs.

Icecold
TAAT Member
Reactions:
Posts: 1294
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold » Sat Sep 11, 2021 3:32 pm

Hmmm.. what the heck. That is definitely where I would expect the setting to be. That's more or less where it is on my Gigabyte board. Bifurcation has been around for awhile.

The guy on [H] got back to me and they're actually Supermicro boards not Asrock but I'm waiting to confirm they do support bifurcation before I buy at least one.
Skillz wrote:
Sat Sep 11, 2021 12:49 pm
I wonder if AMD has a better chance of their boards supporting this since AMD isn't strict on their PCI-e lanes available to their CPUs.
Maybe, but I initially saw this setting talked about all the way back in the Intel Sandy bridge days for getting an extra PCIe slot, so I'm really not sure.

Icecold
TAAT Member
Reactions:
Posts: 1294
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold » Sat Sep 11, 2021 3:58 pm

These are the motherboards available -

2 x of these: https://www.supermicro.com/en/products/ ... d/X10SRM-F
1 x of these: https://www.supermicro.com/en/products/ ... d/X10SRL-F
1 x of these: https://www.supermicro.com/en/products/ ... d/X10SRi-F

The only one that I think even has a shot at working for this purpose is the last one. I might buy it just to see. This stuff gets fairly confusing with having to consider how many PCIe lanes are available etc.

Edit - that one only comes up to 4 x8 slots and 1 x4 slot with bifurcation. I may still buy it,

Icecold
TAAT Member
Reactions:
Posts: 1294
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold » Sat Sep 11, 2021 10:38 pm

I ended up buying this one - https://www.supermicro.com/en/products/ ... d/X10SRi-F plus a 5930k. I'm pretty sure it should be sufficient for at least 4 high powered cards, but we'll see. 4 cards per machine could be feasible for me maybe, but I'd really prefer to be in 6-8. I'll have to do more tests to see if x4 limits it.

Icecold
TAAT Member
Reactions:
Posts: 1294
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold » Sun Sep 12, 2021 5:47 pm

I ordered some bifurcation adapters from C-Payne. I'm about to order some of these risers as well - https://www.amazon.com/GELRHONR-Express ... 02&sr=8-12 I would imagine the C-Payne adapters will take a long time since they're shipping from Europe. I think I should be able to do 4 video cards at x8 and 1 at x4. I'm thinking something like 4x3070's at x8 and a either a 3060 or 2070 at x4, or something along those lines. The good part about that being that it won't be close to maxing a single 120v 20 amp circuit.

I think I probably just need to order the EPYC stuff @Skillz linked earlier in this thread, though for the next build if I don't want to be limited to 4 or 5 video cards.

StefanR5R
TAAT Member
Reactions:
Posts: 1190
Joined: Wed Sep 25, 2019 4:32 pm

Re: Multiple GPU build ideas

Post by StefanR5R » Mon Sep 13, 2021 3:53 pm

Icecold wrote:
Sat Sep 11, 2021 10:38 pm
I ended up buying this one - https://www.supermicro.com/en/products/ ... d/X10SRi-F plus a 5930k.
Does a plebeian i7 support registered DIMMs like this server board requires? And does this board accept an i7 in the first place?

Added in 3 minutes 17 seconds:
Icecold wrote:
Sun Sep 12, 2021 5:47 pm
I'm thinking something like 4x3070's at x8 and a either a 3060 or 2070 at x4,
I ran a 1080Ti + 1080 system for a while, and the different GPU performance and how the boinc client coped with it bothered me. (Well, I could have worked around that by using dedicated boinc client instances for the different GPUs. But I didn't, back then.)

Icecold
TAAT Member
Reactions:
Posts: 1294
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold » Mon Sep 13, 2021 4:03 pm

StefanR5R wrote:
Mon Sep 13, 2021 3:56 pm
Icecold wrote:
Sat Sep 11, 2021 10:38 pm
I ended up buying this one - https://www.supermicro.com/en/products/ ... d/X10SRi-F plus a 5930k.
Does a plebeian i7 support registered DIMMs like this server board requires? And does this board accept an i7 in the first place?

Added in 3 minutes 17 seconds:
Icecold wrote:
Sun Sep 12, 2021 5:47 pm
I'm thinking something like 4x3070's at x8 and a either a 3060 or 2070 at x4,
I ran a 1080Ti + 1080 system for a while, and the different GPU performance and how the boinc client coped with it bothered me. (Well, I could have worked around that by using dedicated boinc client instances for the different GPUs. But I didn't, back then.)
I didn't even think about the ECC issue. That CPU doesn't support ECC. I purchased it from somebody as a combo deal, and as far as I know he booted it up with that CPU in it. Maybe that motherboard can use non ECC ram and it's just not documented. I do have a couple E5-2670 v3's I rarely use that I could use if need be instead, and then move the i7 over to the machine I pull that from. That's a good point as well about the differing GPU performance being annoying to deal with in terms of bunkering, etc. I should probably try my best to make the builds all the same GPU, if I have enough of those GPU's available.

StefanR5R
TAAT Member
Reactions:
Posts: 1190
Joined: Wed Sep 25, 2019 4:32 pm

Re: Multiple GPU build ideas

Post by StefanR5R » Tue Sep 14, 2021 1:02 pm

Icecold wrote:
Mon Sep 13, 2021 4:03 pm
as far as I know he booted it up with that CPU in it.
Well, hopefully that's the case. Presence of ECC support in the DIMMs is not the issue, but rather a) if the HSW-E (as supposed to -EP) can deal with registered DIMMs (I don't know, never looked into that… or maybe you have to plug unregistered DIMMs then, even though this board is specified for RDIMMs and LRDIMMs?), and b) if the mainboard's BIOS accepts this CPU in the first place. For sure, if this CPU with its higher clocks work, then that's preferable for a GPU build.

Icecold
TAAT Member
Reactions:
Posts: 1294
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold » Tue Sep 14, 2021 7:39 pm

This should work. I'm going to mess around with it more once a few of the remaining parts come in, but the motherboard and CPU arrived today. I just booted up the board with just some random DDR4 non ECC RAM I had sitting around and was able to boot into bios and it detected the RAM, showed the correct CPU identifier, and I didn't notice any errors or issues.

I wonder how necessary ensuring the ram is running in quad channel is. I probably should just for the sake of reducing any potential bottlenecks. I should be able to get this up and running with 3 GPU's at least relatively soon, and then I think the bifurcation risers I ordered will take the longest since they're shipping from overseas.

Am I correct in assuming with this having a BMC and IPMI that I will be able to access bios settings, or choose my OS on dual boot, etc. across the network without physical access to it? If so that would be incredibly useful. I've never actually used that feature before so I wasn't 100% sure.

I'm going to have to give this some further thought in terms of how many GPU's to use per build. I hadn't really thought of the annoyances of managing multiple cards that have different performance on the same system, and I have a pretty mixed bag in terms of card types.

Skillz
Site Admin
Reactions:
Posts: 1367
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz » Tue Sep 14, 2021 9:05 pm

Icecold wrote:
Tue Sep 14, 2021 7:39 pm
Am I correct in assuming with this having a BMC and IPMI that I will be able to access bios settings, or choose my OS on dual boot, etc. across the network without physical access to it? If so that would be incredibly useful. I've never actually used that feature before so I wasn't 100% sure.
Yes, but you need to set that up. I've never done it, but that's what it is for yes.

StefanR5R
TAAT Member
Reactions:
Posts: 1190
Joined: Wed Sep 25, 2019 4:32 pm

Re: Multiple GPU build ideas

Post by StefanR5R » Wed Sep 15, 2021 1:51 am

You can access IPMI via compatible programs locally on this computer without having to attach the BMC's ethernet port.
Edit, example on Linux, to view the onboard sensors: sudo ipmitool sensor

But more convenient is the remote GUI through the BMC's ethernet. Just enter the BMC's IPv4 or IPv6 address into a web browser. By default, the BMC pulls its IPv4 address from your DHCP server if you have one. But you can also configure a static IPv4 address for the BMC in the BIOS (alternatively, in the remote GUI if you found the dynamic address).

The remote GUI requires username and password. On older Supermicro boards, it's ADMIN and ADMIN. On newer boards, it's ADMIN and a string of 10 capital letters which should be printed on one of the stickers on the board. Although the previous owner could have had changed that.

Once you are in the remote GUI, you can also view the BMC's VGA video through a feature called iKVM console with an HTML5 capable web browser, as well as pass keyboard and mouse events to it. This lets you access the BIOS screen on boot, as well as the normal OS desktop after boot.

You actually need IPMI or the remote GUI of the BMC if you want to check many of the onboard hardware sensors (fan speeds, some onboard temperatures, voltages).

The BMC is always on and accessible, as long as the PSU provides 5V standby power to the board. You can use the BMC to power on and power off the rest of the computer.

Server board manufacturers generally require a paid licence for a subset of the BMC features. But Supermicro enables all features (except for some management functions which are useful in larger organizations) without paid license.

Icecold
TAAT Member
Reactions:
Posts: 1294
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold » Wed Sep 15, 2021 6:28 am

That is pretty incredible, thank you for the detailed explanation! Just needing to use a web browser for the remote GUI makes it even better than I expected. I'm not sure why but I had assumed there would be some utility I'd need to download or something.

Should I be concerned with pulling too much power from the PCIe slots? The best configuration from a bus bandwidth perspective would have me running 2-3 cards with unpowered risers and 2 more with powered risers.

StefanR5R
TAAT Member
Reactions:
Posts: 1190
Joined: Wed Sep 25, 2019 4:32 pm

Re: Multiple GPU build ideas

Post by StefanR5R » Wed Sep 15, 2021 1:08 pm

StefanR5R wrote:
Wed Sep 15, 2021 1:51 am
This lets you access the BIOS screen on boot, as well as the normal OS desktop after boot.
PS, that's only true if the BIOS, or the OS respectively, are configured to use the BMC as graphics card. Obviously the BMC doesn't capture video from other video outputs.

Post Reply