Multiple GPU build ideas

Hardware upgrades, reuse, repairs, and optimizations. Fire hazards and bleeding edges.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Multiple GPU build ideas

Post by Icecold »

I've been considering for awhile now the most effective way to put a bunch of GPU's into a single machine for BOINC and Folding, and I figured a post here to crowd source ideas could make some sense. I really prefer having machines that are specifically for either GPU or CPU BOINC projects, since it makes it a lot simpler not having to reserve CPU threads to not starve the GPU, etc.

X99 seems to be the the sweet spot in terms of having enough PCIe lanes while also being affordable. A first generation Threadripper is an option too but the motherboards are pretty expensive. There are a decent amount of X99 motherboards that have 4 PCIe 3.0 x16 slots, and a Xeon with the right amount of lanes would be pretty inexpensive. I would imagine it would be possible using bifurcation to make that into something like 8 x8 slots?

I was thinking of using a mining rig frame like this one - https://www.amazon.com/gp/product/B08XJ ... UTF8&psc=1 and then using full size non powered risers like these - https://www.amazon.com/Thermaltake-Gami ... ics&sr=1-5

One concern I did have, though, is that each card could be pulling up to 75 watts through the PCIe slot. Using non powered risers I'm not sure at what point it would risk damaging the motherboard. I don't know of any powered risers, though, that are not PCIe x1 which would cause performance issues.

Basically, I'd be wanting to setup something like this - http://www.gpugrid.net/forum_thread.php?id=5197 but probably with 3070's.
Skillz
Site Admin
Reactions:
Posts: 1854
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz »

Here are the bifurcation risers that guy used.

https://c-payne.com/products/pcie-bifur ... rd-x8x8-3w

How can you tell if the motherboard supports this feature?
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

Woah - that has a 6 pin power connector on it. That should solve the "I hope I don't wreck the motherboard by drawing too much power through the PCIe slots" concern I had.

I'm not sure if the motherboard manual/specs would list bifurcation support or not. Ian&Steve C.'s post mentions most ASRock motherboards support it.

I wonder just how much PCIe bandwidth most projects need and if running the cards at something like PCIe 3.0 x4 would be a bottleneck. Something like this should be able to power 8+ gpu's (but it would need to be 240v) - https://www.parallelminer.com/product/b ... -200-240v/
Last edited by Icecold on Sun Aug 22, 2021 3:34 pm, edited 1 time in total.
Skillz
Site Admin
Reactions:
Posts: 1854
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz »

Its hard to tell, but it looks like he used those to split the x16 slots into two x16 slots (x8, x8) but those x8 slots on the motherboard seem to have a x16 riser on them 1:1. or something.

There are different sizes available of those cards he used for the x16 to two x16 slots.
https://c-payne.com/collections/pcie-x8 ... ation-pcbs

Seems he used a x16 to x16 riser to go to that card, then tilted the card sideways and plugged the GPUs into them directly.

Just noticed your edit.
You can run more than one of those style PSUs to power the GPUs. I already have one of those server PSU's with a breakout board to power my x8, 280X GPU setup I was building. It's 1000W @ 110V.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

It seems like once you get past 5 or 6 cards you're getting fairly close to the limits on a 20 amp 120v outlet depending what type of cards they are. I would like to build a machine with all 7 of my Firepro S9150's and I think they use around 200 watts each. I should check it again with a killawatt to be sure. It seems like if you can use 240v it's definitely worth it as far as better efficiency, higher output, etc.
Skillz
Site Admin
Reactions:
Posts: 1854
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz »

AMD EPYC 7251 8-Core 2.1GHz - $310
https://www.ebay.com/itm/284208651449?e ... X&LH_BIN=1

Asrock Rack Server Motherboard EPYCD8-2T SP3 Socket EPYC CPU - $452
https://www.amazon.com/Asrock-Rack-Serv ... B07PGLF6ZB

Then you'll just need RAM, HDD (or SSD), PSU, those riser cards and a case and you'll be ready for GPUs.

Added in 33 seconds:
Icecold wrote: Sun Aug 22, 2021 3:48 pm It seems like once you get past 5 or 6 cards you're getting fairly close to the limits on a 20 amp 120v outlet depending what type of cards they are. I would like to build a machine with all 7 of my Firepro S9150's and I think they use around 200 watts each. I should check it again with a killawatt to be sure. It seems like if you can use 240v it's definitely worth it as far as better efficiency, higher output, etc.
You'll have to undervolt them like he did. He said his 8-card rig w/ 2070s pulls around 1600 - 1700W from the wall.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

I think I'm going to have an electrician quote me out on running some 240v to my garage. A 30 amp 240v circuit should be able to do some damage in terms of powering some GPU's and I feel like should be pretty reasonable as far as costs go. The heating, dryer, etc. is all natural gas, so I think there should be plenty available.

I think if I were to use Haswell / x99 stuff though it would be much less expensive. A solid X99 motherboard can be had for ~$150, a Xeon with the right number of lanes is probably < $50. If I were building just one of these the $450 motherboard wouldn't be a huge thing, but I'm thinking about building like 4-6 of them.
Skillz
Site Admin
Reactions:
Posts: 1854
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz »

Well hopefully GPUs come way down in cost soon, because 30+ 3070s will cost $30k alone. lol

Only reason I picked that setup specifically is because we know it will work. I don't know what to look for to determine if a motherboard will support that feature or not.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

I already have the GPU's it's just trying to consolidate it vs being a couple in a machine here, there, etc.

It wouldn't all be 3070's, but I have 7 Firepro S9150's that could make one machine, a bunch of Vega 56/64's that could make another, enough 3070's for another machine, maybe a mixed machine with 2080ti's and 2070's. I'm kind of a GPU hoarder :shock:
Skillz
Site Admin
Reactions:
Posts: 1854
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz »

Holy crap. lol

I stopped buying GPUs since the recent Mining "boom" has caused them to increase so much in price. Even the 280x GPUs that I was once picking up for around $70 shipped have gone up over $100 shipped now. So I stopped buying them, but I also ran into an issue where I can't run some of them together in the same rig. They work fine running 1 at a time, but once I add them both together they won't boot. Tried both in more than one rig, same outcome. So now I have to buy the same GPU in order to get them to work together.

Also, those S9150 Firepros are extremely expensive now as well. I was gonna buy a bunch of them and the 3D Printed fan shroud for them on eBay and rock 8 of them in my setup, but they're 100s of dollars now.

For the record, Milkyway@Home doesn't seem to be bottlenecked by x1 lanes. I have a 280X running in a x1 rise card and it doesn't seem to produce any less than the others in x16 slots.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

Yeah the Firepros seem to run fine on x1. I was using x1 powered risers for them for MW and it didn't seem to affect it. They even did 400,000 PPD on Folding@home on the risers, but I'm not sure what they would have done on an x16 slot. I considered selling the Firepros now that they've gone up in value but I probably missed the best time to now that prices have stabilized somewhat. Most of mine I paid ~$100 for, and they are way higher than that now. I do hope we see a time where 2080ti's are going for $400-$500 used again, though.

I wonder if your issue could be resolved by turning on 'above 4g decoding' in your bios. It usually requires a bios update but with that setting turned on it handles multiple cards much better AFAIK
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

I'm not sure how feasible this idea is without spending a ton of money unless I scale down the number of GPU's per machine. It sounds like I could do 6 GPU's at x8 with a 1st gen threadripper. If I'm buying EPYC Rome hardware it somewhat defeats my purpose of wanting a GPU only crunching machine and not juggle CPU projects on it. The only other thought I have is to go dual socket Haswell Xeon, but I'm not sure how well that would work.
User avatar
pututu
TAAT Friend
Reactions:
Posts: 194
Joined: Thu Nov 05, 2020 11:12 pm

Re: Multiple GPU build ideas

Post by pututu »

@Skillz , you should have bought the Asus X79 Pro that came with CPU and RAM from me. You can run 4 PCIE lanes with 1x16 and 3x8. ;)

I've two X99 Asrock motherboards which I only bring them on during challenge since they suck so much power. One is Extreme 4 the other is X99M Fatal1ty.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

@pututu is there a list of performance data for running BOINC projects on PCIe x1 vs x4, etc. that [H] has and would be willing to share? @Skillz mentioned there was somebody named atp1916 that had tested a ton of different projects on mining risers, etc. I tried to PM him over at [H] just to see if there was any rough advice he could provide me but it doesn't appear that his account is still active.
User avatar
pututu
TAAT Friend
Reactions:
Posts: 194
Joined: Thu Nov 05, 2020 11:12 pm

Re: Multiple GPU build ideas

Post by pututu »

@Icecold , I think the list or the info posted by atp1916 may be in the [H]DC private forum strategy which no longer accessible when Kyle migrated to a new web layout or something. Skillz originally set this up. Knowing Kyle he doesn't seem to have any interest in bringing this private forum back. We did contact him via Pocatello, our sole DC forum moderator but no luck.
Skillz
Site Admin
Reactions:
Posts: 1854
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz »

Yes, the information was once posted in the private forums. Can't believe Kyle dropped them like a sack of potatoes. So glad I left that forum.

I wish we could get in touch with atp in hopes that he archived the tests he done or his memory is better than mine.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

That's a bummer, but I appreciate the reply. It would be fairly daunting to try to test each BOINC GPU project at x1, x4, x8, etc. Especially since x4 may not bottleneck a 2070 but potentially would a higher performance card like a 3070 or 3080. I have a strong feeling that in about 2 years there will be a flood of used GPU's sold on the cheap, so figuring out a way to stamp out builds with a large number of GPU's might be a good idea. When eth goes proof of stake and a ton of people sell their GPU's it wouldn't be totally outside the realm of possibility that a 3070 could be $300-$400.
StefanR5R
TAAT Member
Reactions:
Posts: 1661
Joined: Wed Sep 25, 2019 4:32 pm

Re: Multiple GPU build ideas

Post by StefanR5R »

Icecold wrote: Mon Sep 06, 2021 2:55 pm It would be fairly daunting to try to test each BOINC GPU project at x1, x4, x8, etc.
That's not necessary if the PCIe bandwidth usage of many projects was identified before. After that, perform x1/x4/x8 comparison tests at selected projects only.

Though even determining the bus bandwidth usage is not trivial, since it is generally fluctuating. The probing interval should be short (e.g. 1 s, which is the minimum which nvidia-smi for example supports AFAIR), the probing period should be long enough (e.g. one entire task duration in case of uniform tasks; repeated over different tasks in case of tasks with variable performance characteristics). And after the data were gathered, not just an average but more importantly one or more higher percentiles (e.g. 95-percentile and 80-percentile?) should be obtained from the sample.

Remember though that this is not just a per-project thing. Windows and Linux differ substantially due to their different driver stack architectures; Windows requires more bus bandwidth to do the same. (And of course, big GPUs want more bus bandwidth; NV more than AMD, …)

Folding@home is widely regarded to be the project with the most hunger for bus bandwidth. I imagine GPUGrid comes close, because what I heard is that it is performing quite similar types of simulations.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

I appreciate the insight @StefanR5R . That makes sense - I would do an initial run though of all projects, noting their bus usage and then would only need to test the ones at x1, x4, x8 that I've identified as using a lot of bus bandwidth.

I could probably just test my most powerful AMD card and my most powerful Nvidia card. It seems like a relatively safe assumption that if a 3080 isn't bottlenecked at that bus speed that a 2070, 3070, etc wouldn't be as well? Unless there's something I'm not thinking of.
User avatar
biodoc
TAAT Member
Reactions:
Posts: 1014
Joined: Sun Sep 15, 2019 3:22 pm
Location: Massachusetts, USA

Re: Multiple GPU build ideas

Post by biodoc »

@Icecold , you may want to PM @IEC in the other forum. At one point he built a multi-gpu rig in this thread.

More in this thread including some PCIe bandwith data posted by @StefanR5R on his 3-GPU rigs.
crashtech
TAAT Member
Reactions:
Posts: 1544
Joined: Sun Sep 15, 2019 4:45 pm
Location: Idaho, USA

Re: Multiple GPU build ideas

Post by crashtech »

I am reading this with interest but also with a slight hopeless feeling as I have only been able to buy 2 GPUS in the last year...
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

Do AMD cards not report PCIe bus bandwidth? I could have sworn it would show in GPU-Z on Nvidia cards, but I'm testing with an AMD Vega56 right now since it what was already in the PC and it doesn't show.

I think I should be able to test this pretty easily now that I've looked into it a little bit. The first motherboard I checked here has bifurcation support - a Gigabyte B450 Aorus M. I couldn't find a setting to limit a PCIe slot to x4 or 8 BUT I can turn on bifurcation and it does exactly that. The options are 2x8, 1x8/2x4, 2x4/1x8 and 4x4. If I set it to 4x4, even without any bifurcation risers or anything it limits the card in the slot to x4 according to GPU-Z. Testing x1 performance is easy enough just using an x1 riser. This isn't an ideal setup for bifurcation due to limited PCIe lanes in non server or workstation Ryzen, but it should make a decent test bench. It was encouraging that just some random Gigabyte cheap AM4 motherboard has bifurcation support though. This machine is also one I have setup as dual boot Windows and Linux mint, so it should make testing between both pretty simple.

biodoc wrote: Fri Sep 10, 2021 10:33 am @Icecold , you may want to PM @IEC in the other forum. At one point he built a multi-gpu rig in this thread.

More in this thread including some PCIe bandwith data posted by @StefanR5R on his 3-GPU rigs.
I appreciate the information. I may reach out to IEC, and that second thread has a lot of useful information as well. The one thing that may be fairly unique compared to IEC's build though is that an RX470 being a much lower performance card than something like a 2080ti or 3070 may not bottleneck as much on an x1 slot. I would like good Folding@home performance as well in case I go all out on the December race or the one with [H]. Still good info though, thanks!
crashtech wrote: Fri Sep 10, 2021 3:58 pm I am reading this with interest but also with a slight hopeless feeling as I have only been able to buy 2 GPUS in the last year...
It's taken a ton of persistence, and willingness to pay prices above what a 'sane market' would be. I probably made it sound like I have more cards than I do as well, but not intentionally.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

Does anybody know if the 'estimated time per fold' in Folding@home updates as the task runs or is it an average? I'm just wondering if it would be reasonable to start a task, run it for 20-30 minutes and then change the PCIe bus speed, run it for 20-30 minutes and see if the TPF changes.
User avatar
pututu
TAAT Friend
Reactions:
Posts: 194
Joined: Thu Nov 05, 2020 11:12 pm

Re: Multiple GPU build ideas

Post by pututu »

@Icecold , are you sure the ascrock x99 extreme 4 support pcie bifurcation? I'm still thinking if I should retire this old faithful.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

pututu wrote: Fri Sep 10, 2021 10:04 pm @Icecold , are you sure the ascrock x99 extreme 4 support pcie bifurcation? I'm still thinking if I should retire this old faithful.
I am not sure at all :lol: I'm flying blind for the most part, and mostly going off of posts on the bifurcation thread at [H] and this post here- http://www.gpugrid.net/forum_thread.php?id=5197
Ian&Steve C. wrote: you can't do this on most consumer motherboards, but it seems to be well supported on Asrock motherboards (even consumer) and some higher end HEDT boards
If you have it in a currently running(or easy enough to boot up) machine though, it should show as an option in the bios if you check that. I think with the number of PCIe lanes though it would limit a multi GPU build(at PCIe 8x) to 5 or 6 GPU's. I think a 5 or 6 GPU build on a mining rack, though, would be pretty sweet if it wasn't bottlenecked by PCIe bus speed. 5x3070's would do some real damage in a lot of projects. I'm trying to test currently if 4x would be a bottleneck for a lot of projects but am still trying to figure out the best ways to test that.

I'm running F@H on a Vega 56 right now on an X1 riser and it's showing 1,090,757 PPD in Windows. I was getting ~1,204,211 PPD at x4 and x8 in Windows. I'm starting to think the bus speed is much less of an issue than I was expecting but these are very early tests. I would prefer to have as few bottlenecks in all projects as possible, though, so I still think this is something worth exploring. I'm going to test a 3070 next which I think will provide a lot more insight.
User avatar
pututu
TAAT Friend
Reactions:
Posts: 194
Joined: Thu Nov 05, 2020 11:12 pm

Re: Multiple GPU build ideas

Post by pututu »

I can't find anything about bifurcation on the asrock x99 extreme4 in the manual which you can download here: https://download.asrock.com/Manual/X99%20Extreme4.pdf. Can do 3 GPUs full lanes with the xeon E5-2697v3 that I have on the MB at x16, x16 and x8. The other PCIE2 can only do pcie2.0 x 4. There is also a molex connector on the motherboard which you probably need to use if you have more than 2 GPUs, I think.

The system is currently in a closet. I only run it during challenges as the E5-2697v3 is a power hungry chip with max turbo speed of 3.6GHz for up to 2 cores, 3.4GHz for 3 cores, etc.

Let me take a look over this weekend. Shouldn't be that difficult. I think I still have the original BIOS which is version 1.90, IIRC.
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

pututu wrote: Fri Sep 10, 2021 11:06 pm I can't find anything about bifurcation on the asrock x99 extreme4 in the manual which you can download here: https://download.asrock.com/Manual/X99%20Extreme4.pdf. Can do 3 GPUs full lanes with the xeon E5-2697v3 that I have on the MB at x16, x16 and x8. The other PCIE2 can only do pcie2.0 x 4. There is also a molex connector on the motherboard which you probably need to use if you have more than 2 GPUs, I think.

The system is currently in a closet. I only run it during challenges as the E5-2697v3 is a power hungry chip with max turbo speed of 3.6GHz for up to 2 cores, 3.4GHz for 3 cores, etc.

Let me take a look over this weekend. Shouldn't be that difficult. I think I still have the original BIOS which is version 1.90, IIRC.
That seems to be the challenge with bifurcation.. I think it has often been added in as a bios update and just not mentioned. Above 4G decoding seems to be similar - it often shows up in newer bios's without any documentation of it.

I have a bunch of Haswell Xeon stuff already, but none of it really makes sense to use for the GPU build idea. I guess I should go through them and see if any of them support bifurcation, but I have some of those knockoff $75 X99 boards from China, I have a dual Xeon Dell, etc. none of which seem like a good fit.

Those C-Payne bifurcation risers have a 6 pin PCIe power connector, so I think that may negate the need for any additional power to the motherboard(like that molex connector you mentioned)
Skillz
Site Admin
Reactions:
Posts: 1854
Joined: Sun Sep 15, 2019 3:03 pm

Re: Multiple GPU build ideas

Post by Skillz »

From what we have learned with bifurcation. Documentation on it is nearly non-existent. The board could have support for it, but not mention it in the manual. Best way to know 100% for sure is to boot into the BIOS and look for it.

was beat to the reply by @Icecold :lol:
User avatar
pututu
TAAT Friend
Reactions:
Posts: 194
Joined: Thu Nov 05, 2020 11:12 pm

Re: Multiple GPU build ideas

Post by pututu »

@Icecold , @Skillz , Updated Asrock X99 bios to latest 3.80. Navigate from "Advanced" → "Chipset Configuration" menu, I see no option to change the PCIE attributes within the Chipset Configuration menu. Let me know if I missed out something.
Chipset1.png
Chipset2.png
PCIE.png
Icecold
TAAT Member
Reactions:
Posts: 1447
Joined: Thu May 07, 2020 7:46 pm

Re: Multiple GPU build ideas

Post by Icecold »

That should be the correct bios page. Maybe that specific board doesn't support it? :( I did get a message from my WTB post on [H] form somebody that said "I have a handful of full-size AsRock Rack server mb's that are socket 2011v3. They all support bifurcation. " so I'm in the process of trying to buy at least one of those to see how it works out.
Post Reply