HowTo 2x 2080TI with NVLink and memory pooling in DAZ

[PandaPenguin]

2x 2080TI with NVLink and memory pooling

so I got my 2nd 2080TI today and the standard RTX NVLink bridge from Nvidia (3slot)

after assembling everything I followed these steps to enable NVLink on my two 2080TI in Windows 10

You must be registered to see the links

the basic setup tested positive for a working NVLink connection and I proceeded with adjusting DAZ for the Peer Group Size of 2
(Puget also mentioned that both GPU need a display attached to establish the working NVLink)

then I proceeded with testing:
I have a scene with 5 characters and a lot of clutter in it that failed in the past on a single 2080TI
so for good measure I first tested if it was still failing to CPU on the current DAZ version... and it did
(1.7 GiB geometry and 4.6 GiB textures)

rendering the very same scene (after a DAZ reboot to flush everything) with active NVLink, DAZ sucessfully rendered the scene

Iray distributed the textures evenly on both GPUs!

another test with a way smaller scene resulted in the same evenly distribution of textures across both GPU on NVLink and a mirrored distribution without NVLink

every other vram consumption (geometry, lights, ect) still gets mirrored onto both GPU so, as stated in the patch notes, only textures benefit from NVLink memory pooling so far

System Details:
Windows 10 Pro x64 - Version 1909
Nvidia Studio Ready Driver 442.92
DAZ Studio Pro 4.12.1.117
GPU0 INNO3D RTX 2080TI iChill Frostbite
GPU1 INNO3D RTX 2080TI iChill Frostbite
Nvidia RTX NVLink Bridge (3Slot)
ASRock X299 Extreme4
Intel i7-7800X
64GB DDR4 Crucial Ballistix Elite 2400
Samsung 960 EVO 250GB NVMe SSD

 

[Deleted member 1121028]

2x RTX 2080 ti, oh boy that's something I wish I could try .
I guess geometries 'pooling' is way more complex than textures?
Ligths gain should be marginal outside really uber-complex scene.
You could also use a 3rd shity card just to use as display .

With this you should go crazy and

You must be registered to see the links

the heaviest texturing.xyz skins on a g8f model just for fun.

 

[PandaPenguin]

You could also use a 3rd shity card just to use as display

according to Puget System there is no way around the GPUs having a display attached (real or fake EDID) for NVLink to register properly

so a 3rd GPU for display is not gonna do anything.... also I wouldn't have any more space to add a 3rd GPU on this motherboard anyway

 

[rodneyeatme]

2x 2080TI with NVLink and memory pooling

so I got my 2nd 2080TI today and the standard RTX NVLink bridge from Nvidia (3slot)

after assembling everything I followed these steps to enable NVLink on my two 2080TI in Windows 10

You must be registered to see the links

the basic setup tested positive for a working NVLink connection and I proceeded with adjusting DAZ for the Peer Group Size of 2
(Puget also mentioned that both GPU need a display attached to establish the working NVLink)

then I proceeded with testing:
I have a scene with 5 characters and a lot of clutter in it that failed in the past on a single 2080TI
so for good measure I first tested if it was still failing to CPU on the current DAZ version... and it did
(1.7 GiB geometry and 4.6 GiB textures)

rendering the very same scene (after a DAZ reboot to flush everything) with active NVLink, DAZ sucessfully rendered the scene

Iray distributed the textures evenly on both GPUs!

another test with a way smaller scene resulted in the same evenly distribution of textures across both GPU on NVLink and a mirrored distribution without NVLink

every other vram consumption (geometry, lights, ect) still gets mirrored onto both GPU so, as stated in the patch notes, only textures benefit from NVLink memory pooling so far

System Details:
Windows 10 Pro x64 - Version 1909
Nvidia Studio Ready Driver 442.92
DAZ Studio Pro 4.12.1.117
GPU0 INNO3D RTX 2080TI iChill Frostbite
GPU1 INNO3D RTX 2080TI iChill Frostbite
Nvidia RTX NVLink Bridge (3Slot)
ASRock X299 Extreme4
Intel i7-7800X
64GB DDR4 Crucial Ballistix Elite 2400
Samsung 960 EVO 250GB NVMe SSD

That's awesome. I looked about six months ago or so and everyone was all, "Well, I guess it should work but.."

 

[PandaPenguin]

That's awesome. I looked about six months ago or so and everyone was all, "Well, I guess it should work but.."

ikr.... but it is only working since DAZ updated to Iray RTX 2020 in the version 4.12.1.76.
the current release version is 4.12.1.117 at the time of testing

this should also work with 2070S, 2080 and 2080S

 

[FeaturedOn]

according to Puget System there is no way around the GPUs having a display attached (real or fake EDID) for NVLink to register properly

so a 3rd GPU for display is not gonna do anything.... also I wouldn't have any more space to add a 3rd GPU on this motherboard anyway

That's true that every GeForce card defaults to WDDM, but you may be able to try TCC mode to enhance the render time and reduce the video-RAM that's allotted to Windows. Titans, Quadros, & Teslas can switch to TCC mode, easily, for headless operation. Usually frees up 1 to 2gb of vRAM for jobs.

 

[PandaPenguin]

That's true that every GeForce card defaults to WDDM, but you may be able to try TCC mode to enhance the render time and reduce the video-RAM that's allotted to Windows. Titans, Quadros, & Teslas can switch to TCC mode, easily, for headless operation. Usually frees up 1 to 2gb of vRAM for jobs.

I guess you didn't read it all

there is no TCC for GeForce cards in Windows and even if there was, you still have to have a display attached to both GPUs to make this work ... which will put you in WDDM

 

[FeaturedOn]

We were already agreeing. Since you can't run headless, and can't enforce/trick TCC you lose the tiniest bit of RAM.

I run my Titans headless, in TCC, & windows on my render box, and didn't know if I could offer any advice.

 

[osanaiko]

I love both of you (no homo) PandaPenguin and FeaturedOn: Panda for bringing this new info I never considered, and Featured because of his charity renderfarm.

 

[i11uminati Productions]

With 2080ti's getting cheap I am thinking of getting a second with an nvlink. I suspect this would be comparable to a single 3090 in both rendering times and vram (22 vs 24). Also cheaper since selling my existing card for 500 and paying 1500 comes out to 1000 out of pocket but adding a second 2080ti plus nvlink would be quite a bit less. Also, it looks like the 3090 will be very hard to obtain.

My question for anyone using nvlink with Daz is how close to double the single card performance are you getting? From the Daz forums I understand not all aspects of the scene use the shared vram pool and that rendering isn't actually twice as fast. Mainly I want to know how close to the virtual 22 can you realistically get?

 

[PandaPenguin]

With 2080ti's getting cheap I am thinking of getting a second with an nvlink. I suspect this would be comparable to a single 3090 in both rendering times and vram (22 vs 24). Also cheaper since selling my existing card for 500 and paying 1500 comes out to 1000 out of pocket but adding a second 2080ti plus nvlink would be quite a bit less. Also, it looks like the 3090 will be very hard to obtain.

My question for anyone using nvlink with Daz is how close to double the single card performance are you getting? From the Daz forums I understand not all aspects of the scene use the shared vram pool and that rendering isn't actually twice as fast. Mainly I want to know how close to the virtual 22 can you realistically get?

you don't get 22GB, its about 18GB at best, depending how much Windows will reserve for itself

but rendering speed is twice as fast since you use twice the CUDA... and that is NVLink independent, since NVLink has no other performance boost than memory pooling for vram (only texture memory anyway)

dont know about your used market situation, but its a pita to get the correct NVLink bridge here since they are no longer produced (RTX 3090 comes with a new generation of NVL)

 

[i11uminati Productions]

you don't get 22GB, its about 18GB at best, depending how much Windows will reserve for itself

but rendering speed is twice as fast since you use twice the CUDA... and that is NVLink independent, since NVLink has no other performance boost than memory pooling for vram (only texture memory anyway)

dont know about your used market situation, but its a pita to get the correct NVLink bridge here since they are no longer produced (RTX 3090 comes with a new generation of NVL)

Thanks for your response Panda. Great information. So would you still do that setup today if you hadn't already done it? Looks like I can still buy an EVGA nvlink locally for the 2000 series gpu's. I have a blower 2080ti and plan to get the second as fan cooled. Does that make a difference for the bridge?

 

[PandaPenguin]

be very careful with the bridge you grab!
and both GPU must have the same design or you wont be able to connect them (NVLink connectors in different positions)
the blower GPU should be a "reference design PCB" so if the 2nd GPU is too, they can be linked (check this list for "reference PCB"

You must be registered to see the links

)

 

[i11uminati Productions]

Okay just to make sure I've got this correct before I make a stupid mistake:
1) If I want a non-blower card this time it must be reference card shape because those big beefy 3 fan cards won't line up correctly.
2) There are separate NVLink devices for reference cards so make sure it is specifically for them.
3) Look at my motherboard and see the positioning of the new card to count whether there will be 1 or 2 unused slots between them.

 

[FeaturedOn]

That and to be sure that your motherboard supports x16 on two Pcie slots. Which two it supports will help you know the exact spacing of the NVLink. The specific reason for your cards being matched pairs is because they'll be sharing data from memory directly between themselves and having the same memory modules, speed, etc. is essential.

 

[PandaPenguin]

they'll be sharing data from memory directly between themselves and having the same memory modules, speed, etc. is essential.

the first part is correct, the second, not so much but it would ofc not hurt to have the same exact model twice
the reason why they need to be the same design (not model) is the position of the NVLink connectors on the PCB!
the NVLink bridges are rigid (not like those flexible SLI connectors from the past) so even a slight position mismatch will get you into hot waters

and I can tell you from personal experience, not even with the exact same model you have a garuanteed position match
this is a shot of my two INNO3D iCHILL Frostbite 2080TI....
in two revisions they changed the position of the connectors buy just a few millimeter but that was enough to not be able to use the NVLink bridge... I had to gut the thing, grind off a millimeter of the ends and use the bare PCB without the plastic shroud

 

[gamersglory]

2 RTX 3090's in NVlink FTW!!!!

 

[megaplayboy10k]

Gonna need a big motherboard to fit two 3090s in there...those cards look huge.

 

[I'm Not Thea Lundgren!]

Gonna need a big motherboard to fit two 3090s in there...those cards look huge.

Not if they're water-cooled

 

[gamersglory]

The 3090 Kingpin comes with a 360mm AIO