If you run out of vram Iray will fall back to the CPU and not use the GPU for rendering at all. There are tons of tricks you can use to get ram usage down. You can safely assume that 90% of the trouble you will face with vram is from the textures so try to optimize those as best you can. If you can get your hands on Scene Optimizer that can do some of the work for you, most times that and removing any objects not visible to the camera (directly or in reflections) will get your ram usage way down.
Here is an example I did recently- 3 characters, a building, an armored car, lots of "clutter" total vram usage was about 6gb.
You must be registered to see the links
This one also had three characters and a more complex backdrop geometry wise, I got this one down to just over 3gb vram.
You must be registered to see the links
Two tricks I use often to hide low-res textures used on the scene are dust effects and instanced objects. So it sort of truly is "smoke and mirrors"
Depth of Field works really well too since it will blur the objects you reduce the textures resolution on, sometimes to the point of only being able to discern colors and shapes.
Also, keep in mind that sometimes Iray just craps out and kicks over to the CPU when the scene should fit in your vram. When in doubt, re-start Daz and try again.