One way would be to use the scene optimizer and scale down the textures but I don't think that's what you are looking for.
Well... If you want to understand the render times, you need to understand how iray works. To put it simple iray calculates every light ray (distance, wave length, reflections,...).
So if you make an outdoor scene with an HDRI and a model or maybe an object the light will travel from the source (HDRI) to the model/object and bounces from there to the camera or in the infinity of your dome. Because it won't bounce much, the light travels a short distance which means fast calculations and low render time.
If you now make an indoor scene where you can see the room as a cube and you place your objects/models, camera and lights in this cube. You will see that the light rays travel from the source(s) to the model/object/walls of the cube and then they will bounce a lot, from wall to wall, from model to wall etc and finally to the camera. The distance and the amount of bounces increase and this results in longer calculation and render times.
The easiest solution would be to delete some walls of the cube preferably the ones behind the camera. A nice tool for this is the "iray section node/plane". This is a infinite plane which deletes every object on one side. Now the light rays can bounce out of the cube which means less bounces and therefore shorter distances to calculate and lower render times. A positive side effect is, that now the light from your HDRI can travel in your cube/room and you will get a nice ambient light.
Another way would be to tell iray how often your light rays should bounce. You can find this in render setting->optimization->max path length. The default is -1 which means for iray something like infinite bounces but this is not 100% correct because after a bunch of bounce iray just stops automatically. Now you can change this and see what happens. I put some examples in the spoiler below.
Hope this helps. If something I said is wrong feel free to correct me.
