Bella's boss sends Bella home not because she was talking to her husband, but because they were about to get intimate, and that's not appropriate in a library.
I agree with you about the "depth of field." But that rules out your theory from the start. Why would Isabella have a hallucination/projection of James and at the same time insert the man with the glasses into her hallucination/projection?
That doesn't make any sense.
Let's assume that what DPC shows us in the flashback of Isabella James is what it is. A man is visiting his wife at work. Both are about to do something naughty in a library where there is supposed to be dead silence. The boss sends Bella and James home and Isabella's stalker is already waiting outside the library.