There's always a psychological aspect to games like this, where it's the sister who has the feelings first. Whenever it's the brother in games, usually he's just a perv who wants to bang his family and has no real emotions behind it.
While it's true that Nikki probably harbors romantic feelings for her brother, the one thing she never accepts is that it's possible to live those out. She uses her "training" of him as a way of letting out some of those desires, but once she sees her Brother's chance to be happy with someone else, she backs off and tries to find her own happiness else where. She looks for a guy like her brother, because she already knows she can fall in love with someone like that.
Would I like it if she were able to come to terms and try to find some way to live that incorporates her feelings into her happiness? Yes. However, Nikki follows her own logic and basic common sense. She knows they can have some sex and emotional experiences when it's just the two of them alone in a room. But out in the wide world, there's no way it can happen without the acceptance of others. Acceptance that's she's determined will never be given.