What determines which portrait/perception is used? After the Fairhaven effect wears off, I've had it flip back and forth for seemingly no reason.
That portrait is how you are perceived by others.
It changes at certain points, like your femininity being high enough, wearing female clothing, breast and butt getting to a certain size, hair long enough, etc. Many of these can change it by itself if it gets high enough, but it will also change with different combinations of them. Each feature has a point value attatched to it, and when the combination gets high enough it will show a woman.
For example: Your wearing mens or low fem clothing, your fem is below 40, you have average butt and AA or A cup breasts, and chin length hair you will (most likely) see if man, but if any of your physical features increase, or you put on more feminine clothing, it will change to a female.