I must admit, when I first saw your post quoted, it raised my hackles, so I went back to read your original. In fact, your example illustrates your point well, and the whole is well reasoned.
The problem is that the type of situation you describe is rare compared with the number of times people condemn others for behaviour that they deem immoral due to religious prejudice, cultural intolerance or generational bias.
The truth is almost always more complex than is apparent. Another example, somewhat in support of your point, is incest. Some claim that incest is fine for same-sex couples, or if the couple ensures that no pregnancy can take place, since the only logical reason for prohibition of incest is to avoid genetic disorders. However, this claim ignores the typical power dynamics that occur between siblings or parents and children, especially if the older of the pair is male.
My point is that we should avoid simplistic analyses of all kinds. "Whatever you enjoy is fine if it's not hurting anyone" is an overreaction to the kind of bigotry that leads many, whose sexuality does not fit socio-religious norms, to live (and too often die) in fear. Equally, it is simplistic to dismiss the sentiment behind that phrase entirely, merely because some exploit it to justify behaviour that is, in fact, demonstrably harmful, once we dig a little deeper.