With that way of thinking, then only those who go to the stadium should care about the result of the match and those who watch on TV should not.
I think it's simpler, when you like something it generates enthusiasm and when you're not liking the result you're disappointed, the annoyance is more than justified and finding something to complain about is normal, people have to vent in some way.
It is also productive since criticism helps to improve or realize what works and what doesn't or simply the degree of satisfaction with the game.