That's crazy, I feel the exact opposite. P4's cast feels like the friends you had back in highschool and the adventure just feels like hanging out with them after school. Naoto and Rise have pretty dramatic backstories, but everyone else is really grounded, so the group has a great dynamic and all the girls feel really appealing.
P3 felt like workplace colleagues with their own cliques, and P5 was a bit all over the show. The only ones that mattered to the plot were Makoto and Futaba, so the rest just felt like they were hanging around.
I prefer persona 3 as my overall favorite in the series aswell, but what you say here is valid. I think P3 tells the better overall story and has a cooler setting, but p4 has that small town friend group vibe on lockdown. Both games do what they set out to do very well in my opinion, its just a matter of preference. I liked p4 when i first played it over a decade ago but i wouldnt return back to it (same with p5) but i replayed p3 many many many times over the years and keep returning back because i like the story of that setting.
I was very frustrated with p5, but i think thats because as i kept playing through it i had to go over so, so much fluff that i didnt care about. The story moved extremely slowly, and and most characters i liked felt like a rebrand of already existing character in p3 (Makoto and Mitsuru being extremely similar for example)
I think for this game, persona 4 was the correct choice. Small town mystery is a very good setting for an h-game.