But how clearly do you remember a face you saw for two weeks twenty years ago? I don't think deja vu would be strong enough to make your mind jump to possibility of time travel.
If it was just some random person that you saw around, then you would be right. However, given the types of interaction, their memory/memories of the MC especially, would be incredibly strong/unforgettable.
It's the apparent lack of any form of recognition of the fact that feels weird/unnatural.
Ultimately, the MC is probably going to end up becoming his own father, at which point the likeness would be far easier to dismiss, but that isn't the case in the current timeline. Therefore, you'd expect comments/suspicions/questions varying from simple paternity of the MC, all the way up to the crazy, time travel (even just as a sarcastic/jokey comment).
I guess it could be argued that such interactions do or have occurred when the MC isn't present, but you'd still expect to overhear the odd conversation/hear the odd rumour.
I guess it could also be argued that such things do or have occurred, but the MC and Kim are unaware, as they never lived that timeline, which raises a number of other questions/issues (e.g. what happened to the MC/Kim of that timeline?).
Time travel is a fun, but incredibly complex concept with multiple differing theories.
