And never washed it or decided to donate it/throw it away because of the painful memories it might cause.Plot! Or, thinking more rationally, he does say he never wore it again after that time with her.
Obviously plot requires that he read the note now but the entire idea of a dirty shirt hanging in a closet for two years and then, when you’ve finally started to turn it around, putting it on (despite any memories) without washing it is mildly disgusting. I mean didn’t James even remark that it smells of her?