I think it's not that he can't remember faces rather than he can't clearly remember the whole face of his exes? like he can remember their lips, nose or eyes as something distinctive or linked to each individual but when he tries to recall the whole face he comes up with a generic face?
try to remember the whole face of someone you haven't seen in a long time. it's actually hard and you may inmediatly remember only a part of their faces (like their profile) before you're able to come up with the whole picture.
sorry for the long answer but I guess it's something like that?
lmao forget about this metaporical explanation. I just read Taku has prosopagnosia but recognises Haesoo face for plot purposes
Explains why Taku stares at people like that, imagine not being able to remember faces. Sounds scary...But I mean who could forget Heasoo's face. He's gorgeous.