Rather than being upset about recalling past memories, he seems more upset that she wouldn't ask more about him