Because he is a monster, if he spends too much time in a dungeon, he becomes a part of that dungeon. If he becomes too strong the assimilation happens more quickly.
I believe OnionSenpai is on the right track. I also suspect that the lower percentage of assimilation the less he can "reconnect" with his original existence (or life #1). I suspect he loses memories as well because it has been a while since he has mentioned Mistress Succubus.
I don't get this assimilation thing?