It’s just not in their character to stick to him just because he’s weaker though. It’s that situation where they admire his drive and if he ever was someone they had to coddle, they would more than likely lose interest altogether. This is just the tip of the iceberg in terms of depending their relationship. That’s why them refusing to go has as much to do with hoping he gets out of his slump, than it does with them feeling too bored and uninterested in the mission to bother going.
They only started doing that when they think his strength is worth or close to them tho. Those guys don't have drive because they r too good. They r the personification of Enui, lmao. And Encrid provides that fun they're seeking. Now that Encrid isn't showing potential, they r loosing interest. That's why they can't be bothered to go

It's crazy they are staying behind without knowing about his regressing. he could die and not comeback guys??? Hello??? lol. Guess they realized he improves a lot whenever there is a "life or death" situation