He can't because, thanks to the fact that he has the winter magician's heart, chef can withstand the cold. But the winter magician's can't bear the heat. And the desert even during winter is still hot, at least according to our world. Not sure this is right though
The Winter Magician would visits Chef's castle instead like 3 months too?