illness only hat makers have?... what?

Mercury poisoning. There were a lot of mercury derivatives that used to be used in hat making. The Mad Hatter's disease.

ooooooooh that one
it just sounded so funny in the context of this manga, like "shi, i'm ill, let's go alone to the island"

It really does, but that's kind of what people did. If you were sick you went to the coast or the countryside cause the cities were so dirty. XD