They're in a temple/dungeon that isn't filled with water from what I remember. That stream you see in the middle of the corridors is the only water in the dungeon, I believe. Here's one solid look at what I'm talking about:
http://www.mangago.zone/read-manga/isekai_yururi_kikou_kosodateshinagara_boukensha_shimasu/mdx/mpn_chapter-2423767/pg-18/
Gotcha! That makes more sense
Yeah. I just remembered that in the last chapter, they had discovered that there was air in an underwater dungeon (or labyrinth, as stated in the manga). And that is the one they are currently exploring. ^-^
http://www.mangago.zone/read-manga/isekai_yururi_kikou_kosodateshinagara_boukensha_shimasu/mdx/mpn_chapter-2400810/pg-12/
Who wonders how they're cooking underwater?