I don't think it's a filler story. If I'm not wrong, the sword emperor once told us what Gongja need first is love rather than skill (it's in the murim arc). Because Gongja finally knows love, finally have someone he really cares about, he finally saw his life as precious and so he doesn't immediately end himself whenever he wants.
Let's imagine if Gongja never meet raviel, he will, of course, going to end himself whenever the chance happens. Gongja just going to become the always self-sacrifice gongja, he won't have someone waiting for him to come home, he won't think a second thought to end himself since he never meets Raviel and make a promise to her. And if I'm not wrong (again) Gongja once said he probably would end up like the constellation killer if he never meets Raviel. Just someone who life for nothing.
So in short the romance arc is necessary for his mentality.
I always Heard about how romance ruin an action manhwa. Had read a lot of action with romance that blended so well that it enrich the manhwa, so it never brother me. But.... Don't get me wrong, the romance arc sure good. If it a seperated manhwa i would rate it 5/5. Yet, how it corporated in the story seems mismatched. The wife sure loveable, she's strong and smart. And super beautiful to the top. It just feel like "filler" story that could be removed and nothing change. I wish what i felt wrong. Yeah, now i kinda understand why they said it kinda ruin things.