is he romantically in love with her sister? or just toxic brotherly love?

For now to my interpretation (and hope) is that it's a toxic familial love.

It's a toxic familial love. Rezef has never had anyone. He is angry and this is his way of getting "revenge". He has a father who never loved him, a sister who never saved him (although it isn't Cayena's fault), and no mother to look out for him. All of that sort of fucked up with his head. He wanted to originally kill Cayena (based on the first few chapters), but now, he's obsessed.