he was shocked because Yoo Won was initiated the hand holding in PUBLIC. He was bold here since their relationship isn't exactly public
I suppose that could be it as well, it's just that from the way Yoo Won and Hee Bum had been acting, it seemed to me that Yoo Won knew about Jumi's feelings for Hee Bum and knew what Hee Bum was about to have to go through to confront her. There were quite a few meaningful silent glances between them and Yoo Won left first which made me think that it was something they both knew Hee Bum would have to do. Yoo Won left to give Hee Bum privacy with Jumi to confront her and i think that is why he looked down while holding hands with Yoo Won, because he felt bad for Jumi and wasn't looking forward to what he was about to have to do. At the very least Hee Bum knew what he was about to do because he just kind of sat there brooding about it for a second before confronting her.
Could well be that and a combination of things. In fact, when I first saw it, I thought it was that--the shock he was showing affection in a crowded place. But then with this chapter, I wonder if it's a feeling of surprise at the touch in public plus a feeling of disgruntlement that he wasn't talking to him but to Jumi, plus a feeling of discomfort at knowing Jumi's feelings and what's going on behind her back. Complexities.

I think Hee Bum was harsh on her so that she wouldn't have false hope guys Σ(っ°Д °;)っ