Why do some people like to try to make a quiet and calm person angry?

I notice this a lot actually. That if you are the sort of person who is quiet and keeps to yourself more people flock to you but only to try to rile you up. It could be that deep down I'm just short tempered and they mean no harm but I notice this a lot. What is so exciting about making a person who is usually quiet and serene lash out? The thing is I really don't want to lash out because I know I will get physical and hurt someone. I know I will so I try so hard to keep my cool and act like I don't care. Do I sound like I should seek anger management? I never get violent but the thoughts are there and I get so angry so easily. I cannot take a teasing joke. There's only three people I allow this from-all female and all related to me. It's like I have this wall and if people try to get cozy with me I feel violated and extremely offended.