Crossing the Hate Speech Line

Our constitution protects freedom of speech. Is that line crossed when it includes physical violence, or when it hurts someones feelings? Can I sue my neighbor for calling me "Fag", or do I have to wait until he hits me over the head with a shovel?