Help! with my anger around this perve/ peado?

My temper is not a bad one at all, I do loose it but bever badly. This all started two years ago when this man used to watch me and a friend in this park at school lunch breaks when I was 15, He used to stare at us obsessively but it didn't bother me cos he never did anything and he never followed us. Ever since he used to do that, for the past 2 years I have had the odd time where I have seen him, and again he would just watch and I was always with mates so it didn't bother me but I have seen him alot. But about 4 months ago I was in a public toilet, as I looked in the mirror I saw his head sticking out from under the cubicle and staring at me/ boys in the urinal. He instantly noticedI saw him and so came out and followed me out of the toilets, this freaked me out as now I was on my own. The worst part is that as he walked fairly close he stared at my groin constantly which angered me badly and I quietly swore at him and he noticed and went away. Today I saw him again, I was walking up the road to my local leisure centre when I saw him, he was really close before it hit me who he was cos I was in a world of my own, as I looked up into his eyes they were black and he looks like the kind of freak to see in a horror film, he stared at me again, checking every part of my body out, I could literally feel my heart beat faster, I could even feel adrenaline, my fists were clenched and I wasn't scared but I now wanted to hit him so badly, I felt unstoppable and as if I could have destroyed him in a single blow, but instead I controlled it and walked past, he stared at me as I walked away, i continued to feel this angry for about 5 mins until I calmed, the problem i have is if I'm with my younger brother and he checks him out I will loose that self control and go in fists first so please help?