Special Education students are awesome man! honestly, I would stay his friend, I think it's really cool that you took the time to make friends with a person like that.
If he really is preventing you from doing anything else and the situation is that delicate, just go talk with the school worker that is working with him. Tell them what is going on and that you still want to be friends with him but you also need some time to do stuff for yourself. I'm sure they will work with you to figure out something that works.
All the kids I've ever met with down syndrome, Autism or the like have been pretty awesome people.
Your life really doesn't look that hard to me, you can articulate well, you can communicate effectively with others.
I know it seems embarrassing for him to come up and say HI to you or whatever else is going on. I'm sure a few people giggle at you or whatever. But you know what, they are just children and rather superficial.
I have a lot of respect for people willing to go the extra mile in situations like this, even if it is inconvenient.
Say hello to him when you see him, maybe even include him in some of the game you do. It might seem like you will lose some cool points at this stage of your life. But trust me, it will pay off a hundred fold when you see people will have a lot of respect for you. And the only people who give you a hard time are just fucking ass holes.
