It's a sport, and it requires quickness, hand eye coordination, and a lot of practice to be really good. It doesn't require one to be a great athlete, for sure, but there's no requirement in the Olympics for sports to require that. Hell, Golf is going to be an Olympic sport in Rio, and it doesn't require awesome athleticism either.
We can agree on that!