I think you've hit upon an interesting distinction. But isn't it more of a spectrum than a simple either/or?

For instance, I think most people would accept that baseball and football are sports -- yet they have umpires and referees making judgment calls.

On the other hand, Scott McCloud, in _Understanding Comics_, proposes a theory of Art that definitely includes most Sport. IIRC, "Art is any activity not directly related to survival or reproduction." Rather broad, but I often find it useful.