Chess would be a game, but an exclusively mental game (of course, some amount of forearm strength is required to move the pieces
)
Basically, we can classify along three orthogonal dimensions.
1. Games vs. Sports (based on whether there is an active adversary who interferes with your actions) (e.g. tennis vs. sprinting)
2. Physical vs. Mental (e.g. tennis vs. chess)
3. Individual vs. Team (e.g. tennis vs. basket ball).
In order to make comparisons meaningful, I think we should restrict to "Individual Physical Games".
p.s. Sure, I agree that athleticism is involved in both sports and games.