I voted Knoxville. Never been to Athens, but a lot of my good friends went to UG there and loved it.
Knoxville, though, seems to be trying hard to improve, and doing a decent job of it. Every time I return it seems a little better.
Also, as someone who, like OP, grew up in East Tenn., I absolutely thought that D.C. was giant city when I lived there.
(Rankings, Profiles, Tuition, Student Life, . . . )