I've visited both and went with UVA.
First and foremost, cost should be your first and primary consideration. If you're picking at all based on anything else when costs are unequal, you're doing it wrong.
Assuming costs are completely equal, the towns are different in a few ways, but the differences are not so substantial as to make one a clearly better option in QOL. They are both really fun college towns. AA will be more fun in the fall because college football basically consumes the town and creates a really cool energy that UVA will never really match because the football is pretty abysmal most of the time. The people around both towns were nice and very obsessed with their university because it's basically the entire reason the town functions and thrives. I felt the foodie scene in Cville was a bit better than AA, but that's also a bit unfair because I've now had time to really explore Cville, whereas my experiences in AA are pretty limited. AA feels a bit bigger and more bustling than Cville, and the night life at AA felt a little more robust too, but I mostly chalk that up to Cville nightlife being heavily concentrated in two distinct parts of the city (the Corner and Downtown mall) that made the former feel very UG heavy and the latter being a bit more for the older, less interested in club scene type nightlife.
I'd say, based on my entirely limited experience at
Michigan Law, that the students were laid back, genuinely nice people. I got a
slightly more vibe of people going out of their way to tell me how great Michigan was (and this is entirely anecdotal), and it came off a bit weird to me. UVA preaches their collegiality, students were/are really laid back and genuinely nice as well, but when I visited (and also now while attending), UVA students don't really worry about where they place in the world. They just love being here (and I'd imagine a Michigan student would say the exact same thing, FWIW). Also pretty important is the actual school's location. Michigan is in the heart of the campus, so you'll feel much more like a part of the entire university community. At UVA, you're pretty distinctly separated on North Grounds with Darden. I personally prefer that because it really allows the law students to select out of dealing with UGs if you want to, because UGs become pretty insufferable pretty quickly as a graduate student. But this is entirely a personal preference thing, and if you want to feel a more substantive connection with the parent university, Michigan is unquestionably going to give you more of that.
You're right that the "fratty" thing is way overblown around here regarding UVA, but that is what it is. If you didn't get the fratty vibe when you visited, it wasn't because you just weren't seeing it. There will be folks who go out a lot, play a ton of softball, and maybe act slightly like they're former fraternity/sorority types, but there's also significant parts of the student population that are the exact opposite. UVA can be whatever you want it to be (and I'd imagine this is also true of Michigan).
I'd caution making a decision based on any of the above reasons other than cost. These are peer institutions that substantially place their graduates into similar outcomes (UVA places more in BL+FC pretty consistently, but there's a pretty pointless discussion about actual placement power and self-selection here. Just assume they have substantially similar ability to put you into the type of job you want).
). In fact, theres a few fraternity/sorority houses SOMEWHAT near the law school (DDD and SigEp). But, those houses are always pretty quiet. So be wary of finding housing there. There's almost no reason for undergrads to walk through the law quad besides taking pictures etc... For nightlife, the more grad area is State St/Main st. versus south U.