Also, does the ranking/prestige of the starting firm matter in terms of lateralling
I rarely type long posts either because I don't have time, or because I don't give enough of a shit, or because I find so much wrong in a post that it's not even worth it. But the coinciding of insomnia and boredom has made this day, of all days, your lucky one. Here goes...
1) Experience matters a lot more. If you're getting the kind of experience that would nurture skills transferrable and valuable to the kind of litigation a V5 or a V10 firm does, and you're getting good experience relative to your class year, then you're fine for really almost any but the most truly selective firms (i.e., WLRK or something like that).
2) You honestly might have just dodged a bullet. I would seriously give your summer firm a chance and get this "prestige" notion out of your head. The very concept is useful for making ignorant law students work egregious hours for less hourly pay (divide your market salary over the course of how many hours you bill in a year, and you'll see what I mean) and your partnership prospects might be considerably higher. In addition, while top firms might do more interesting cases (and this is not really that true), a lot of them are massive litigations with armies of lawyers. It's very reasonable that the only skill you'll own in your 4-5 years at that firm is how to conduct and manage doc review, which would ironically make you less marketable than as an associate with more substantive experience in the area at a "lower ranked" firm. This isn't to say that V10 lit is all doc review, but it is to say that just because the case is "more interesting," doesn't mean it's better.
3) There are plenty of litigation firms that do extremely interesting cases and where you can get a lot of great experience, but aren't necessarily V10s (or even ranked, for that matter, such as W&C, Susman, Boies, Munger, etc.)
4) If you're stuck on lit and you believe your grades were good enough for a V5-V10 (whatever that means; they're not that selective), you should consider clerking. A good clerkship will give you a good shot at having a second shot at some selective firms, and some that don't fall within the V5, V10 order of thinks (think W&C, Susman, Boies, Munger, etc.).
5) If you really do have the greats for a "V5" (again, whatever that means), think about doing a shit-ton of mock interviews. If you think you batted below average despite your grades, odds are your personality and/or interviewing skills are shit. Work on those pronto, because if they continue to be shit you will flunk out of 3L OCI, clerkship interviews and lateral interviews.