Top Law Schools

I’m currently a junior (read first through third year) in a firm in the V50 and am thinking about lateraling for various reasons. Some of the firms I’m interested in are midlaw firms or boutiques. Would it be a bad idea for a junior to try to lateral to these kinds of firms instead of another biglaw firm? Basically, I’m wondering whether a move like this would hurt my career (i.e. whether it would make it harder to lateral later, go in house, or go gov). Any advice is appreciated.