Some months back I went from a relatively small company of 500 employees to a super-large Fortune 100. A month and a half later, I went to a company which is even smaller than the first company. In this list, the first two points hit home for me.
http://www.forbes.com/sites/ericjackson/2011/12/14/top-ten-reasons-why-large-companies-fail-to-keep-their-best-talent/
At my current organization, the title of point 7 is right on the money. We have some really sharp folks doing really cool stuff and that makes me happy. Odd as it sounds, it's always great to be around smart people, hoping it'll eventually rub off on you. Being in a large company also makes you feel smaller. Being a security practitioner makes me a control freak by design so I'd like a position where I have more influence, hence the move I made from the Fortune 100.