In my humble opinion, I think this, as everything that refers to personal preferences, is something we are born with.
Ok, not totally, but maybe a tendence to lean into that direction. In my case, I've found that I like loli because of my own "perfect woman" stereotype: Short, long haired, small tits.
And OF COURSE the moe factor is very important. In loli you find an innocence and cuteness you won't find in older girls.
So, wrapping this up: Either you don't like older girls 'cause a) you've found out they're full of issues, b) the ones you know were a bad experience for you, or c) you are indeed a lolicon perv. If that's the case, welcome to the club xD