My brother and his girlfriend know I'm into loli. I go to conventions with them and show them the lolicon doujinshi and pervy figures I buy, and answer questions she has about moe. Heh...so at least I can be a bit open about it, even if people don't share my interest. As long as people understand, and accept you for it...that's good enough I think.
I think my mom realizes too. She once asked me, "Are you some kind of sexual deviant?" Oh, and, "Do you know whose panties these are?" (I don't have any sisters) We're a laidback family...^_^"