We've all seen it, even if it is often said in jest. "You can't say 'oh my god!' you're an atheist!". As if our disbelief in deities precludes us from using any religious language that has percolated it's way into popular vocabulary.
It's a silly idea, but the more you think about it, the sillier it gets. Christians are commonly told not to take their lord's name in vein. I get chastised sometimes for exclaiming 'Jesus Christ!' when I'm surprised or startled, because it's disrespectful. If doing such a thing is indeed disrespectful... then are we not, in fact, the very people who should be using such exclamations? We don't believe your god even exists, let alone that he needs to be accorded respect. When I say 'Oh my god!' in a disrespectful manner, this fits in perfectly with my worldview that religious ideas are silly and should be abandoned.
When you tell me that as an atheist, I'm not allowed to use such utterances, you're implying that you, as a Christian, are allowed to do so. And in doing so, you're disrespecting the very deity that you profess to love and worship.
I say, you, as a believer, are the one who isn't allowed to take the lord's name in vain. Me? I disrespect gods all the time. It's kinda what I do. Makes perfect sense that I'd do so in the arena of expletives as well.