Why does it seem like most of the people who are against weed prohibition already smoke weed?

I don't mind decriminalizing it, but I'd rather not look like a f__king moron by ociating with doped up liberals who think that minor laws against marijuana is the worst evil that has ever been done in human history.