Just about everyone I know has gun(s) and many have concealed carry permits. I just assume that I everyone I come across is carrying and act accordingly. If I happen to run across a lunatic, I at least know I have a chance to defend myself.
There is no reason to be afraid of you're neighbors owning guns. There is reason to be afraid of neighbors who are insane, gun owning or otherwise.