It is no surprise that cisgendered accuse transgendered of thinking the same way they do, while simultaneously asserting that we're insane. This is because of the inherent insanity required of them to think of some of the disgusting things the cisgendered are so quick to accuse us of doing.
Bathroom arrangements are based on a very arbitrary need to suppress women, and civilized countries are beginning to remedy this by eliminating all segregation. First, what is a bathroom for?
If you are thinking of anything more than eliminating waste and perpetuating sanitation then you are probably a pervert. Nothing wrong with perverts as long as they don't assume everyone else thinks the same way.
Thus the reason that people who assume transgendered are there to "peep" or some other horrible act are actually quite disgustingly scary. They are speaking out of jealousy, believing it's unfair for someone else to have the opportunity they would use for such heinous acts.
The reality is that transgendered people are less likely to sexually harass anyone for many reasons, the first being basic fear of prosecution. Knowing how we are treated by society makes us too frightened to commit any actual crimes, especially ones that make others look bad.
Another reason we are inherently safer is because we don't want to make anyone else suffer, we have suffered enough. Causing suffering after going through what we go through would bring us to tears, empathy is like that, and most people against transgendered people's rights are lacking in empathy.
For a fun bonus, all religions paint transgendered people as more divine than the cisgendered. One of the reasons religious people are so violent toward us, we do what they fear, we live more pure based on their mythology.