I agree that it's not something we should be super concerned about. I personally believe it is possible to increase your odds but not land exactly what you want. Whether possible or not, if your dice rolling habbit involves looking at the dice in your hands as you bounce them around till they're "just right" people are going think you have some secret technique. Find a new technique that doesn't bother people. Same reason I don't wear a shirt that says "I have a bomb" in the airport. It bothers people and you're going to have people judging you, right or wrong.
That's pretty much where I am with this too. Like I said, If you get a bad vibe off of the way someone does something and they aren't agreeable to some kind of compromise, quietly bring it up with the TO and ask them just to keep an eye on them. No need for an official ruling just yet.
But if something fishy does seem to be going on, the TO can handle it from there.