Ah, but what about an AI using them for terrible terrible things?!
Kidding aside, humans will certainly use them for terrible, terrible things. We already have unmanned machines in warfare, destroying, spying, stealing and killing. That will never end. If it can be weaponized, it will be weaponized. Like you said, robots are not something to be afraid of, it is the people controlling them and what they are capable of. It works the same for everything else; guns, blades, nukes, biological weapons, vehicles, etc, etc.
Consider the positive applications though. For example, that one that throws the cinder blocks would be a great asset in Nepal right now.