It really depend on what it is.
Unit patch is one folks should not wear unless they're in milsim game
Airborne, Special force are two comes to mind. There are others just as nice but don't offend others ie: Infidel
In terms of rank, same thing. if you don't earn it at very least position it so people can tell, ie wear Captain patch on its side (equal sign)