I've had this happen to me before, where the package was marked as refused and sent back, and yes, it has to be handled by the person/business that handled the package. in my case the package was sent back to the sender and they charged me for return shipping and shipping back out to me.
One thing you can do is edit your address, that second address line that is never used put in "Hold for Pickup" the item will not make its way to a postal carrier and gets diverted to your local pickup point, in my case a pharmacy that has a postal outlet. Every package that I've had sent since goes directly there and you never have to worry about if the postie just doesn't like you, which i was starting to think was the case.
BTW: "out for delivery" just means that it is with a Postie, or a parcel courier, they can have the package for upto 2 days before they make delivery depending on the weight of the package.