Not only is it unsightly, but it will continue building up and wearing away at the hinges if you don't remove it promptly. Rust can also leave nasty stains running down the surface of your door.