Go to your dashboard > appearance > customize > alerts

You can leave your alert message in the content area, and then once the alert is no longer needed, you can leave the content blank to disable it.