Virginia building code requires a permit for replacement of water heaters. Most likely, your county or city also requires a permit for replacement. Your plumber should know this information, is he a licensed plumber? Without the proper permits, I agree that you may not be able to rent out the property. Best if you take the time and walk the paper work through so that you are legal.
If the plumber did the installation, then the plumber is legally required to pull the permit. An owner should not be pulling a permit for a supposedly licensed plumber who Knows better.
If you hired a license plumber you probably can go after the plumber if he didn't get the proper permits or notify city/county if necessary. Ck with county/city to see about the permits he should have done.
If it were me I'd tell the tenant (after I found out that all the permits were taken care of) that I understand she wants to move and tell her (and put into an agreement) I agree to terminate the lease as of (a date in the spring where it's easier to find Tenants) and consider myself to be lucky to be rid of this tenant She will probably find something else she's unhappy with.