It seems with that clause that you would be in the clear with them but I would consult an attorney. It may be worth it to offer them a little bit of cash for their time so that you do not have to worry about being taken to court. An attorney may be able to send them a stern letter about what would happen if they persue the legal route and lose but either way you will probably have to shell out a few bucks to get this too go away. did you point out to them the clause where they had to wait until you got the permits and if so what did they say? I guess they could say that since you signed the lease they were assuming you had all permits. Best talk to an attorney and good luck.
I was thinking about offering a couple hundred to move on, knowing that contractually I may not be held liable for anything though. I am concerned though that offering any money up front would be seen as acceptance of the fact they believe I owe them money at all.