Every saved card stores a cardholder address that the card-issuing bank uses for verification and fraud prevention. When you add or edit a payment method, you pick one of your organization's addresses (or create a new one) as the cardholder address. That is why such an address shows a payment-method usage badge and cannot be deleted while the card exists.
Let us know if this answered your question.