Roles removed when Server Tag unequipped#593

I’ve just set up something that when you equip the server tag of my server that you get access to a channel with color roles. (reaction roles)
But when they unequip the tag they still have the color role, even though they dont have access to the channel anymore. Is there a way to fix this, so when they unequip the tag they dont have that role anymore? It would be a great addition, in my opinion.