So turns out the was a huge bug in this plugin for a long time.
It would work if only a single player was on the server or if all the players online had the effect. But the was a chance the plugin would stop working if a player without the effect joined the server.