You claimed that with ServerSelectorX installed, people could use /server <name> to go to restricted servers. There is no way that ServerSelectorX causes this, it's impossible for bukkit plugins to modify BungeeCord commands.
To make sure only certain players can click on an icon in the server selector, you can buy the premium version which supports permissions (for join items, opening the menu and clicking specific icons)
Discord chat log:
https://i.imgur.com/glbCJ8R.png