Yes that's a limitation. The current widget is quite basic and is not hidden if only the current language is avalaible and you chose not to hide the current language.
If you want to have such a widget, you will have to code it yourself, using the 'raw' argument in pll_the_languages which let you get all necessary informations to make a fully custom language switcher.