Good one, I haven't thought about the admin-login url. I've tried it and you can indeed login with the other domain aswell and this changes the language domains in Polylang. Though in my case it all still works.. even the alternate url's in the head is still working.
the main thing is that you must not save the language settings on an other domain than the primary-language domain.