I just noticed that when I uncheck "Activate languages and translations for media" this does not remove the language attributes that were previously assigned to media items while this was still checked. So the previously uploaded media items keep their language attribute and continue to not be available throughout all languages. In addition, unchecking this setting removes the language switch, locking the media items to their previously selected language.
I might be missing something -- I haven't used polylang for very long at all -- but I thought disabling languages for media should also remove the language attributes from existing media. Thoughts?