Hi!
This plugin correctly replace the file of a media (all languages at the same time):
https://wordpress.org/plugins/enable-media-replace
If you want to delete a translation, just click on the 'delete permanently' red link (either in the media list table or in the edit media panel).