I believe that this should get the attachments of the translated parent:
$args = array(
'post_type' => 'attachment',
'post_parent' => pll_get_post($post_id, 'en')
);
$attachments = get_posts($args);
You have to set PLL_AUTO_TRANSLATE to false as otherwise Polylang will come set the post_parent to the current language.