OK, found the issue. Evidently my category IDs were not really integers. After forcing them using intval everything is working fine.
I'm thinking about including an option so that Postie works natively with Polylang.
There were a couple of typos in your original code. For other's reference here is the working snippet.
foreach ($post['post_category'] as $key=>$cat) {
$term = term_exists($cat, 'language');
if ($term) {
global $polylang;
$polylang->set_post_language($post['ID'], $term['term_id']);
unset($post['post_category'][$key]); // otherwise I believe that we will create a new category
}
}