I figured out where this behavior comes from. This was caused by entering http://www.example.nl in the domain (mind the www.). When visiting the site without the www. you would be presented with the default language.
Redirecting all non-www traffic to www has fixed this behavior.