You are right, and it is not easy to fix it without a filter being made available by the Views developers to modify this.
It appears get_terms() is called four times on a page that contains a View with a single taxonomy filter (which is why the pre_get_terms hook is fired four times).
Unfortunately, unlike an API filter such as wpv_filter_query, which provides the $view_id, $view_settings, and $view_args, so that it is possible to determine what triggered the filter, the built-in pre_get_terms hook includes details of the $query, but these offer little or no useful information about what called get_terms() for us to be able to target just the pre_get_terms call we need to modify.
So, while the code I provided is successfully able to narrow down the contents of the filter control to only include the required terms, it also evidently interferes with further get_terms calls Views appears to make when it comes to building and submitting the query.
Any solution would require going through the $query object available to pre_get_terms to see if it is possible to isolate the specific calls to get_terms that we need to modify while avoiding changing the rest.
You could do that, or... yes, wait for the developers to add a filter to customise the filter content.
