That is expected.
At the moment you load the View on the page, the results are not filtered by the Front end Search yet, but just by the Query Filters and other Views settings in the backend.
As soon you then fire the Front End Search, you will see the expected results.
What you can do to "solve" this, is to use manually entered values in the Custom Search filter, when you insert or edit the filter in the views backend, you can choose to insert the values manually in to the select Field.
I added a Screenshot of that setting.
As you see I do not use a Empty value hence that view, by default, will then return initially results with the first set value.
But this is a cumbersome solution if you have many posts and values to take care of.
On that case it's better to either use the Custom Queries in the backend only, if you do not want the user to let change the results, or, to filter the View, before you render it on the page, so to show only posts with fields that have values, and then later let the user filter by certain values.
This Requires custom code.
You would need to run a filter on wpv_filter_query:
https://toolset.com/documentation/programmer-reference/views-filters/#wpv_filter_query
That would allow to filter the Posts found, just as in the code example in the DOC above, and return only posts with the field (exists).
Later, in the view you can still use a Custom Filter on the front end for that field.
If you require development help for the project regarding Custom code, I can recommend the Toolset Contractors:
https://toolset.com/contractors/