When we select and deselect between category results are not showing properly.
Select a category -> Will show result for only that category.
Deselect this category and select another category then -> first show result for selected category(second time category) and after one or two second show all result (all filter removed) automatically.
You can see this in following video . hidden link
At around 0:18 / 0:19 it filters properly and shows one category listings (for a second...)... but then again it shows all the sites
Please note that you need to choose the correct JS hook "The custom search has been triggered" and "The custom search results have been updated" and show/hide your element based on the hook triggered. There is no other workaround.
When we enable the pagination for the archive, filter starts working fine as expected. If this is a server side problem that server is slow then how it can be works when pagination enabled.
I think You answer the question yourself. When pagination is enabled you have a few records to fetch and display rather fetching all the records.
There is no workaround I can offer you at the moment other than you need to write your custom jquery code to show/hide the filter based on the trigger of ajax hooks I shared. Even I shared the test site where you see there is no -pagination added and it works as expected: hidden link
As shared, the only workaround could be you should use those ajax hooks to show/hide element based on the ajax action. Like use the hook "The custom search has been triggered" to hide the element and then "The custom search results have been updated" you can show the element again.