It'd require custom HTML and CSS, something we cannot assist in the Support Forum, as the design is done by the webmaster, while the structure with Toolset - for which we can provide full assistance.
To alter the checkboxes (or any other input kind of Toolset Views Search) you need to apply CSS to the basic HTML elements produced by the Toolset ShortCodes (the select inputs, checkboxes etc are all built with basic HTML).
First, you'll need to ensure the Toolset ShortCodes load "raw", means they load no other style but what minimally necessary (HTML and browser basics).
This is done in the GUI when inserting Filters, you can choose to output them with or without styles added. Choose without, hence, raw.
After that, your View on the Front End will still show the inputs (checkboxes, or any other) but with basic browser display, no Bootstrap or else applied.
Hence you can now add your own HTML around it, and apply CSS to it as well as to the input itself.
I'd probably wrap the entire Search input shortcode in a DIV and give it a fixed size so that inside of it I can apply the scroll you mention and group the checkboxes in a square (same amount on several lines) inside it.
When you insert Toolset ShortCodes for the search you can also pass a "class" attribute, which helps you add your own custom CSS classes to its output, and you can pass "style" which accepts valid CSS as input for inline style.
If you need someone to make the exact layout of the search you mention I'd suggest contacting a contractor from https://toolset.com/contractors/ as Toolset Support couldn't provide full stack HTML and CSS layouts for this, but we can show how to get there step by step within Toolset, and show the things that Toolset will help you within this case.
Natively in Toolset the checkboxes options when inserting the searches is all that's available, custom styling should be applied through custom CSS applied to the generated HTML and wrapping HTML as above suggested, see an example here for the basic HTML and CSS: https://stackoverflow.com/questions/12021338/scrollable-area-with-checkboxes
Note that a Toolset ShortCode will produce the label and inputs for each item without you need to add the HTML for it, that is what our Search ShortCode is there for.
So you will only work with the HTML around it, and apply CSS to both the generated and your custom HTML.
To find classes and ID of generated items you can always use the Google Chrome Console Inspector tool, which is the best resource to find quickly the ID or class of an element to address later with CSS or even JS.
In the loop things are different.
Here we not only offer a Loop Wizard to add basic loop layouts, but we also offer integration with several builders that help to visually organize the Loop's content. However such builders cannot be used in the Search part.
Note that the new Views Beta available at https://toolset.com/account/downloads/, contains a new approach to create Views which starts in the post editor, maybe you want to give it a try to later suggest improvements to the filter section?
I'd suggest doing that on staging sites only, of course.
Thank you for understanding the Support limitations.
