Thank you for getting in touch. The custom sort order you are referring to, is it to sort the products or are you referring to sorting the custom field filter options.
Please provide some clarity on this one so that I can best advise.
These filters are checkboxes or are they select fields ? Also are you using Blocks to create the search or the Classic editor. E
The reason is because you can specify the order in which your filter's options are being displayed. See my screenshot
For the blocks based search simply click on the filter to edit it and on the right hand side of your window you should see the options to edit the field. Just select the ordering you want from there.
If you're still unsure you can provide me with a screenshot of the filter on the backend and highlight it so that I can have a better understanding of what exactly we are handling.
I've escalated this to our 2nd tier supporters for further checks. However I do suspect that it is a case that the sort is being done as a string but you have decimal values as a part of that list to sort.
Ofcourse a sort like this will fail, so checking with our 2nd tier team to see what can be done in this case.
Our 2nd tier team has made a request to our developers to have this as a feature, however I cannot guarantee that this will be implemented and if the feature is approved I cannot say when it will be added as well.
This function can allow you to modify the orderby name clause and allow it to sort numerically rather than a string. If this is something that you're unsure of how to implement then I would recommend that you contact a developer for them to have a look and implement it for you.
I found another workaround. I renamed the slugs. So the slug for size 3 is now 030, and the slug for size 3.5 is now 035. That way, I can sort on the slug. 030 and 035 come before 10, 20 etcetera, so that works for now.