Thank you for sharing the admin access.
Reviewing the website and the number of Toolset components that it is using, it is expected that you see a major improvement in TTFB value when the Toolset plugins are not in use.
In summary, TTFB consists of:
1. Request to Server
2. Server Processing
3. Response to Client
As 1 and 3 are related to the communication aspect, for the sake of this discussion let's keep them constant and only focus on the "Server Processing" part.
The more features a website has, the more processing the server will need to do in the background. On your website, Toolset plugins are managing a large number of custom post types, custom field groups, front-end forms, content templates, and views, layouts, etc. This means that the most fundamental operations like data storage (Types), data presentation (Views, Maps, and Layouts), data submission (Forms), and data access (Access) are being controlled by Toolset plugins.
When you disable these Toolset and other third-party plugins, there is not much left for the server to process and you notice the lower TTFB values.
We're constantly working on the performance offered by Toolset plugins, but a careful balance needs to be maintained so that no compromises are made on the ease-of-use and flexibility side.
As a website manager, what you can do is limit the processing efforts of the server, for your front-end visitors, using cache and static content/resources optimization techniques. You'll find some good points in the following article, but use them only as a guideline as it may not always be possible to follow each and every recommendation:
hidden link
I hope this explanation helps.