Let’s start with the shortest definitions of both:
Custom fields are used to store extra information for a post
Taxonomies are used to group posts together
In other words the simplest two «rules» would be:
Use custom fields if you need to add additional data to a specific post
Use taxonomies if you have information that is shared across many posts
Taxonomies Have Archive Pages
One of the biggest differences with taxonomies is that they have archives. In WordPress, each taxonomy and each taxonomy term get their own archive page that lists all the posts grouped under it.
For example, a real estate site might have a taxonomy called «Property Type» with terms «Houses», «Offices», and «Apartments». Going to an URL like mysite.com/property-type/apartments/ will list all properties marked as apartments. And of course, Toolset allows you to easily design such archive pages.
Taxonomies can be Hierarchical
Another thing that makes taxonomies different is that they can be hierarchical. You can have taxonomy terms that act as a parent, with multiple nested child terms.
In other words, you have to use taxonomies if you need categories and subcategories.
Listing Posts by Custom Fields and Taxonomy
Choosing to use custom fields or taxonomy also impacts the way you can query and list posts.
First, you can order posts by the values of a specific custom field. However, you cannot order them by taxonomies. Instead, you can only display a group of posts that belong to the same taxonomy term.
For example, on a real estate site, it makes sense to order posts by the price custom field. However, it makes no sense ordering them by their taxonomy for type (i.e. houses, apartments, offices) because each taxonomy term has a whole group of posts.
Another thing to consider when listing posts is performance. Displaying posts by taxonomy is faster than displaying them by custom fields.
Adding new Custom Fields vs. new Taxonomy Terms
To add new custom fields to your posts you must be a site administrator. Also, you can only add custom fields from the backend using Toolset (or some other 3rd-party plugin).
On the other hand, users other than administrators can add new taxonomy terms. For example, adding a tag to an article like news is a completely usual action for any user that can create a new post.
Because of this, if your site provides a front-end form for adding new posts, users can even add new taxonomy terms right from the site’s front-end.
How to Choose?
You can often achieve the same using custom fields or taxonomies. But your choice impacts the way you will develop the site.
Here is a general list of recommendations on which option to use.
Use custom fields when you want to:
Add information that is specific to the post
Add information that you want to order posts by
Add post data whose primary use is to display additional information for the post
Use taxonomies when you want to:
Add information that is shared (the same) between many posts
Add hierarchical information (i.e. categories with subcategories)
Have an archive page on the front-end so you can browse posts by categories