How Post Relationship Replaces Post Reference and Repeating Fields

   Amir

Discussions

August 26, 2016
44

The next major feature for Toolset is “many to many relationship”. In this post, I want to introduce what’s planned and show what you can achieve with it.

What one-to-many and many-to-many relationships are

One-to-many means that one object “has” many other objects, or that many objects “belong” to one object. For example, a “project” may have many “revisions”. All revisions belong to a project. A revision cannot belong to two projects.

Many-to-many means that items of one side can connect to many items of the other side. For example, look at “events” and “speakers”. An event has many speakers and each speaker can appear in many events. In many-to-many relationship, there is no meaning to “belongs”, because each item can connect to many other items.

How post relationship replaces “post reference” and “repeating” fields

The most popular feature requests for Types have always been “post reference fields” and “repeating fields groups”. We’ve been a little hesitant to add these fields because it would make displaying content with Views a real pain. The “repeating field groups” in ACF, for example, use serialized data. This is super problematic for filtering purposes. In order to run a search over field values, you’d need to first de-serialize the fields and then search in PHP. Obviously, for a site that has more than a few hundred posts, this would be impossibly slow.

We checked a little (really, a lot) and understood that almost always, when you use post reference fields and repeating field groups, you’re creating some sort of relationship. For example, if you have a CPT called “flights” and you create a “post reference” field to save “landing airport ID”, you’re actually setting up an implicit relationship.

When you create a “house” CPT and use a repeating fields group for “photos”, you’re implicitly defining a new type “photos”, setting up its fields and connecting it to “house”.

Once you declare that “a house has photos” and “photos have XYZ fields”, you’re making it possible to:

  • Add photos to houses
  • Display the photos of a house when displaying a house
  • Display the house when displaying a photo

But all of this is only possible once you declare the relationship. This is why using “field reference” fields without declaring what they do is so problematic for building sites without PHP.

Toolset 2.3 will make “field tables” look and feel exactly like “repeating fields”

So, we understand that post relationship actually lets you achieve the same thing as “post reference” and “repeating fields groups”. But you know very well that the post relationship in Toolset, so far, is much less convenient to use than the repeater fields in ACF. That’s correct and we’re finally fixing it for good.

In Toolset 2.3, you will enjoy a streamlined workflow that lets you:

  • Add a “repeating fields group” directly
  • Add a “post reference” field directly

In the GUI, you will not see the need to set up the post relationship. Types will understand that when you add a “post reference” field you’re telling Types that this post has a parent. And when you create a “repeating fields group”, you’re actually setting up a child type and its fields.

All this will happen in the background, without any delay to your work and without any additional load on the database. True, there will be an entry in the ‘posts’ table for every row that you add in a repeating fields group. This entry is insignificant compared to the huge saving you’ll get when querying for these

How post-relationship will be stored in the database

Toolset will create a new table for the post relationship. This table will hold both ‘one to many’ and ‘many to many’ relationships.

The benefit of using a custom table is performance. Before version 2.3, getting the list of all related items in a many-to-many relationship would take order-N queries (one query per item). In Toolset 2.3, this would take only a single query.

It’s not just the DB queries. Having the relationship stored in its own table will make the PHP a ton simpler and your designs much simpler too. You will be able to just say “get me all the songs by an artist” (which are connected as many-to-many) and you can create a simple View to display them.

Detailed design, GUI and release schedule

We have a team of 5 developers working full time on many-to-many relationship in Toolset now. Our aim is to complete the detailed designs and GUI mockups during September. Then, around November we should offer the first working versions. This is a very large project, so changes can happen. In any case, it’s the major upcoming development in Toolset and we hope very much that you’ll like it.

Feedback?

What do you think? How would you use post relationship? Which features are most important for you? Leave your comments and we’ll get back to you.

 

Comments 44 Responses

  1. This sounds excellent. As always, I am working on a project now that would benefit from this update. Looking forward to seeing this progress as it sounds like it will make Types indispensable for creating slightly more complex websites.

  2. Excellent and much needed feature, and a great post to sum it all up.

    Looking forward to seeing this implemented – good luck 🙂

  3. Hi does this mean I can create a meal type and then add ingredients list with description and quantity for each ingredient?

  4. Very well decision. The OOPs & normalization design is appreciated since decades for effectiveness and efficiency. Seeing relationship table coming will made a lot more fast and convenient. I hope it will benefit parametric search a lot, and search performance.

    Are you planning to have a detail look into search performance as well?

  5. Amir,

    Thanks for publishing this explanation, as you promised. Certainly helps a great deal with planning to allow for how it will work. I’m extremely pleased you considered the existing issues with query overload we face now on alternatives.

    Looking forward to the coming M2M2M update release.

  6. Types is the entryway for Toolset and it is nice to see it getting these significant enhancements. Thank you for the update as there is a lot of interest in this area.

    I’ve been looking at relationships in terms of hierarchy and nesting and see a problem area. Take a look at bbPress for example. Forums are defined as one content type, topics as another, and replies as another, which makes perfect sense. However, look at how these are displayed in the admin. Logically I would expect to see something like:

    Forums -> First Forum -> First Topic (2 replies)
    -> Second Topic
    -> Second Forum -> Another Forum Topic

    Instead you see:

    Forums -> First Forum
    -> Second Forum

    Topics -> First Topic (forum 1)
    -> Next Topic (forum 2)
    -> Another Topic ( forum 1)

    Replies -> Reply to Next Topic (forum 2)
    -> Reply to First Topic (forum 1)

    Etc. The logical view is only available on the front end. In Justin Tadlock’s Message Board forums plugin it seems he solved this using custom programming and rewrite rules.

    Another example is using WordPress for stories. If you wanted to have multiple stories with chapters, displaying those on the back-end was difficult. I would want to see something like:

    Stories
    Story One
    Chapter One
    Chapter Two
    …
    Story Two
    Chapter One
    Chapter Two
    …

    Do the enhancements planned address this kind of ordering / nesting/ display issue?

    • Sorry, the indentation didn’t take. There is supposed to be indentation on each of those lines to show the hierarchy. Hopefully you can figure out what I’m trying to show. If not, I’ll create an image and link to it.

  7. Would I be able to add custom fields to a particular page or post instead of creating a custom type (like I do in ACF)?

    • You can assign a custom fields group today to pages according to the Content Template that they use. This way, if you display some fields for a certain template, only pages that use that template will offer these fields in the editor. I realize that you’re asking for a direct assignment to posts, but it’s not implemented right now.

  8. I have CPT for Articles and Books on the one hand and a CPT for Autors. I need to be able to choose (checkboxes) from a list of authors when I create an Article or a Book.
    Do I need to wait for Toolset 2.3 Many-To-Many relations or can this already be achieved with the current version?

    • Normally, authors have a one-to-many relationship with articles and books. This is because a book or an article can belong to exactly one author. There are very few publications that belong to multiple authors. If you absolutely must allow connecting a publication to multiple authors, you need to wait for the upcoming many-to-many relationship (which is in development). If it’s OK for you to restrict a publication to one author, you can built it right now with Toolset.

      We have a tutorial for how to set up one-to-many relationship and display that related data:
      https://toolset.com/documentation/user-guides/getting-started-views/part-8-one-many-relationships/

      Does this help?

      • Hi, in this case we have many books that were written by multiple authors, so the many-to-many relationship is important to us. Any ETA on that?

        I have looked at the tutorial, but you can only CREATE a child from the parent screen, what we need is to SELECT a child/children from the parent screen. We have all books and all authors already in separate CPTs. We would now like to select authors from a dropdown in the parent (book, article) edit screen.

        • Yes, selecting multiple parents will be the main GUI for the upcoming many-to-many relationship. Then, you will be able to create a View that lists the books by an author and the authors who wrote each book.

          I’m hoping to have this working in about 4 months.

          • And selecting multiple children also? Because it is more work to open 5 author edit pages and link parent Book, then to open 1 Book and select 5 authors from there.

            4 months? This site needs to be live next month 🙁 What do you suggest in the meanwhile?

  9. This sounds excellent and likely to enhance numerous solutions to complex CPT relationships.

    It’s great that you’ve communicated this roadmap to us. That really is appreciated.

    Once implemented, what are the implications for sites where we have already set up many-to-many relationships?

    Are there plans to facilitate a changeover to the newly available structures/features?

    • We’re going to offer a separate data-migration plugin, which will convert implicit many-to-many relationships (via an intermediary object) to the new design. This will take care of:

      * The relationship definition
      * The actual related data

      However, you will still need to edit your Views and Content Templates to adjust the display. Our migration code can only handle the data. It cannot automatically edit your front-end display and use the new data correctly. We will explain exactly how to do this and give instructions to edit your Views and Content Templates.

      Does this help?

      • That sounds promising, Amir.

        My immediate concern is specifically about the data – working through 1000s of posts and relationships isn’t very appealing, so having a data-migration plugin to convert relationships would certainly be welcome.

        Updating some Views and Content Templates would be a small price to pay. 🙂

  10. I am using Post Connector plugin on a site where I needed many-to-many relationships in toolset. Will there be a migration option?

    • Initially, we’ll create a migration tool from the current post relationship in Toolset to the new many-to-many relationships. Then, we were planning to look at posts2posts and see if we can create a migration tool. Let’s see how popular Post Connector is for other Toolset users and we’ll decide when we can create a migration tool for it too.

  11. THANK YOU!

    Would appreciate an early access to the migration tool so I can start planning and testing out the required changes.

    • Sure. It’s still in the future, because the entire project is not implemented yet, but you’re get it.

  12. If I understand well, that means no more 3rd object to create relationship ?
    If so I’m gonna dance to praise you for that!

    • These intermediary objects will move to a custom table and become optional for you. If you NEED to include fields with the connections, you can enable the intermediary objects. Otherwise, no intermediary post will be created and it will all be inside the new relationship table. You will be able to query directly the related objects without having to pass via the intermediary items.

      • Let’s say I have workshop with many teachers. Teacher have phone number field. So if I want to display a loop for all teacher & phone field, I’ll still need the intermediary object?

        • In this case, I don’t see a need for the intermediary objects. The teachers have their fields and the workshow have their fields. An intermediary object can be teacher-assignment-to-workshop, but you don’t need it.

          When you display a workshop, you will be able to include a View that loops through the teachers in it. Likewise, when you display a teacher, you can include a View that loops through the workshops that the teacher participates in.

          There are other cases when you do need an intermediary object, but not in this case.

          • I’ve been waiting for this one! Thank you!
            Namaste (*

            In what kind of case this intermediary object still be needed ?

            • Let’s say you have “songs” and “records”. A record can include many songs and a song can appear on many records. If you want to know on which TRACK a song appears on a record, you need the intermediary object. That object connects between “songs” and “records” and can include its own information. In this case, the track number is the meta information that you want to add to the connection.

              Does this make sense?

  13. @Amir, thanks for the roadmap, really appreciate it.

    I’ve been trying to avoid using ACF for the reasons you mention and because it’s so redundant to your plugins. This sounds like I at least will no longer have the need for ACF.

    I have an ongoing, large scale application that will benefit from performant many-to-many relationships (without custom code). I’d love to alpha/beta test.

    Thanks!

    • Sure. This development is still far from completion. As soon as we have a working version, we’ll release a first beta. This is major development for Toolset, so I’ll write a separate email newsletter when it’s ready.

  14. Sounds interesting. Will there be a migration from current post relationships to repeating fields? Will there be an advantage to doing that migration? Right now, I have a relationship between an equipment test and individual test points for that test. Should/can that be moved to a post type with repeating fields?

    Will relationships get “cascading delete”? Right now, if I have a relationship between say, test and test points, if I delete an test, the associated test points are orphaned. They exist in the database but have no parent object. There is no way to delete them besides running a custom query in SQL.

    Thanks

    • We are going to add the “cescading delete” to Views. Like you wrote, it makes no sense to leave children once parents are deleted. This will be optional and you will be able to control if it happens.

      Yes, there will be migration to the new schema. The old relationship schema would continue to be supported, so that existing sites don’t require any change. The migration will allow you to move your data to the new schema and enjoy its new features.

  15. Many to many relationships is a most welcome addition. I’ll be honest I used types for one project and then moved to Pods for others as it supports M:M out of the box. Hopefully the implementation will be straightforward.

    You should check out how the Pods team did it for ideas.

    • Actually, we are basing our design for many-to-many relationship on the Ruby on Rails framework. We are planning to offer a very complete solution (also expanding the existing one-to-many), hopefully with a GUI that will make it easy to use.

  16. How is this going to effect the current many-to-many relationships that were set up the old way, with an ‘invisible’ relationship CPT sitting between the two actual CPT? For example, a CPT ‘Instructor’ is the child of a CPT ‘Teaches’, which is also the parent of a CPT ‘Class’, so that an instructor can teach multiple classes, and a class can have multiple instructors. Is there going to be a migration tool, or will it need to be completely re-done? Will it break those old setups during the update?

    Also, is there an approximate release date? I am on the verge of starting a massive 6 month project that will have many of these many-to-many relationships, and I don’t want it to break because of the update.

    • Yes, there will be a migration tool. First, the current schema will continue to be supported. Your sites will not change and a result of this new development. The new schema will offer more features, easier use and better performance. You will get a migration tool that will move your data. It will set up the new relationships, based on the current ‘manual’ relationships. It will also migrate the data itself. What you will need to do yourself is to change the Views which display the content.

      This is going to be a fairly long development cycle. We’re looking at 4-6 months for all of Toolset plugins (which is like to major releases for WordPress).

      • Awesome, thanks for replying.

        Sounds like it should be pretty straight forward, but I’ll be sure to try it on on the staging server first.

        I’m excited to not have to deal with the intermediary CPT anymore, they just had a lot of rough edges, especially when I had to allow clients to add/edit them. I’ve actually started to hide the WP dashboard for clients and just expose the bare minimum they need via cred forms

    • Our upcoming implementation for many-to-many will cover what p2p does, but we’re actually basing the architecture on the “Rails” framework. The definition covers a wider scope than what p2p includes.

  18. Hi Amir,
    can you send us the detailed designs and GUI mockups during September. than we have a feeling of the new interface. As well I would like to be a beta tester in october.
    I would like to start the new project with the new table structur. I seem that the “rube on rails” looking will be a start in a WP as a CMS.
    thanks for help Henryk

  19. This will just make Toolset all the more powerful. Already have several projects on hold that can utilise this feature. You approach seems sound and logical. Looking forward to the stable release.

    • Good to hear you are looking forward to this feature. Others as well. Our devs are woking on this and once the beta version is available we will write a blog post about it.