[Closed] Posts get connected to wrong elements in many to many relationship

[Author]

Posts

[takoL]

Hi Waqar,

We said earlier that we could not replicate the issue on your sandbox.
This was is mistake, we actually can replicate the issue on your sandbox.
However, you really need to look for the posts that went wrong.
For instance see post with id: 788.

Regards

[Waqar]

Thank you for waiting and for sharing your further observations.

After much testing and troubleshooting, I've shared what we've discussed along with the test results about the inconsistencies with the concerned team for further review.

I'll keep you updated through this ticket and appreciate your continued time and effort for this investigation.

[takoL]

Hi Waqar,

Thank you for escalating to 2nd tier.
I know you said you will keep us updated, but still I have to ask 🙂 ... do you have any updates, however small?
Did anybody have a chance to look at it? If so, were they able to replicate the issue? If not, do you have any indication of when they might be able to?

Also one update from our side. Since the relationships were all mixed up, we had to repair the toolset_associations and toolset_connected_elements tables for several of our clients. Upon inspection we saw that the toolset_connected_elements tables had a lot of records with duplicate group_ids (see attached image for screenshot from my localhost)... which we think should not be the case? So maybe that's something worth looking at?

We removed the duplicates from the toolset_connected_elements table and removed all references to these duplicates from the toolset_associations_table. Than we reconnected the elements for which we were able to determine what the relationship should have been. Very time consuming... but this did seem to fix the mixed up relations. We now have to check the databases for all of our clients (at least) every day and if any anomalies are found we have to cleanup the database again. Not something we can keep up for very long. So really, any update however small would be much appreciated.

Thanks in advance and cheers,

Tako

[Waqar]

Thank you for waiting and your observations are close to our findings.

Our testing and research confirm that the inconsistencies are related to how the relationships are saved in the database.

Whenever a post is joined in any relationship for the first time, its entry is included in the "toolset_connected_elements" table. Post ID of that post is saved under the "element_id" and every record in the table is given a new unique "id".

The actual post-relationship connections are saved in the "toolset_associations" table based on those new "ids" from the "toolset_connected_elements" table.

However, when bulk posts are generated and connected in relationships at the same time and through simultaneous connections, the overlapping occurs between the newly inserted post IDs and element_ids. As a result, unexpected relationship information is saved in the "toolset_associations" table.

The repeated tests show, that separating the post-insertion and post-relationship connection method (though a new array that I recommended earlier) reduces the chances of this overlapping, but, unfortunately, it is also not 100% safe.

This limitation of the post-relationship structure has been highlighted to the concerned team. However, since it will involve major restructuring and code revamping, it will take considerable time and effort and therefore, it won't be possible for us to share any time estimate.

For now, for all the cases where you need to generate the posts in bulk and then join them together, you'll need to adopt a workaround, that doesn't rely on post-relationships.

Have you considered storing the related child post IDs, in a repeating custom field in the parent post? This will not help in the front-end search filters, but, you'll still be able to show the related posts on the parent and child posts.

[takoL]

We appreciate the effort your team put into reproducing and analyzing the issue. However, the suggested solution is not going to work for us.

We rely heavily on toolset relationships and have built a complex learning environment where tasks are assigned, handins are submitted and assessments are made with rubrics etc.
We use relationships and the insert/connect algorithm in many parts of this system so it is not an option to move post ids for relationships to post meta. We would need to rebuild our application, which would take us months.

We have had to take down two of our large clients. They have been waiting for a solution for over a week. They could not wait any longer so we had to put one of them back online with a database rollback to August 24th. For the other one, we had to manually clean up & rebuild associations and connected elements because there was too much new content for a complete roll back. We have in total 23 installations (schools / educational institutions) starting up this week that are also at risk of having this issue.

This issue has taken up all/most of our team's time for more than 10 days now. The upcoming days we will keep actively monitoring the databases of all our clients that can have this issue and we will keep fixing relationships / connections as we go. But we have to be very clear on this (and I hope you can understand) that we need a hotfix soon. Not a rewrite of your code in a next release, but a hotfix to take off the pressure for the time being.

We think we can justify the need for a hotfix by pointing out that other Toolset clients might also be affected by this if they have built high traffic sites with Toolset relationships. In our experience, the impact on data can be quite big: data loss / data breach and very cumbersome to repair or sometimes not even possible. This implies the urgency for a hotfix and we kindly request you to forward this consideration to the team involved.

[Waqar]

The development team is already reviewing this report and I've passed on your concerns and the request for a hotfix too.

Due to the nature of this matter, I suspect it will take some time, however, I'll keep you updated on the progress.

[takoL]

Can you provide us with an update on the progress on resolving this issue?
Is has been 4 weeks since the latest reply...

[Waqar]

I've requested an update on the progress and will share it with you, as soon as I'll hear back from the concerned team.

[takoL]

One week ago you requested an update from the concerned team.
You still haven't heard from them?

[Waqar]

Unfortunately, I haven't received an update yet.

Although, I can't control how the tasks are prioritized, I've sent a reminder again and will let you know, as soon as I have some news/feedback.

[Waqar]

Thank you for waiting and I've received a pre-release version of the Toolset Types plugin, which includes a fix for this issue.

In my tests, I was no longer able to reproduce this issue using this version and you can download it from:
hidden link

I hope this helps and let me know how it goes.

[takoL]

We will test this pre-release and will let you know the outcome

[Waqar]

Please take your time and let us know how it goes.