[Resolved] Incorrect information being populated in Related Ads field

[Author]

Posts

[simonM-5]

Hi Support

I refer to a previously opened ticket (https://toolset.com/forums/topic/related-posts-not-being-populated-consistently/) where we claimed that the Related Ads were not being populated correctly. Yesterday we did intense testing and conclude once again, that the Related Ads are not being 100% CONSISTENTLY correctly populated, however approx 80-90% of the time of the time it seems to be being done right.

- In the past (as in the aforementioned ticket) we saw cases where the Related Ad was simply NULL.
- In our Production subdomain, we came across a case yesterday where 2 Related Ad names were populated into the same field, resulting in <NameOfAdOne><NameOfAdTwo> being populated/shown on the front end.
- Yesterday in our dev subdomain, we repeated our normal business process over and over and produced cases of the INCORRECT Related Ad being populated, and also the slug of a Message (not even an Ad, ie the wrong post type altogether ?!) being populated, which is completely wrong, since the column should contain names of Ads (either Job Ads or Nanny Ads).

In our concrete testing yesterday, we logged in on two different computers:
- one as the Nanny: {access details removed}
- one as the Family: {access details removed}
and went through our business scenario repeatedly with inconsistent results.

I enabled debug log on the dev subdomain, however no related errors were populated.

To reproduce this (in our case out of 24 cases it only worked correctly 22 times):

1) Log in as the two separate users (best would be two separate browsers, as Chrome sometimes cross references windows/tabs, even when private browsing unfortunately)
2) As the Nanny, click Find a Job
3) Click the View Ad on the Job Ad
4) Send a message via the box. This message should show up in this Nanny's Outbox (click the Messages icon in the menu > Outbox)
5) Log in as the Family and click Messages icon in the menu > Inbox.

In our testing, at the very least, the results were the same for the Nanny and the Family, be they correct or incorrect.

Viewing the messages in the Nanny's outbox (or similarly in the Family's inbox), the following messages contain incorrect Related Ads:
1) Mon 28/03/22 17:40: Related Ad should have been: "1 JAD EN"
2) Mon 28/03/22 18:00: Related Ad should have been: "2 JAD DE"

It is absolutely crucial for our business that this works reliably and consistently 100% of the time and is currently our only showstopper for our planned go-live this week on our www domain.

We would very much appreciate your assistance in correcting this inconsistent behaviour.

Thanks and kind regards
Simon

[Waqar Supporter Languages: English (English ) Timezone: Asia/Karachi (GMT+05:00)]

Hi Simon,

Thank you for contacting us and I'd be happy to assist.

Based on the details that you've shared, the only logical explanation I can think of, for those few inconsistent submissions, can be attributed to the use of new posts forms for these messages.

The Toolset's add new post works in a way, that whenever a page with a form to add a new post is visited in a unique browser, an auto-draft post is saved in the database. When the form submission is successful, that auto-draft post is changed to the actual saved/published post and another auto-draft post is created in its place.

While testing recently on a similar case for another ticket, it emerged that on a high-traffic website, where many users are simultaneously visiting the page with the new post form, this auto-draft post mechanism can result in overlapping of IDs, in very specific circumstances.

Unfortunately, changing or bypassing this mechanism will require a major refactoring of the Toolset Forms plugin and it won't be possible for us to share any time estimate, at the moment.

A fail-safe workaround that I can think of is to change the forms, "Post Form - New Message" and "Post Form - Reply Message" to be "edit forms" and not "add new" forms.

The two related custom code snippets "func-connect-hidden-ids-messages" and "toolset-messaging-code-snippets" are currently responsible for:

1. Connecting the newly created message post to the related job ad or nanny ad ( func-connect-hidden-ids-messages )
2. Updating the custom field and the content part of the newly created message post ( toolset-messaging-code-snippets )

In the new setup, you can set those forms to edit any placeholder dummy message post. The code snippets can be merged into a single snippet and updated so that first a new message post is programmatically added and then the remaining operations are performed on that newly created post and not on the post that is being edited, by the forms.

I hope this makes sense and please let me know if you need any further assistance around this.

regards,
Waqar

[simonM-5]

Hi Waqar

Thanks for the update.

Please remove the login info from the first message, once you have noted it. I still see it in the first message. Thank you! 🙂

Regarding your explanation of how Toolset's add new post functionality works:
"whenever a page with a form to add a new post is visited in a unique browser, an auto-draft post is saved in the database. When the form submission is successful, that auto-draft post is changed to the actual saved/published post and another auto-draft post is created in its place."
-- I got everything right up to the last part - I do not understand why after submitting/saving/publishing a post successfully that another auto-draft would be created in its place.
-- Our dev site is not exposed to search engines on the internet, so it can definitely not be classed as a high-traffic website, and in our intense testing on Monday, I was the only user visiting the page where the new message form was exposed.
-- Can you provide any more details regarding the "specific circumstances" you mentioned which lead to the incorrect behaviour?

Regarding the refactoring of the Toolset Forms plugin - is this plugin bug something that Toolset devs are already working on? Of course I appreciate software fixes can never be given guaranteed dates 😉 but do you have any rough estimates? Weeks? Months? Planned for next Toolset Forms release or not etc? It's just to have some sort of idea.

If I understand your update correctly, you are suggesting:
1) Change the Forms from "Add new" to "Edit" forms
2) Merge the two code snippets into one unified snippet
3) Make changes to the new unified snippet

While 1) and 2) would be easy, I am not a PHP Developer so would need urgent assistance with 3) to get this working again reliably. As mentioned earlier, we planned to go live on Monday this week and came across this bug whilst doing our final testing in the www subdomain and it really is a showstopper for us.

Thanks and regards
Simon

[Waqar Supporter Languages: English (English ) Timezone: Asia/Karachi (GMT+05:00)]

Thanks for writing back.

The underlying limitation that I pointed out earlier, is related to simultaneous visits ( in different browsers and not unique users ) and the use of new post forms, which also involve post-relationship connections.

During testing on my website with a similar setup as your forms, I was not able to reproduce this, so the right course of action would be to perform some tests on your website's clone. Both with your current setup and with my proposed changes and see how it affects the results and if something other than my initial observation is involved.

Please let me know if you agree and with your permission, I'll download the clone of your website, for this testing. The changes in the code snippet shouldn't be too complicated and I'd be happy to assist you with that too.

[simonM-5]

Hi Waqar

Thanks for removing the login details. 🙂

In case it's relevant for your research: Our intense repetitive testing on Monday was conducted by two different users, myself and my colleague. We were on two different networks/computers/browsers etc.

I was logged in as the nativenanny2 and was just viewing the different ads created by nativefamily6, writing a message from the Job Ad - single post and my colleague was then checking the results on her end, logged in as the nativefamily6 (ie did she see the correct Related Ad in nativefamily6's Inbox).

Of course you may take a clone of our website. We really appreciate your assistance in helping us get this critical issue resolved asap as it is preventing us from going live. I have prepared a Duplicator package for you (20220331_nativenannygmbhdev_TS_Support). Please shoot me a quick update when you have downloaded it successfully as it poses a security risk to keep these files on the server.

Please can you also document carefully any changes you needed to make, as I will have to replicate these on our production site once it is working on the dev site.

Would you like me to keep debug.log ON on the dev site while you are investigating? You have access via FTP to the file should you need to review it.

Again, thank you so much! Please don't hesitate to let me know should you need anything more from our side.

Kind regards
Simon

[Waqar Supporter Languages: English (English ) Timezone: Asia/Karachi (GMT+05:00)]

Thank you for the permission and I've successfully downloaded and deployed your website's clone.

I'll share the findings and recommendations, as soon as this testing completes.

Thank you for your patience.

[Waqar Supporter Languages: English (English ) Timezone: Asia/Karachi (GMT+05:00)]

Just wanted to update you that I've been performing some tests on this over the weekend and will resume work on this today.

Will share the findings with you, by the end of the day today.

[simonM-5]

Hi Waqar

Fantastic! Looking forward to hearing from you today!

Kind regards
Simon

[Waqar Supporter Languages: English (English ) Timezone: Asia/Karachi (GMT+05:00)]

Hi Simon,

I apologize for the delay as this investigation is taking longer than I expected.

I'm just completing the tests and will get back to you with an update within the next few hours.

regards,
Waqar

[simonM-5]

Hi Waqar

No problem, we appreciate that this ticket is complicated and probably requires a lot of testing on your side first, and remain very grateful for your assistance in this matter.

Kind regards
Simon

[Waqar Supporter Languages: English (English ) Timezone: Asia/Karachi (GMT+05:00)]

Thank you for waiting and I've narrowed down what causes those abnormal message entries.

At the database level, Toolset doesn't use the actual post IDs for storing the information about the relationship connection between two posts. Instead, it allocates a new and unique ID to each post in the table "{table_prefix}_toolset_connected_elements" and then uses that new ID to store individual relationship connection information in the table "{table_prefix}_toolset_associations".

On your website, the custom code snippet "func-sync-language-duplicates-on-submission" includes a function attached to the hook "cred_save_data" to automatically create language duplicate posts for the newly created message, when the forms "Post Form - New Message" and "Post Form - Reply Message" are submitted.
( this is fine as it is needed for the proper functioning of WPML )

But, at the same time, in the custom code snippet "func-connect-hidden-ids-messages", there is a custom function attached to the hook "cred_submit_complete", to automatically connect the created message and the relevant job ad/nanny ad in the respective relationship, when these same two forms "Post Form - New Message" and "Post Form - Reply Message" are submitted.

The abnormality arises because in WPML the post-relationship connections are maintained with respect to only the primary language post and when both these processes are run in parallel, at times, the incremental IDs used for the relationship connection match/overlap.

Thankfully, overcoming this is not very complicated. Your website is already using numeric custom fields for the "Message" post type for storing "Message from", "Message to", and "First Message ID" information. You can remove the post reference fields "Related Nanny Ad" and "Related Job Ad" and also the relationships "1 Job Ad to n Messages" and "1 Nanny Ad to n Messages".

Next, you can add two more 'number' type custom fields "Related Nanny Ad" and "Related Job Ad" for the "Message" post type and use them for storing the ID of the relevant nanny ad or job ad.

In the custom code snippet "func-connect-hidden-ids-messages", you'll make small changes, so that instead of joining the message post with the related nanny ad or job ad post, it updates their post ID in the new numeric custom fields:

add_action('cred_submit_complete', 'func_connect_hidden_ids_messages',10,2);
function func_connect_hidden_ids_messages($post_id, $form_data){
	// if a specific form
	if ($form_data['id']==1250 or $form_data['id']==1251){
		if(isset($_POST['current-posttype']) and $_POST['current-posttype']=='job-ad'){
			// to be removed: toolset_connect_posts( 'related-job-ad',$_POST['wpcf-related-job-ad'], $post_id);
			update_post_meta( $post_id, 'wpcf-related-job-ad', $_POST['wpcf-related-job-ad'] );
		}else if(isset($_POST['current-posttype']) and $_POST['current-posttype']=='nanny-ad'){
			// to be removed: toolset_connect_posts( 'related-nanny-ad',$_POST['wpcf-related-nanny-ad'], $post_id );
			update_post_meta( $post_id, 'wpcf-related-nanny-ad', $_POST['wpcf-related-nanny-ad'] );
		}
	}
}

The last step would be to update any views which are showing the related message posts so that instead of the post-relationship query filter, they use the custom field query filter.

I hope this helps and please let me know if you need any further assistance around this.

[simonM-5]

Good morning Waqar

Thank you very much for your detailed response.

1) In the code change: I am assuming that the new fields to hold the Related Ad IDs must be called Related Job Ad and Related Nanny Ad again, or at least have slugs related-job-ad and related-nanny-ad, correct?

2) Is it ensured that WPML language duplicate posts will also correctly pick up the correct IDs every time? In other words, no matter what language the Ad is viewed in, there will always be 2 pairs of IDs associated? I'll try to explain what I mean:

So if a user is using the German version of the site,
- the WPML language original of the (new/reply) Message post will be DE, and its language duplicate EN.
- The post ID of the Related Job Ad or Related Nanny Ad would also be the post ID from the German version of the Ad (be that a WPML original or a WPML language duplicate).

A similar pair would also have to be created, so when the user views the site in English that the same connections are made, ie
- the EN Message post (in this example a language duplicate) would have to pair up with the EN post IDs of the Related Ads (whether they are WPML language originals or WPML language duplicates).

It is a lot to process, especially as I am working on another project today, but I think I have understood the steps that need to be taken and understand what the code change is doing. As they say, the devil is in the detail. 🙂

I will do my best to have a go today and give you feedback. It may be after your shift has finished.

3) Did you manage to get this working reliably in your local copy? Could you maybe send me a link to a Duplicator package of your local copy, which I could fire up on a sandbox subdomain and use as a reference while making the changes in our dev (and later www) environments? That would be a huge help.

Thanks and regards
Simon

[simonM-5]

Hi Waqar

I have implemented all the steps as per your advice, but we are still not getting the desired results unfortunately.

I created a simple test case in the dev environment to illustrate the results.

a) Logged in as nativenanny2 after making all the code changes outlined below in CODE CHANGE STEPS TAKEN
b) Clicked View Ad button on 1 JAD EN (which has Post ID 22700 in EN)
c) wrote a message (The job Ad's author is nativefamily6)
d) viewed the inbox of nativefamily6 - he is seeing the correct Job Ad, but twice, like this: "1 JAD EN1 JAD EN". I would have expected the related-nanny-id field here to be null, and therefore display nothing for that field. I checked the Message record in WP Admin. The ID 22700 - the post ID of the EN version of the Job Ad - is actually being populated into the related-job-ad field correctly, but unfortunately it's also populating the related-nanny-ad field too. In my opinion related-nanny-ad should always be null for this entire message thread, based on the if statements in your corrected version of the code snippet "func-connect-hidden-ids-messages", since the logic "if is a nanny ad" should return false.
e) viewed the outbox of nativenanny2 - same thing - "1 JAD EN1 JAD EN"
f) opening the message conversation by clicking the user's name under FROM or TO (either as nativenanny2 or nativefamily6), the related Ad is showing (incorrectly, twice) as "CRED Auto Draft 034b28a18d1793d5acebe0b9c2e6c294".
g) viewing nativenanny2's outbox, it's also showing incorrectly, twice, as "CRED Auto Draft 034b28a18d1793d5acebe0b9c2e6c294CRED Auto Draft 034b28a18d1793d5acebe0b9c2e6c294" instead of just "1 JAD EN"
h) viewing nativefamily6's outbox, it's also showing incorrectly, twice, as "CRED Auto Draft 034b28a18d1793d5acebe0b9c2e6c294CRED Auto Draft 034b28a18d1793d5acebe0b9c2e6c294" instead of just "1 JAD EN"

The logic that should happen in my opinion is the following:
1) If the Ad I am writing the message from is a Job Ad and the language I'm currently viewing in is EN, then populate the related-job-ad field in EN record of Message post type with the post ID of the EN version of the Job Ad,
2) If the Ad I am writing the message from is a Job Ad and the language I'm currently viewing in is DE, then populate the related-job-ad field in DE record of Message post type with the post ID of the DE version of the Job Ad,
3) If the Ad I am writing the message from is a Nanny Ad and the language I'm currently viewing in is EN, then populate the related-nanny-ad field in EN record of Message post type with the post ID of the EN version of the Nanny Ad,
4) If the Ad I am writing the message from is a Nanny Ad and the language I'm currently viewing in is DE, then populate the related-nanny-ad field in DE record of Message post type with the post ID of the DE version of the Nanny Ad,
otherwise NULL in all 4 cases.
5) Each message (ie record in Message post type) should ALWAYS have EITHER related-job-ad or related-nanny-ad NULL, since a message can never come from both a Job Ad and a Nanny Ad simultaneously.
6) Logic 5) should qualify for EVERY message in a distinct conversation, ie for every Message with the same First message ID.

I hope I was able to explain well what I meant, and I hope I have done all the changes you requested properly in our dev. Please verify with your expert eye!

Please also comment on the final sentence in step 6) below of CODE CHANGE STEPS TAKEN.

Could the problem also be due to some conflicts with code snippet "toolset-messaging-code-snippets"? Is is still doing its job correctly?

Kind regards
Simon

CODE CHANGE STEPS TAKEN:
1) Deleted all existing posts in Messages in WP Admin

2) Removed extraneous relationships in back end Toolset Settings > Toolset Debug Information and Troubleshooting > Delete leftover intermediary posts > Delete leftover posts
- Deleted 0 dangling intermediary posts. Operation completed.

3) Backed up DEV completely using Duplicator Pro

4) Toolset > Relationships
- Deleted Relationships "1 Job Ad to n Messages" and "1 Nanny Ad to n Messages"

5) Toolset > Custom fields
- Renamed "Message details" to "Field Group for Messages" (not necessary but for consistency with the default names for other custom field groups)

6) Toolset > Custom fields > Field Group for Messages
- Removed Post Reference fields "Related Nanny Ad" and "Related Job Ad"
- Created new number custom field "Related Nanny Ad" related-nanny-ad
- Created new number custom field "Related Job Ad" related-job-ad
- Ensured "Copy from original to translation"" set after saving (would this be the correct setting here actually!?)

7) Alter custom code snippet "func-connect-hidden-ids-messages"
- Made changes as advised in https://toolset.com/forums/topic/incorrect-information-being-populated-in-related-ads-field/#post-2335755

8) Updated all Views using Related Message Posts, to use the Custom Field query filter instead of the Post Relationship Query Filter

Only these two Views were affected:
1254 Messages - Inbox
1256 Messages - Outbox

- Replaced

[wpv-post-link item="@related-nanny-ad.parent"][wpv-post-link item="@related-job-ad.parent"]

with

[wpv-post-link item ="[types field='related-nanny-ad' format='FIELD_VALUE'][/types]" ][wpv-post-link item ="[types field='related-job-ad' format='FIELD_VALUE'][/types]" ]

The following Message-related Views are unaffected:
1255 Messages - Related
19463 Nanny Profile Photo Messages View
19475 Family Profile Photo Messages View

9) Updated Single Content Template: "Message - single"
- Replaced

[wpv-post-link item="@related-nanny-ad.parent"][wpv-post-link item="@related-job-ad.parent"]

with

[wpv-post-link item ="[types field='related-nanny-ad' format='FIELD_VALUE'][/types]" ][wpv-post-link item ="[types field='related-job-ad' format='FIELD_VALUE'][/types]" ]

[Waqar Supporter Languages: English (English ) Timezone: Asia/Karachi (GMT+05:00)]

Thank you for sharing these detailed steps and observations.

I'm currently running some further tests in light of these observations and will share my recommendations accordingly.

I'll also try to share a working duplicator package from my test website, so that you can compare and analyze it, on your server.

[simonM-5]

Hi Waqar

That would be great. Do you have any estimates on when we could expect an update/workaround/resolution? I appreciate that the issue is more complicated than originally thought and I'm not asking you to commit to any date, but if possible, since it is the last issue preventing us from going live, we would like to have a rough timeline based on your experience, so we can discuss our options and plan accordingly. If you could shoot me a quick 2-line message with the current status, it would be much appreciated!

Thank you and kind regards
Simon