[Resolved] Need to import associations but they won't load

[Author]

Posts

[JamesK4399]

I'm not sure why these associations need to be loaded because the Petaluma Office has the necessary information in the fields. But when I try to import them it fails...

Associations loaded:
5 Associations ready to be imported
Relationship Parent Child Intermediary
Local Links Petaluma Best Coffee
Local Links Petaluma Best Hotel
Local Links Petaluma Parks
Local Links Petaluma Getting Around
3 Featured Projects for a Project Category Airports Los Vaqueros Reservoir Expansion Project

Lou has the login information on staging ...

[Beda]

I see no access details shared in your previous ticket with Luo.
Please provide them here again, if possible.

Importing relationships is required if you decided to import them, it won't' happen on its own.
You can read here how to handle relationship imports with Toolset:
https://toolset.com/documentation/user-guides/importing-content-from-csv-with-post-relationships/
It is explained here why you see what you see as described in your opening comment: https://toolset.com/documentation/user-guides/importing-content-from-csv-with-post-relationships/#an-example-of-how-a-csv-would-look-like
This would be the final step after this data is imported by any CSV Importer as post meta, you will see the associations on the Toolset > Export / Import > Associations as you do now.

Just click import, as the Document and GUI expect. Then the process is finalized.

[JamesK4399]

These items are already in the database in the Offices table for Petaluma, CA. No CSV file was involved. They appeared spontaneously. I can’t import them or delete them which seems like a bug to me which is not a huge deal but an aanoyance.

Is English your 1st language? If not then I will ignore your tone. Otherwise you come across as very sarcastic and bitchy. I haven’t encountered that by the support people here.

I have been told numerous times here that I should not provide login unless the message is sent privately. I will be happy to provide that if you send me private option.

Thanks, jim

[Beda]

I apologize if I appeared unfriendly, it was/is not my intention.

I understood that you had to import relationships but they "did not load".
Given you showed the screen where it prompted to finalize the import, by the title and description I understood you did indeed perform an import but that did not work well.
So the concluding steps would be to finalize the import as explained.

I now understand you never performed such import.
There is hence no reason why Types would ask to finalize it, and it's not expected.

I can analyze locally why this happened.
Do you recall after what precise action this started to happen?
This can help greatly to narrow down the reason and create a solution.

I have enabled private forms so you can submit me a duplicate if possible, so I can debug it locally without affecting the site.
https://toolset.com/faq/provide-supporters-copy-site/

[Beda]

I see that you write "My guess is that the users in that office added a few tips but then decided to change them to different categories. So they unlinked the items and left them orphaned. Toolset is probably trying to reconcille those orphans. But since they aren't needed, perhaps you could point me to the tables so I can remove them."

In the screenshot, though that looks like repeating fields, and those are not stored as relationships.
Repeatable Field Groups, those and Post Reference Fields, use the logic of the relationship.
Is the screenshot from a Toolset Form allowing to edit Repeating Fields?

This should not affect the impor screen at all, as those are just fields.
Now if those are repeatable, or reference, then theoretically it could affect the import screen.

These posts, in general, are stored in the native posts table, but the relations are more complex, in several tables and connecting into each other.

I am now downloading the package, to see what exactly this is.
I will update you with the findings.

[Beda]

The Download interrupts, and now the link is a 404

Did you eventually delete meanwhile?
Could you re-instantiate the copy?

I'm now seeing what I can do on staging directly.

[Beda]

These login details tell me "ERROR: The password you entered for the email address {removed} is incorrect. Lost your password?"

I have reactivated the private replies.

Note that online it's difficult to debug this, I need access to the database and will alter things in it, which will likely temporarily destroy the site.
I can then not access it anymore online, hence a duplicate would help greatly.

Thank you!

[Beda]

On the online site I see the relationships mentioned to be imported, do not exist.

I understand those are Repeatable Fields, but the Post in question does not hold the information that is proposed for import.
It holds information for each field, but different than in the import is proposed.

These imports happen only when you manually import relationships as previously explained. They do not happen when someone creates a Repeatable Field item and abandons it.

It seems you are using only the backend, right? There are no Post Forms, hence I suppose the editors of those offices will edit them in the backend.
Now, if someone edits or adds RFG items in the backend, either they get saved on post save as entered by the editor, or they are not saved (if removed with the trash bin icon). But they could not result in orphan posts, during these actions.
It would be like "not filling out" a Custom Field single line, that would not result in an orphan post.

I also see that you ave all plugins necessary for an import installed and active, and our developer confirmed that on the screen, only imports can appear which were performed by following these steps:
https://toolset.com/documentation/user-guides/importing-content-from-csv-with-post-relationships/

That would usually be used to import relationships, in your case, it lists Repeatable Field Groups, which are related posts under the hood.
Each Repeatable Field Group is basically a Post, that is related to its parent (the post which it belongs to).
It is possible that an import done with RFG items produces what you see in the screen if such import was done still following the above-mentioned link but for RFG items.

Now, I am downloading the package you linked, to see 2 things:
- what happens when we import what's offered to be imported
- how to replicate it consistently if possible

I will then update you on the outcome here.

[Beda]

The duplicate holds only the database, this means I will not have any of the FTP files you use on your site.
I still can see the import prompt, one is from Offices "Petaluma" and the other from Project Categories "Airports"

In the case of offices, it's related to RFG, while for the Project Category it's indeed a Post Relationship.
(3 Featured Projects for a Project Category)

These are results of either CSV imports or imports of posts from the standard WordPress import/export.
Also, theoretically, it could happen if someone uses post meta in that special format we use for storing the relationships (whether post or field), but it is highly unlikely.

However - the import results in "5 Associations couldn't be imported".
There are no errors logged or in the console.

I analyzed the code in Types and saw discrepancies in what we do with the data, and what we offer to do with it.
At the very least, an ignore/delete mechanism should allow us to delete those false imports, for example (assuming they would be real false imports).
Also, our code clearly provides methods to detect broken associations and is supposed to filter/delete them out before even offering them

I do however not have enough insight to determine how we detect importable items precisely, so I will involve the developers here, to give us instructions and answers about how this works and whether or not such enhancements could be done.

Currently I cannot offer any better solution but to ignore the import, however, I am aware that this will likely stop you to import anything else too.
I will try to expedite a solution as fast as possible.

[Beda]

I found the cause and solution to this issue.

1. All the offered imports are actually posts where either a CSV import, or a manual edit of the Database, or Custom Code adding meta fields had added a specific meta key with content: the "_toolset_associations_" custom field (hidden field).
2. This is exactly what a CSV import, or any other form of Import that handles relationships, have to do: create such a custom field and populate it with a precise syntax. This is elaborated here: https://toolset.com/documentation/user-guides/importing-content-from-csv-with-post-relationships/
3. In your site, 5 posts have that custom field, but their value is wrong. They hold Query URLs and Links, instead of the unique syntax of Post Names as shown on the above DOC.

This explains:
- why the imports are offered (because there are 5 custom fields with slug _toolset_associations_)
- and why they fail on import, (because they have Queries, and URLs stored as a value, instead of the {!{parent_title}!} + {!{intermediary_title}!} syntax as suggested by the Import DOC).

To resolve this issue:
1. Go to the PHP My Admin database and find all postmeta entries that have _toolset_associations_ in it (there will be 5 posts, exactly the posts that appear to be imported)
2. Delete those meta entries. You will also be able to notice their value at this moment, which will be links, instead of titles of posts.
3. In future imports, or existing automated imports, or eventual custom code, ensure that all meta fields with the _toolset_associations_ part in it also store the proper information as suggested by the doc, for future imports to work.

If you require help with this, I can delete the 5 entries for you in the database, but I'd need a confirmation that I'm allowed to do so, and access to it.

PS:
I still will ask the Developer to ensure something to "ignore" offered imports, so this would not need to be done manually.

Thanks!

[Beda]

I was wrong, there is nothing to be added yet in Types, because testing on a clean install, everything I expected the Developers to add (clear buttons and better checks) is already working.

Basically, on a clean install, if such fields are added as your database had it, with the same wrong values, then the Toolset Import Screen tells me there is a problem and correctly lets me delete those.

Hence there is something on the site that stops this process.
It could be custom code, another plugin or theme, or some option or even the particular import values.

Can you test what I offered as a solution in the last reply, and then let me know if your future imports (if any) do work?

[JamesK4399]

Hi, Beda. I'm so impressed by your thoroughness in researching this issue. I will run through your suggested actions this afternoon and let you know how it goes.

I partner with a local designer here in San Diego who was hired by ESA to help them rebrand the company to attract new talent and create a new website as part of their 50th anniversary. I've been the only person doing the development on the project. I was developing with a limited amount of sample data while on the ESA side the 3 project people were gathering all the photographs and content for the offices, services and projects they wanted to feature on the site.

There are 19 offices in the system and roughly 80 projects with photographs, descriptions of the projects, team members, and quotes from clients or other entities who were part of the project. They started gathering the information before I came on board and used a Word doc. I had hoped to extract the information from those files and do an automated import which is why I purchased the plugin for imports into WordPress which you saw. I only used the import feature to import employees from their HR system and a spreadsheet of clients. I found the plugin was a little limited and the data in the Word doc unsuitable for extracting. The ESA team actually wanted to manually import the projects because it reinforced their understanding of how the system worked.

I'm giving you this history because I'm still baffled as to how this happened. I never did an import of the offices. In fact, the initial office data came from the Drupal data I migrated. That data didn't have anything like the Local Links so I created the repeatable field.

Your message had a lot of information and I need to read it again. I'm going to go directly into the database but did you find a delete button on a clean install?

[Beda]

Yes, locally on a fresh install of WordPress with Toolset, after "breaking" the database entry to mimic what the situation on the issue site was/is, I reloaded the import screen and there was everything I expected:
- Warnings about broken relationships (due to the false entry in the post meta I crafted)
- Button to delete those relationship imports, which effectively deletes the post meta as well.

I do not know why on your site, these warnings do not appear. I would like to investigate this now, after removing the culprit ones I will try to somehow re-add corrupted data to check if the logic then works, on the duplicate.

On fresh installs, it works, and for the current resolution deleting the post meta as mentioned previously should/will solve the current issue.

I think the imports somehow held the slug mentioned in a postmeta column, and hence was added as such. I could not explain this elsehow.

[Beda]

On the Duplicate (or your website) in the PHP MyAdmin find _toolset_associations
There will be only 5 entries, in the post meta table, and 1 in the user meta table.
Focus on the ones in the post meta.

You will see 5 custom fields with meta key _toolset_associations_local-links (4 times) and _toolset_associations_featured-project.
Check their content, those are links, instead of syntax as shown on the above-linked DOC.

So usually, Types at this point would recognize this and display warnings + delete button in the Import Screen.
The funny thing is, as soon you alter the content of the meta entries mentioned, on the duplicate's database, to anything (like "anystring here"), you will immediately see the validation and even the delete button, on the Import Screen, even in the Duplicate.

So the problem seem that the value of the meta might break our validation, but even more interestingly, if you re-add the existing value to the meta, the issue does happen again!

The values you have currently in the database, are only "special" towards other (invalid) syntax as they represent existing things on your site, for example hidden link, or hidden link, which are used as values here, are actually existing URLS on the site.

I tested this on a fresh install, by manipulating the value so to match existing URL's, and could not replicate the issue (means, I got warnings and buttons to delete)

At this point I think we can safely accept the reason as of why the particular values in your site break the logic the way to do remains unclear, because of the following reasons:
1. The issue is resolved immediately if the corrupted data is removed
2. The issue does not repeat, if the other corrupted data is added
3. The same type of corrupted data does not produce the same issue on other installs

Since the syntax is simply wrong, and that being the cause of the issue, either fixing the content of the fields or removing them, is the resolution.

In case other imports need to be performed, the syntax suggested at the document should be used:
https://toolset.com/documentation/user-guides/importing-content-from-csv-with-post-relationships/

I attached a screenshot of what you will see if you alter the meta content of the post meta mentioned just by one letter, on the website or duplicate.
It will show the warnings and delete buttons.
Deleting those will completely, and safely, resolve this problem.

[JamesK4399]

Good news! I was able to find the posts and the postmeta entries for the local links. I deleted them and they no longer show up on the import screen.

This was the meta_key that helped me find them "_toolset_associations_local-links"

However, there is one more that eludes me. It's this one:

1 Association ready to be imported

Relationship Parent Child Intermediary
3 Featured Projects for a Project Category Airports Los Vaqueros Reservoir Expansion Project

I did a search for Airports and Los Vaqueros

I'm not sure which one of these entries is the culprit...

Showing rows 0 - 6 (7 total, Query took 0.0319 seconds.)
SELECT * FROM `wp_posts` where post_title like "Los Vaq%"

ID post_author post_date post_date_gmt post_content post_title post_excerpt post_status comment_status ping_status post_password post_name to_ping pinged post_modified post_modified_gmt post_content_filtered post_parent guid menu_order post_type post_mime_type comment_count

6742 729 2019-03-16 12:55:36 2019-03-16 20:55:36 los vaqueros reservoir inherit open closed los-vaqueros-reservoir 2019-03-16 12:55:36 2019-03-16 20:55:36 0 hidden link... 0 attachment image/jpeg 0

8617 3 2018-05-11 00:06:07 2018-05-11 08:06:07 The expansion of Los Vaqueros Reservoir is... Los Vaqueros Reservoir Expansion Project The expansion of Los Vaqueros Reservoir is an impo... publish closed closed los-vaqueros-reservoir-expansion-project 2019-08-14 16:39:47 2019-08-15 00:39:47 0 hidden link... 233 project 0

15599 3 2019-06-05 13:39:02 2019-06-05 21:39:02 The expansion of Los Vaqueros Reservoir is... Los Vaqueros Reservoir Expansion Project inherit closed closed 8617-autosave-v1 2019-06-05 13:39:02 2019-06-05 21:39:02 8617 hidden link 0 revision 0

17311 730 2019-06-18 15:58:31 2019-06-18 23:58:31 Los Vaqueros Resevoir inherit closed closed los-vaqueros-resevoir 2019-06-18 15:58:42 2019-06-18 23:58:42 8617 hidden link... 0 attachment image/jpeg 0

17313 730 2019-06-18 15:59:10 2019-06-18 23:59:10 Los Vaqueros Resevoir3 inherit closed closed los-vaqueros-resevoir3 2019-06-18 15:59:10 2019-06-18 23:59:10 8617 hidden link... 0 attachment image/jpeg 0

17314 730 2019-06-18 15:59:11 2019-06-18 23:59:11 Los Vaqueros Resevoir4 inherit closed closed los-vaqueros-resevoir4 2019-06-18 15:59:11 2019-06-18 23:59:11 8617 hidden link... 0 attachment image/jpeg 0

18478 730 2019-07-08 10:25:57 2019-07-08 18:25:57 The expansion of Los Vaqueros Reservoir is... Los Vaqueros Reservoir Expansion Project The expansion of Los Vaqueros Reservoir is an impo... inherit closed closed 8617-revision-v1 2019-07-08 10:25:57 2019-07-08 18:25:57 8617 hidden link 0 revision 0