Skip Navigation

[Resolved] Fastest way to gather data dispersed throughout multiple custom post types

This thread is resolved. Here is a description of the problem and solution.

Problem:

Is there something in the toolset API, or a way to construct a toolset query that would return everything in the toolset database, or everything related to a list of Posts, all at once, in one call?

Solution:

Unfortunately, there isn't such kind of built-in API function for:

In your case, you might consider to save the result into cache, then it will be much faster when load it second time, for example WordPress Transients API:

https://codex.wordpress.org/Transients_API

Relevant Documentation:

This support ticket is created 4 years, 10 months ago. There's a good chance that you are reading advice that it now obsolete.

This is the technical support forum for Toolset - a suite of plugins for developing WordPress sites without writing PHP.

Everyone can read this forum, but only Toolset clients can post in it. Toolset support works 6 days per week, 19 hours per day.

Sun Mon Tue Wed Thu Fri Sat
- 9:00 – 13:00 9:00 – 13:00 9:00 – 13:00 9:00 – 13:00 9:00 – 13:00 -
- 14:00 – 18:00 14:00 – 18:00 14:00 – 18:00 14:00 – 18:00 14:00 – 18:00 -

Supporter timezone: Asia/Hong_Kong (GMT+08:00)

This topic contains 4 replies, has 3 voices.

Last updated by Focus on the Family 4 years, 10 months ago.

Assisted by: Luo Yang.

Author
Posts
#1545021

Tell us what you are trying to do?
We're trying to gather all data from a custom post type. This includes custom field data from the CPT, as well as all data from related post types. We are able to do it, but it's taking about 14seconds(give or take) to gather data from 850ish posts. We are trying to figure out what the fastest way to collect the information using toolset. It's somewhat of a time sensitive operation.

Is there any documentation that you are following?
https://toolset.com/documentation/customizing-sites-using-php/post-relationships-api/#toolset_get_related_posts

Is there a similar example that we can see?

here is some pseudo code of the process we are using to accomplish the ends.

1.Get all EPISODE posts from WP of the CPT
2.For each EPISODE post, get CHILD post types(album CPT, Contributor CPT, and character CPT)
3.for each CHILD post type, get the CHILD posts that are related to the EPISODE post (each episode has one or more albums, characters, and contributors associated with it)
4. for each CHILD post, get post meta (each album, character, and contributor have custom fields we need)
5. for each EPISODE post, get post meta (each episode has custom fields we need)
6. for CHILD post of type 'contributor', get INTERMEDIARY POST of CHILD and EPISODE (each episode has a contributor of a certain 'role' that is recorded in the intermediary post meta)
7.Get post meta of the INTERMEDIARY POST

This is the code. (give or take)
$args = array(
'post_type' => 'aio-episodes',
'numberposts' => -1,
'orderby' => 'ID',
'order' => 'ASC',
'meta_query' => array(
'key' => 'wpcf-media-type',
'value' => 'audio',
);
$episode_post_list = get_posts( $args );
get_posts_related_to_episodes_with_metadata( $episode_post_list )

public static function get_posts_related_to_episodes_with_metadata( $episode_posts ) {
$related_post_data = array();
if ( empty( $episode_posts ) ) {
return $related_post_data;
}

foreach ( $episode_posts as $episode_post ) {
$related_posts = self::get_posts_related_to_aio_episode( $episode_post );
if ( empty( $related_posts ) ) {
continue;
}
array_push( $related_post_data, $related_posts );
}
return $related_post_data;
}

private static function get_posts_related_to_aio_episode( $episode_post ) {
$related_posts = array();
if ( ! $episode_post ) {
return $related_posts;
}
$related_children_post_types = \toolset_get_related_post_types( 'child', 'aio-episodes' );

foreach ( $related_children_post_types as $custom_post_type => $relationship_list ) {
try {
$related_child_posts = \toolset_get_related_posts(
$episode_post,
$relationship_list[0],
array(
'query_by_role' => 'parent',
'return' => 'post_object',
)
);
} catch ( \InvalidArgumentException $e ) {
Logger::get_instance()->error( __CLASS__ . ':' . __FUNCTION__ . ':' . $e->getMessage() );
continue;
}

self::attach_necessary_post_metadata( $custom_post_type, $related_child_posts, $episode_post );

$related_posts[ $custom_post_type ] = $related_child_posts;
}
$related_posts ['aio-episodes'] = array( $episode_post );
self::attach_necessary_post_metadata( 'aio-episodes', $related_posts ['aio-episodes'] );

return $related_posts;
}

private static function attach_necessary_post_metadata( $post_type, $posts, $episode_post = null ) {
if ( 'aio-episodes' === $post_type || 'aio-albums' === $post_type ) { // Only episodes and albums use metadata.
foreach ( $posts as $post ) {
self::include_post_meta_with_post( $post );
if ( 'aio-episodes' === $post->post_type ) { // Only episodes have post categories.
self::include_post_categories_with_post( $post );
}
}
}
if ( 'aio-contributors' === $post_type ) { // Contributors have role data.
foreach ( $posts as $post ) {
self::include_contributor_role_with_post( $post, $episode_post );
}
}
}

private static function include_contributor_role_with_post( $post, $episode_post ) {
$intermediary_posts = self::get_intermediary_posts( $post, $episode_post, 'episode-and-contributor' );
$contributor_role = get_post_meta( $intermediary_posts[0]->ID, 'wpcf-episode-contributor-role' );
if ( ! empty( $contributor_role ) ) {
$post->contributor_role = $contributor_role[0];
}
}

private static function include_post_meta_with_post( $post ) {
$post_meta = get_post_meta( $post->ID );
$post->metadata = $post_meta;
}
private static function include_post_categories_with_post( $post ) {
$categories = wp_get_post_terms( $post->ID, array( 'episode-theme' ) );
$post->categories = $categories;
}

private static function get_intermediary_posts( $related_posts, $episode_post, $relationship ) {
$all_related_posts = array();
try {
$all_related_posts = \toolset_get_related_posts(
array(
'parent' => $episode_post,
'child' => $related_posts,
),
$relationship,
array(
'role_to_return' => 'intermediary',
'return' => 'post_object',
)
);
} catch ( \InvalidArgumentException $e ) {
Logger::get_instance()->error( __CLASS__ . ':' . __FUNCTION__ . ':' . $e->getMessage() );
}

return $all_related_posts;
}
What is the link to your site?
hidden link

#1545189

Well, these operations are time expensive when it comes to hundreds of posts and therefore hundreds of related and again hundreds of related-related posts.

That's unavoidable.
14 seconds is probably not unexpected. It's difficult to say how long such query takes, but imagine the operations ongoing in that query.
It's not unexpected to take a good while to load all data.

I can suggest, either working in batches, like a view that lists 10 instead of 800 posts a time
In fact, a view listing 800 posts, and all its related, and related-related data, would take more than 14 seconds, I am pretty sure

After all, you hit one database table over and over again (even if the relationship is in custom tables, the posts and meta, are not)
This is time expensive and usually solved by pagination (batching of the process)

We can't fully analyse Custom Code, but I also do not think there is anything wrong with your code, it's just the expected time for such very expensive query.
Maybe you can split it in steps. First get all parents, then, in a second "load", get the first layer of related data, then the second.

Maybe you can put that using ob_cache into a JSON and use that for the data consumption later, so you can fastly call up all data (being in JSON) on the front end once it's ready (the query)
This, however, includes Custom Code that we can't fully assist here, according to our support policy https://toolset.com/toolset-support-policy/

This is really a problem of WordPress generally and in Views, you can usually solve that by using pagination (if you were to load 800 posts with a View directly at once I fear you could even experience timeouts.)

#1545749

Makes sense. I have one more question on this.
Is there something in the toolset API, or a way to construct a toolset query that would return everything in the toolset database, or everything related to a list of Posts, all at once, in one call? Where the custom posts can be associated to one another by something returned by that call, like foreign keys relating one CPT to another?

#1546429

Hello,

Unfortunately, there isn't such kind of built-in API function for:

construct a toolset query that would return everything in the toolset database, or everything related to a list of Posts, all at once, in one call

See our relationship API document:
https://toolset.com/documentation/customizing-sites-using-php/post-relationships-api/#toolset_get_relationship

In your case, you might consider to save the result into cache, then it will be much faster when load it second time, for example WordPress Transients API:
https://codex.wordpress.org/Transients_API

#1547243

The problem I came to solve, seems unsolvable. There is no toolset way to make queries more efficient