[Resolved] Search results 403 error if there's a single quote in Search field.

[Author]

Posts

[carlS-3]

Tell us what you are trying to do?
I have built my site with custom fields, searches, views, etc. Almost everything works as expected, but if a search query contains a single quote or apostrophe, I get a 403 error page.

Is there any documentation that you are following?
I have not been able to find any Toolset documentation for this issue, but see link below.

Is there a similar example that we can see?
If you visit the Search page on my site and search for "Mooney's" in the Artist/Title field, you'll see what I mean. This field is set to search the built-in post title and content. Or try searching for "Ain't Misbehavin'" in the Track Title field. This field is set to search a custom field. Either one will work if the search is for "Mooney" or "Misbehavin" (no single quote or apostrophe).

I'm using the Bulletproof Security plugin and thought this might be creating the problem. I found this article, which seems to describe the issue, but it appears that my .htaccess configuration is already set (as described in the article) to allow these characters in searches. I tried disabling the plugin too, but that had no effect either.
hidden link

What is the link to your site?
hidden link

Is there something with my WordPress configuration, or my hosting service? Do I need to do something different with my source CSV data file? I would appreciate any insights you can provide.

Thanks,
Carl

[carlS-3]

I made a couple of typos in my original request. Where it says "Mooney" I meant to say "Moody". Sorry for the confusion.

Corrected paragraph:
If you visit the Search page on my site and search for "Moody's" in the Artist/Title field, you'll see what I mean. This field is set to search the built-in post title and content. Or try searching for "Ain't Misbehavin'" in the Track Title field. This field is set to search a custom field. Either one will work if the search is for "Moody" or "Misbehavin" (no single quote or apostrophe).

[Jamal]

Hello and thank you for contacting the Toolset support.

The provided page gives a 404 error, so, I followed the link(Search Sessions) on the top menu to reach this page hidden link

I could not reproduce the issue on this page. In fact, the following link is a search results link for "Moody’s" with a single quote:
hidden link

Am I missing something? Can you provide some steps to follow to reproduce the issue?

In the meantime, if the issue persists, please check if it appears when:
- Only Toolset plugins are activated. It will tell us if there is an interaction issue with another plugin.
- The theme is set to a WordPress default like Twenty-Twenty. It will tell us if there is an interaction issue with your theme.
If the problem disappears, start activating one at a time to track where the incompatibility is produced.

[carlS-3]

Hi, Jamal

Thanks for responding. I apologize for the confusion. I had changed the slug for the search page after I submitted the ticket. I'm glad you were able to find it on your own. It sounds like you followed the same steps that I did, but experience a different result. The steps are as follows:

1. Go to the search page hidden link
2. In the first field labeled "Search Artist/Title" type in the term "Moody's" (without double quotes)
3. Press the Submit button

I have tried this on three different Mac computers using Safari, Chrome, and Firefox. I always get the same 403 error message screen every time. I am uploading a screenshot. I don't have a Windows computer to test on, but I found the error does not occur on iOS devices (iPad and iPhone Safari).

When I examine the URL that you shared, I noticed that there's one difference from my search result URL.
Yours: wpv_post_search=Moody’s
Mine: wpv_post_search=Moody%27s

So it looks like the apostrophe character is being accepted when it is a single "smart" curly quote (as when you type it in), but it is being converted to %27 which does not work when it is typed as a single "straight" quote (as when I type it in on my Mac). This seems to be the default keyboard input on a Mac, so I assume other Mac users would experience the same problem.

As a test, I modified another custom post that contains "Tony's Blues" in the Artist/Title field, then converted the apostrophe characters to straight single quotes and resaved the post. I still get the 403 error if I type the straight single quote version "Tony's" in the search field. If I copy and paste "Tony’s" with the single smart curly quote, I don't get the 403 error, but I see a "No items found" message.

I would appreciate any more ideas you might have.

Thanks,
Carl

[Jamal]

Hello Carl and thank you for your feedback. Today, I am able to reproduce the issue, but yesterday it just worked without issues.

The apostrophe will always be converted to %27 when it is used in a URL. Other characters will also be converted to their respective code. This article has all the characters that are not allowed inside a URL hidden link

From what I can gather so far, this seems to be a server security measure than an issue from Toolset. In fact, I tried to reproduce the issue on a clean install here hidden link

As a workaround to this issue, can you try if activating AJAX on the view will fix this issue?

To verify if this is actually a server security measure, let's migrate your website into our online platform and check if the issue will be reproduced. Please follow the instructions on my private reply(November 15, 2021 at 1:36 pm) to migrate the website. After that, check if the issue appears on our server and let me know what you will get.

[carlS-3]

Hi, Jamal

Thanks for your patience with me, as I am learning all this stuff as I go. Unfortunately, I don't know how to activate Ajax on the view. Can you help me with that first? Do I need to add a snippet to the Custom Code tab in Toolset? I couldn't find a support article that describes the steps.

Looking around, I also discovered that Toolset can integrate with Relevanssi. I am excited about adding this plugin for expanded search ability, but will hold off until we get this other issue resolved. Unless you think there's a chance it could help with the single quote issue.

Thanks,
Carl

[Jamal]

No need for custom code. You can activate AJAX on the view block settings. Check this screenshot for the search section hidden link And for pagination hidden link

Regarding the integration of Relevanssi, I can't tell if it will fix this issue or not. I still suspect it is a server related issue rather than an application(WordPress/Toolset/Relevanssi) issue. If we can test a copy of your website on a different server, or on my local development environment, I would be sure if it is server related or not.

Would it be possible for you to provide a copy of your site? Or to migrate it to our platform?

[carlS-3]

Thanks for your continues support. I'm a little nervous about copying my site. Let's hold off on that option for the time being.

Here's what I've been able to do since your last message.

I activated AJAX on the view block and enabled pagination with manual transition as shown in the screenshots you provided. Now the Search Artist/Title field will accept both Moody’s (with curly quote) and Moody's (with straight quote), but the results are different. The curly quote version yields three result, and the straight quote version yields only one result (different from the other three). Clearly that's because different style quotes were used in the CSV. But how is a user going to know how to put different style quotes in a search? My computer seems to default to the straight quote, but my iOS device defaults to curly. So the same search on these two devices yields completely different results. Ideally, I'd like the search query to treat them the same, or as just a good old apostrophe.

Here's an example of another odd character that works the way I would expect in the search. If you enter Mellé (with e-acute) into the search field you get the same results as if you enter Melle (with normal e). I wish the apostrophe would not be so strict.

I should mention that the source data was pretty old. It originated from a Mac database program that was close to 20 years old. I first created an Excel file, then output a CSV UTF-8 to import into Toolset. Could this be a contributing factor?

Another thing that's happening now is that after I enter a search with the straight quote and try to press the Reset button, the page freezes up with just the spinner graphic. The page does not reset, and I have to navigate away and come back. This doesn't seem to happen if I reset after a search that includes a curly quote.

Thanks again.

- Carl

[Jamal]

I can't really tell without some debugging. View have a debug mode that gives a popup with more details about the view, especially the underlying SQL query. That can help us understand why the different quotes produce different results. However, the debug mode only works with full reload pages, it can't work with AJAX.
https://toolset.com/documentation/programmer-reference/debugging-sites-built-with-toolset/#the-views-debug-tool

Would you allow me temporary access to your website to check it further? Your next reply will be private to let you share credentials safely. ** Make a database backup before sharing credentials. **

If I got stuck and could not generate the debug popup, I'll need to make sure that the original issue is not server related. So, I'll need to check a copy of your website, either on my local development environment or on our online platform. Let me know if you allow me taking the copy?

[Jamal]

Thank you for the credentials. I disabled the AJAX search on the view to generate debug info for the view. This works for curly quote and it is searched against it as it is(curly quote) on the database level. I could not check for single quote because I still get the 403 Forbidden error.

I did some online search regarding single quotes in WordPress, and it seems that WordPress convert straight single quotes to curly single while generating some parts of the page. You can read more about it on this article hidden link

WordPress is capable of storing both types of quotes on the database. But it converts the straight quotes to curly quotes on some parts of the pages. In fact, the following post has single quote on the title, and that is the one returned when the search is performed using single quote:
hidden link

And the following posts have curly quote on their titles, and that's why they are being returned when the search is done with curly quote:
- hidden link
- hidden link
- hidden link

I assume that the CSV file that you used for import has entries with straight and curly quotes.

However, for support rules, we are able to handle only one issue at the time. This helps us to bring you a better service and also helps other users to find all the information here exposed. For that reason I would like to make sure that your original request is fulfilled(403 errors). If you need further assistance with something else, please open a new ticket.

Regarding the 403 errors, I still suspect it to be a server security measure. We can confirm that, by taking a copy of your website and try it on our platform. Once confirmed, you will need to check with your hosting provider. Let me know if you want to continue on it and I'll create a new installation on our partner's platform(CloudWays) and I'll migrate your website to it.