Soft 404's

[PaceyUK]

Just checking my webmaster tools and I noticed I have over 400 soft 404's for pages such as search?r=10038
Literally all of them are search pages. Are these included in the generated sitemap or is google just finding them when crawling the site? If they are included, how can I remove.
If they are not, I can block them being crawled from webmaster tools, right?

[PaceyUK]

Anyone?

[fenderbob]

Not sure what the answer is or what any of this means...

But I'm having the same issue in Google Webmaster tools.

Can anyone help or explain how to fix this???

[fenderbob]

Here's the message:

Googlebot for smartphones identified a significant increase in the number of URLs on http://www.-----.com/ that should return a 404 (not found) error, but currently don't. This can cause a bad experience for mobile users, who might have been looking for a specific page, but end up elsewhere on your website. This misconfiguration can also prevent Google from showing the correct page in mobile search results.

Recommended Actions:

1. Identify the URLs with errors
Open the Smartphone Crawl Errors report in your Search Console account to review the list of sample URLs.

2. Fix the issue
Check your server and CMS settings to make sure that these URLs return a 404 (Not Found) or 410 (Gone) HTTP response code in response to requests for non-existent pages. You may need help from your server administrator or hoster for this step

3. Verify the fix
Once you've fixed the URLs with errors, make sure that Googlebot for smartphones can access and see your content properly, or that they return a proper error result code. Use Fetch as Google with the "Mobile: smartphone" option enabled.

[glennrocksvb]

Don't they return a page saying the search result expired? That's not a 404 error.

[fenderbob]

PaceyUK

Do you have a bunch of custom search pages? Like pages that include a custom search module? I'm noticing that the spike in my 404 pages are ALL linked from my custom search pages. When I follow the link from webmaster tools (saying it's returning a 404), the vbulletin search page does load as normal (site logo, style, etc.), but with this message: "Search result already expired, please try a new search."

[fenderbob]

Don't they return a page saying the search result expired? That's not a 404 error.

[ATTACH=CONFIG]n4338593[/ATTACH]

Yep! That's exactly what I'm seeing. But Google is calling it a 404 error for some reason. Webmaster tools is telling me mine spiked to over 160 soft 404s over the past few days.

[glennrocksvb]

You could file an improvement request in JIRA to have it changed.

http://tracker.vbulletin.com/secure/...91&issuetype=4

While you wait, you could edit /core/vb/api/search.php if you want.

Change: (line 278 in 5.2.0)

throw new vB_Exception_Api('invalid_search_resultid'); 

to:

throw new vB_Exception_404('invalid_search_resultid'); 

Note that it is not recommended to edit existing php files. You will have to apply the change every time you upgrade. Do this at your own risk.

[fenderbob]

You could file an improvement request in JIRA to have it changed.

http://tracker.vbulletin.com/secure/...91&issuetype=4

While you wait, you could edit /core/vb/api/search.php if you want.

Change: (line 278 in 5.2.0)

throw new vB_Exception_Api('invalid_search_resultid'); 

to:

throw new vB_Exception_404('invalid_search_resultid'); 

Note that it is not recommended to edit existing php files. You will have to apply the change every time you upgrade. Do this at your own risk.

Thank you!

I've created a new issue here: http://tracker.vbulletin.com/browse/VBV-15658

Hopefully I've correctly explained and identified all the technical aspects to this issue.

[glennrocksvb]

You can link this thread in the ticket in order for them to get more info of the issue.

[Wayne Luke]

You should exclude Google from indexing search results by excluding /search in your robots.txt.

Normally these are not linked anywhere but your users may be linking them for some reason. Searches only last about an hour before they are deleted and they will return the error above.

[glennrocksvb]

But in any case, shouldn't returning 404 make more sense since expired search result pages don't exist anymore?

[fenderbob]

Wayne Luke

I don't have a robot.txt file but I'd like to make one to fix this for the time being.

What other sub-directories should be excluded in this list? I think I've seen the googlebot trying to index private messages once, but I could be mistaken.

[Wayne Luke]

Anything personal or that you don't want to have Google Index.

/account
/member
/online
/memberlist

[PaceyUK]

I don't have any custom search pages or anything like that, but I have updated my robots.txt and added /search. Fingers crossed that will help, it went over 900 soft 404's today.