No: 'noindex' detected in 'robots' meta tag

[Blackhorse]

Hello,

I found out that google has not indexed more than one million url of my site, and the reason is No: 'noindex' detected in 'robots' meta tag

How can I solve this or how could i allow google to index url of my site?

I claim there is no problem with my robots and it allows indexing

Please your guidance

Thanx

[Blackhorse]

Availability --> Excluded by ‘noindex’ tag

URL will be indexed only if certain conditions are met

Discovery --> Not checked in live tests
Crawl Time ---> Apr 29, 2019, 8:27:00 PM
Crawled as --> Googlebot smartphone
Crawl allowed? --> Yes
Page fetch--> Successful Indexing
Indexing allowed?--> No: 'noindex' detected in 'robots' meta tag
User-declared canonical--> N/A
Google-selected canonical --> Only determined after indexing

[Wayne Luke]

Remove the noindex meta tag from your site. We don't add this. There is an option to add it for links leading off your site but we don't add it to individual pages.

[Blackhorse]

Hello Wayne,

could you please guide me how could i remove these meta tag or where could i find them?

thank you

[Wayne Luke]

They would be in your page headers. Check your robots.txt file, header template, head_include template. I don't know how to remove them because I don't know how you added them.

[Blackhorse]

They would be in your page headers. Check your robots.txt file, header template, head_include template. I don't know how to remove them because I don't know how you added them.

Thank you Wayne,

Here is my robots

User-agent: *
Disallow: /admin/

User-agent: AdsBot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Mobile
Allow: /

sitemap: https://www.XXX.org/core/store_sitemap/vbulletin_sitemap_index.xml

sitemap: https://www.XXX.org/core/store_sitemap/vbulletin_sitemap_starter_1.xml

sitemap: https://www.XXX.org/core/store_sitemap/vbulletin_sitemap_page_2.xml

sitemap: https://www.XXX.org/core/store_sitemap/vbulletin_sitemap_node_0.xml 

===

And This is the header template, i have never changed any, I think it is the same as in vbulletin default:

<vb:if condition="$page['noindex'] AND $page['nofollow']">
        <meta name="robots" content="noindex, nofollow" />
    <vb:elseif condition="$page['noindex']" />
        <meta name="robots" content="noindex" />
    <vb:elseif condition="$page['nofollow']" />
        <meta name="robots" content="nofollow" />
    </vb:if>
    <vb:if condition="$vboptions['facebookactive']"><meta property="fb:app_id" content="{vb:var vboptions.facebookappid}" /></vb:if> 

=====

head_include template is empty

[Wayne Luke]

That looks correct. The IF conditionals are to control what is indexed. You don't want your login page indexed so that would be marked noindex internally.

What is the URL structure of the urls that are being flagged by Google? Can you paste some of the URLs here?

[Blackhorse]

Hello Wayne,

I appreciate your reply and noticeable effort, Thank you

Here is some urls that google didn't index and they are all public and huge numbers of them:



====

Also some serach pages which i think it is better to be deleted from database but i don't know how or what table

[Wayne Luke]

Anything with explain=1 on it shouldn't be indexed. Google shouldn't even be accessing these pages. Do not run your site with Debug Mode on.

Search Pages should not be indexed. Search result pages last 1 hour. After that they are deleted from the system.

I don't know why the noindex is being added to other pages. It isn't standard for the software to do that.

[Blackhorse]

Hello Wayne,

Anything with explain=1 on it shouldn't be indexed. Google shouldn't even be accessing these pages. Do not run your site with Debug Mode on.

But these url with explain are pure topics and public, How could I delete explain then to allow it to be indexed?

====

I don't know why the noindex is being added to other pages. It isn't standard for the software to do that.

Is there anyway to add indexor follow instead?

What steps you recommend to follow to solve this?

Thanx

[Wayne Luke]

URLs with ?explain=1 will only occur when your site is in debug mode. You should not run your site in Debug Mode.

I can't recommend a method for fixing your issue because it doesn't occur in a standard installation of vBulletin and there are no options to add "noindex" to URLs.

Rebuild and submit your XML Site Map again. Make sure your site is not in debug mode when doing so.

[Blackhorse]

Hello Wayne,

What happens if i remove these lines from the header template?

<vb:if condition="$page['noindex'] AND $page['nofollow']">
<meta name="robots" content="noindex, nofollow" />
<vb:elseif condition="$page['noindex']" />
<meta name="robots" content="noindex" />
<vb:elseif condition="$page['nofollow']" />
<meta name="robots" content="nofollow" />
</vb:if>
<vb:if condition="$vboptions['facebookactive']"><meta property="fb:app_id" content="{vb:var vboptions.facebookappid}" /></vb:if>

Thank you

[LockedOnLaw]

Hello Wayne,

What happens if i remove these lines from the header template?



Thank you

If you remove the conditionals every URL of your site will be publicly indexed, including your hidden forums. That would not be advisable.

[Blackhorse]

If you remove the conditionals every URL of your site will be publicly indexed, including your hidden forums. That would not be advisable.

It will be indexed but i think all secret forums such as admin forums still will not appear publicly as i am making it secret in admincp itself

right?

[Wayne Luke]

Removing the conditions will most likely not change anything in your output. There are only two pages listed in the Route table with noindex set to true. These are the register and subscription pages.

Only one page has nofollow set to true. That is the register page.

There is no interface for you to turn this on/off for any other page at this time. Like I said, I don't know how this is getting added to your pages. When I visit your site, I do not see noindex nofollow meta tag in the page source code anywhere. nofollow is used on links to send private messages to users (not sure why you allow guests to send private messages but...)