Robots and site indexing problems

[Letzride]

I am in over my head on this one - about 1 months ago I submitted my forum to google search console for indexing. When I used to google "V6 Camaro Forum" I used to see my forum in the search results in various locations on the first page with my meta description for my main forum page....now today when I google it this is what I see:



When I click on the learn why it takes me to this page: https://support.google.com/webmaster.../7489871?hl=en

Which essentially says this message means that Google could not access the page in order to create a useful description. When I click the"how do I fix this link" It starts talking about robot txt files are blocking this page and telling google not to crawl it.

Similarly Google Console has come back and sent me a message saying new issue found: indexed, though blocked by robots.txt

So - how can I correct this issue in cloud vbulletin? Is there an area I can access robot.txt files? I can't see how asking google to index the site would have negatively done something to my existing site - there must have been a setting off before?

[Wayne Luke]

That isn't a valid vBulletin Cloud URL. We require a three-part domain name including a sub-domain such as WWW at all times. You need to make sure that the domain above redirects properly and does not try to obfuscate the URL for the user and they see the three-part domain in their browser's address bar.

The only pages blocked by robots.txt are search results and an old vB4 MAPI url. Search Results only last one hour and should never be indexed. The vB4 MAPI URL doesn't actually exist on Cloud but is used by the mobile apps. Here is the entire content of the robots.txt file on Cloud:

User-agent:*
#we're blocking /search requests since those search pages are temporary anyway.  
Disallow: /search
#we're blocking /forumdisplay.php since it's a vB4 MAPI method.  
Disallow: /forumdisplay.php

The same file is used by all Cloud Sites.

Make sure you don't have a robots.txt at camarov6.com which is outside our cloud network.

[Letzride]

What isn't a valid Vbulletin cloud URL? The Camarov6.com? So in order to fix this I need to have anything without www. redirect to www.camarov6.com? Would that be with http:// or https://

How can I ensure this is happening? : "You need to make sure that the domain above redirects properly and does not try to obfuscate the URL" Is there a test or way I can verify I have redirected everything properly?

Make sure you don't have a robots.txt at camarov6.com which is outside our cloud network.

Prior to about 24 hours ago I didn't know much about anything on robot texts- is there a way to verify this? or is it safe to assume that if I didn't create one and put it in my host domain file directory that it wouldn't exist?

Sorry if any of the above are dumb questions.

[Wayne Luke]

Yes, you need to redirect everything from camarov6.com to the HTTPS version of your cloud URL.

When you type camarov6.com in the browser's address bar, it should change to the HTTPS version of your cloud URL.

To see if there is a robots.txt file at your host domain directory you would have to log in via FTP or talk to the provider of the domain.

[Letzride]

So I just confirmed with hostgator my provider and they confirmed that all of my urls:

camarov6.com

https://www.camarov6.com

forum.camarov6.com

https://www.forum.camarov6.com

All redirect to https://www.camarov6.com

So I am confused again - I also checked my files in my control panel by my provider and did not find any robot.txt files pertaining to my forum site...

[Mark.B]

In Webmaster Tools you need to tell Google that the URL is https://www.camarov6.com. Not https://camarov6.com.

Also, going to https://camarov6.com does not redirect me to https://www.camarov6.com.

[Letzride]

Thank you for pointing that out - I had to purchase an SSL certificate in order to redirect https://camarov6.com and now I have done that - that redirects to https://www.camarov6.com

In Google Search Console I have specified my preferred domain is https://www.camarov6.com (which is the new web master tools)

I am still showing the same result when I google "v6 camaro forum" in that I have no details to display- I'll give it a few days for google to reindex the site, I am hoping this will fix the issue?

[Wayne Luke]

Have you submitted an update XML Sitemap with the new URL? That can speed up the process a bit though it can take 4-6 weeks for Google to update depending on the size of your site and the traffic it gets. Larger sites are updated more frequently.

[Letzride]

Have you submitted an update XML Sitemap with the new URL? That can speed up the process a bit though it can take 4-6 weeks for Google to update depending on the size of your site and the traffic it gets. Larger sites are updated more frequently.

Do you mean rebuild a new one now that I have the additional https://camarov6.com redirected to https://www.camarov6.com?

Would that make a difference in the sitemap? If so I will do that next

[Wayne Luke]

You need to build and submit a site map for https://www.camarov6.com that is your site. https://camarov6.com is just a redirect. You want the content on your site indexed, not a redirect.