google crawl/adsense?

[Salome]

Hi everyone,



Was hoping someone who had a similar situation, or knows more, might be able to give some insight.



Two months ago I had my webmaster make some changes to my forum. Upgraded from 3.8 to 4.2.0, new skins, new, new, new. So exciting! I tried some different ad placements and they were doing well. I had been averaging around 4 dollars a day and after all these updates, redesign and new ad placements I had started averaging around 8 dollars a day.



On 4/24 I asked my webmaster to install the vb facebook app. This is the last noodling on my site/server... And there were issues. I think something might have inadvertently been altered during this time?... The last week of April, my adsense dropped to a dollar a day. I have the same number of visitors, the same number of clicks, same number of page views – none of that changed. I couldn’t figure out what was causing this. About a week of these results and I got an email from google saying that google could not crawl my website. I followed the instructions in google’s email. I put the code in the robots.txt and uploaded it. I went to webmaster tools and under current status the DNS has a green check the Server Connectivity with a yellow exclamation (Google couldn't access your site because of a server connectivity issue.) and the Robots.txt Fetch has a green check. I went to my adsense and entered crawl login and checked it.



When I look in webmaster tools > it only shows one attempt during feb/mar/apr on april 4/26 it shows errors attempts 275/656 42%

I have not seen a difference since in adsense earnings, which means google has not tried to crawl again, right? That’s why the revenue dropped because they couldn’t see/couldn’t display relevant ads.



How long does it take for google to crawl? Can I make a request? Did I do everything I should to make sure google can see my site?



I realize completely this is an adsense not a vbulletin issue… but so many of you use adsense on your forum…I look forward to any input.





Thanks

[myths]

just fix your error from webmaster tool thats all

[Salome]

Hi,



thanks for responding. I'm sorry, I know this is really elementary stuff. But I'm unsure if I fixed it. I uploaded the robots.txt via ftp several days ago but when I am in webmaster tools > health > crawl errors and it shows robots.txt with a green check. However, when I mouse over the green check it says "Crawl postponed because robots.txt was inacessable".Have I failed to do something? What would you do to fix this in webmaster tools?

[myths]

why you are using robots.txt ?

dont use robots.txt for your whole forum use it only for those which one you dont want crawl and index on search engines (google)

[Salome]

I did it because that is what the google email said to do. It said open your robots.txt and paste User-agent: Mediapartners-Google

Disallow:
So I did that. And this is what was already in there:

User-agent: *

Disallow: /calendar.php

Disallow: /editpost.php

Disallow: /members/list/

Disallow: /misc.php

Disallow: /newreply.php

Disallow: /newthread.php

Disallow: /printthread.php

Disallow: /private.php

Disallow: /register.php

Disallow: /report.php

Disallow: /search.php

Disallow: /showgroups.php

Disallow: /online.php

Disallow: /subscription.php

Disallow: /threadrate.php

Disallow: /poll.php

Disallow: /attachment.php

Disallow: /avatar.php

Disallow: /faq.php

Disallow: /usercp.php

Disallow: /profile.php

Disallow: /vbseocp.php

Disallow: /login.php

Disallow: /member.php

Disallow: /sitemap/

Disallow: /archive/



Is this right?

[myths]

Disallow:

User-agent: *

mean robots and spider will not crawl your all .php which you mention in robots.txt