Spiders crawling member URL's

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Fleiding
    Senior Member
    • Jan 2012
    • 152
    • 5.3.x

    Spiders crawling member URL's

    All spiders are crawling member URL's like this:
    Code:
    /member/200-brent?200-Brent=&tab=visitor_messaging&page=150
    Is there a way to only let them crawl /member? Not quite sure what to put in my robots.txt.
  • Wayne Luke
    vBulletin Technical Support Lead
    • Aug 2000
    • 74132

    #2
    Disallow: /*?*

    See: https://stackoverflow.com/questions/...fic-parameters
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud demonstration site.
    vBulletin 5 API

    Comment

    • Fleiding
      Senior Member
      • Jan 2012
      • 152
      • 5.3.x

      #3
      Thanks Wayne!

      Comment

      widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
      Working...