Millions of page hits. Why?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • TimothyJMcGowan
    New Member
    • Dec 2017
    • 6
    • 5.3.x

    Millions of page hits. Why?

    I've cribbed this query from a SiteGround support tech who escalated my problem there. I'm asking it here in the hope that this is a common enough problem that you can spot the solution.

    I am having issues with CPU usage on forumania.com. In the AW stats, 4k unique visits but over 1 million page hits. I tried blocking some of the IPs with the greatest number of hits and ended up blocking our own actual users. There is little evidence of bots doing this, and more importantly, no reason our actual users should be hitting the site this hard.

    What might we have misconfigured?

    For what it's worth, we started as a vBulletin Cloud customer and moved to our own server.

    Copied from sticky topic:
    In order to get your questions answered quickly please provide the following information:
    vBulletin Version: 5.3.4
    PHP Version: 7.1.13
    MySQL Version: Unknown (directions to obtain version are out of date)
    Any Addons installed: Only default in our case: vB Cloud Admin and VigLink
    Does the issue occur in a default style? N/A
    Does the issue occur using the English language provided? N/A
    Error message on the screen: N/A
    Browser and Browser version used. N/A
    Did you clear the browser cache and did the error continue? N/A
    A list of steps that can be used to recreate the issue. N/A
    If the issue is an Invalid Server response, the web server and PHP logs that correspond with its timestamp. N/A

    You can find your MySQL version under Maintenance -> Diagnostics -> Server Variables -> MySQL Settings (Perhaps you can. I cannot.)
    You can find your PHP version in the AdminCP under Maintenance -> View PHPInfo
    You can find your addons in the Admin CP under Hooks & Products -> Manage Products.
  • Wayne Luke
    vBulletin Technical Support Lead
    • Aug 2000
    • 73981

    #2
    Sounds like DDOS. Did you piss anyone off lately? People can buy a denial of Service Attack online. What are the user agents and IP addresses of these visitors?
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud demonstration site.
    vBulletin 5 API

    Comment

    • TimothyJMcGowan
      New Member
      • Dec 2017
      • 6
      • 5.3.x

      #3
      That's the point: The highest hitters were our own actual users. I know it sounds like DDOS. That's why I started just blocking IPs as a stopgap measure until I could figure out what was going on. And then our own users, who are decidedly not trying to bring us down, especially so many of them, started complaining of a 403 error, if I remember the number correctly. They couldn't get to the site at all. We've blocked a few Russian IPs, but again, they're not the ones flogging the server.

      So I really think something has been misconfigured, either because of our move from Cloud to self-hosted or because we went with defaults that we should have somehow known we should have set. For instance, we're not using memcached, which it seems might help in general, if not for this issue.

      It has been a few days since I last looked at our activity, so I'll go through that all again tonight. But I'll be very careful to assess whether an "attacker" is actually a logged-in, in-good-standing user this time.

      Comment

      • Wayne Luke
        vBulletin Technical Support Lead
        • Aug 2000
        • 73981

        #4
        Settings in vBulletin won't increase the actual traffic of your site or cause IP Addresses to make more connections. A client of some sort has to be doing it.

        You have to actually look at the logs from the web server and determine which IPs are causing the most traffic and where they are originating from. This is not something that can be done reliably within the vBulletin software. Especially since multiple devices can share the same IP address.
        Translations provided by Google.

        Wayne Luke
        The Rabid Badger - a vBulletin Cloud demonstration site.
        vBulletin 5 API

        Comment

        • TimothyJMcGowan
          New Member
          • Dec 2017
          • 6
          • 5.3.x

          #5
          Does this help? I compared the IP addresses shown in CPanel's AWStats with the IP addresses vBulletin logged when the users signed up. (We opened less than two months ago, so the IP addresses used at sign-up are still fairly reliable.)

          These are this month's stats, from the 1st through the 5th.

          As I write this, vBulletin shows that we have 2,113 topics and 31,200 posts. These stats show that the top user, for example, has pulled up three times as many pages as we have topics and has nearly four times as many hits. And she's an idiot. She's not scraping our site or intentionally damaging us. Same, basically, with the rest. They're just users with no other life, and they love to talk politics and religion and to find artful ways to call each other names or fling invective without abridging the forum rules.

          99.8.158.211
          Pages: 6,995
          Hits: 7,695
          Bwidth: 38.62 MB
          Verified prolific user

          69.201.85.146
          Pages: 5,232
          Hits: 6,243
          Bwidth: 35.65 MB
          Verified prolific user

          76.98.124.175
          Pages: 4,819
          Hits: 5,203
          Bwidth: 54.49 MB
          Verified prolific user

          66.172.238.157
          Pages: 4,297
          Hits: 4,458
          Bwidth: 21.53 MB
          Verified prolific user

          108.216.225.185
          Pages: 3,987
          Hits: 4,662
          Bwidth: 44.60 MB
          Verified prolific user

          I don't have time tonight to go through the rest -- but if anyone's attacking us, they're not working at it nearly as hard as our busiest real users, who are posting actual, apposite messages in our forums.

          Comment

          • Wayne Luke
            vBulletin Technical Support Lead
            • Aug 2000
            • 73981

            #6
            Are you sure those are pages and not requests? There is a difference. Each page will have multiple requests for the server. There is the HTML code, Javascript files, CSS files, image files, etc... Each attachment and module on the page can also add additional requests to get data and information from the server.

            Do you have browser caching turned off in the software or on the server level?
            Translations provided by Google.

            Wayne Luke
            The Rabid Badger - a vBulletin Cloud demonstration site.
            vBulletin 5 API

            Comment

            • TimothyJMcGowan
              New Member
              • Dec 2017
              • 6
              • 5.3.x

              #7
              Copied and pasted from the AW Stats report in CPanel, which simply seems to call them pages, not requests. I'm confident you know better than I what it actually means; I know what it says: pages.

              Again, we simply copied our Cloud service to our new server. Thanks for the tip about browser caching. That's the feature through which the individual's Web browser downloads all the pages for all the links on the page the user is looking at just in case the user clicks on any link on the page, all in order to speed up the page load; right? That could certainly slam the server. I'll look for those controls in the software, and then I'll see what I can do to control that through CPanel as well. I'll report back any result, whether effective or not.

              Comment

              • Wayne Luke
                vBulletin Technical Support Lead
                • Aug 2000
                • 73981

                #8
                You can tell vBulletin to send headers to the user's browser to ignore caching. We do not recommend this. The setting is under Settings -> Options -> Cookie and HTTP Header Options. It is called "Add No-Cache HTTP Headers". This should be set to Yes.
                Translations provided by Google.

                Wayne Luke
                The Rabid Badger - a vBulletin Cloud demonstration site.
                vBulletin 5 API

                Comment

                • TimothyJMcGowan
                  New Member
                  • Dec 2017
                  • 6
                  • 5.3.x

                  #9
                  Did you pick the wrong word? Surely that should be set to No; right?

                  From the text on the page in Settings:
                  Add No-Cache HTTP Headers
                  Selecting yes will cause vBulletin to add no-cache HTTP headers. These are very effective, so adding them may cause server load to increase due to an increase in page requests.

                  From the help text for that item:
                  If you enable this option, no-cache HTTP headers will be added to each page. These cause full page data to be re-requested from the server every time the user gets a page, which will increase bandwidth; it may additionally cause an increase in server load as it will cause more pages to be served.

                  And I think our .htaccess file is set properly for caching:
                  DirectoryIndex ForumCenter.htm
                  <IfModule mod_rewrite.c>
                  RewriteEngine On

                  # In some cases where you have other mod_rewrite rules, you may need to remove the
                  # comment on the following RewriteBase line and change it to match your folder name.
                  # This resets the other mod_rewrite rules for just this directory
                  # If your site was www.example.com/forum, the setting would be /forum/
                  RewriteBase /

                  #To redirect users to the secure version of your site, uncomment the lines below
                  #RewriteCond %{HTTPS} !=on
                  #RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

                  # Send css calls directly to the correct file VBV-7807
                  RewriteRule ^css.php$ core/css.php [NC,L]

                  # Redirect old install path to core.
                  RewriteRule ^install/ core/install/ [NC,L]

                  # Main Redirect
                  RewriteCond %{REQUEST_URI} !\.(gif|jpg|jpeg|png|css)$
                  RewriteCond %{REQUEST_FILENAME} !-f
                  RewriteCond %{REQUEST_FILENAME} !-d
                  RewriteRule ^(.*)$ index.php?routestring=$1 [L,QSA]

                  # Because admincp is an actual directory.
                  RewriteRule ^(admincp/)$ index.php?routestring=$1 [L,QSA]

                  </IfModule>

                  <IfModule mod_deflate.c>
                  AddOutputFilterByType DEFLATE application/atom+xml \
                  text/javascript \
                  application/x-javascript \
                  application/javascript \
                  application/json \
                  application/rss+xml \
                  application/vnd.ms-fontobject \
                  application/x-font-ttf \
                  application/xhtml+xml \
                  application/xml \
                  font/opentype \
                  image/svg+xml \
                  image/x-icon \
                  text/css \
                  text/html \
                  text/plain \
                  text/x-component \
                  text/xml
                  </IfModule>

                  <IfModule mod_expires.c>
                  ExpiresActive On
                  ExpiresByType application/x-javascript A1209600
                  ExpiresByType text/javascript A1209600
                  ExpiresByType application/javascript A1209600
                  ExpiresByType text/css A31536000
                  ExpiresByType image/x-icon A2592000
                  ExpiresByType image/icon A2592000
                  ExpiresByType application/x-ico A2592000
                  ExpiresByType application/ico A2592000
                  ExpiresByType image/gif A2592000
                  ExpiresByType image/jpeg A1209600
                  ExpiresByType image/jpg A1209600
                  ExpiresByType image/png A1209600
                  ExpiresByType application/x-shockwave-flash A1209600
                  ExpiresByType font/ttf A2592000
                  ExpiresByType font/otf A2592000
                  ExpiresByType font/x-woff A2592000
                  ExpiresByType image/svg+xml A2592000
                  ExpiresByType font/truetype A2592000
                  ExpiresByType font/opentype A2592000
                  ExpiresByType application/x-font-woff A2592000
                  ExpiresByType application/vnd.ms-fontobject A2592000
                  </IfModule>

                  <IfModule mod_headers.c>
                  Header set Connection keep-alive
                  <filesmatch "\.(ico|flv|gif|swf|eot|woff|otf|ttf|svg)$">
                  Header set Cache-Control "max-age=2592000, public"
                  </filesmatch>
                  <filesmatch "\.(jpg|jpeg|png)$">
                  Header set Cache-Control "max-age=1209600, public"
                  </filesmatch>
                  <filesmatch "\.(eot|woff|otf|ttf|svg)$">
                  Header set Cache-Control "max-age=2592000, public"
                  </filesmatch>
                  # css and js should use private for proxy caching https://developers.google.com/speed/...geProxyCaching
                  <filesmatch "\.(css)$">
                  Header set Cache-Control "max-age=31536000, private"
                  </filesmatch>
                  <filesmatch "\.(js)$">
                  Header set Cache-Control "max-age=1209600, private"
                  </filesmatch>
                  </IfModule>

                  #don't allow some files that shouldn't really be present to be directly accessed.
                  #note that attachements should never be directly accessed by the webserver because
                  #we have permissions on the that are checked in the PHP code.
                  <FilesMatch "(^#.*#|\.(bak|config|dist|inc|ini|log|gz|tar|zip|sh|sql|sw[op])|~)$">
                  Order allow,deny
                  Deny from all
                  Satisfy All
                  </FilesMatch>

                  AddHandler application/x-httpd-php71 .php .php5 .php4 .php3

                  <Files 403.shtml>
                  order allow,deny
                  allow from all
                  </Files>

                  Any other thoughts?

                  We don't have memcached enabled. Should I bring that up in a different thread, or can we deal with it here? Or is there a FAQ I missed on that for 5.3.4?

                  -- Tim

                  Comment

                  • Wayne Luke
                    vBulletin Technical Support Lead
                    • Aug 2000
                    • 73981

                    #10
                    I don't know why you're getting millions of hits. The software appears to be running properly on your site. Are you running Google Analytics on your site? If not, you should as that will give you a better view of the traffic since you don't seem to have access to the raw Apache logs.

                    Apache's raw access logs look like this:
                    Code:
                    ::1 - - [08/Feb/2018:09:19:24 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/cp_logo.png HTTP/1.1" 200 5860
                    ::1 - - [08/Feb/2018:09:19:24 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/admincp_modhd_back.png HTTP/1.1" 200 99
                    ::1 - - [08/Feb/2018:09:19:16 -0800] "GET /vb5_live/admincp/index.php HTTP/1.1" 200 11341
                    ::1 - - [08/Feb/2018:09:19:25 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/admincp_btn_secondary.png HTTP/1.1" 200 85
                    ::1 - - [08/Feb/2018:09:19:26 -0800] "POST /vb5_live/login.php?do=login HTTP/1.1" 200 465
                    ::1 - - [08/Feb/2018:09:19:31 -0800] "GET /vb5_live/admincp/index.php HTTP/1.1" 200 1346
                    ::1 - - [08/Feb/2018:09:19:34 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/admincp_hd_back.png HTTP/1.1" 200 155
                    ::1 - - [08/Feb/2018:09:19:34 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/cp_expand_ltr.png HTTP/1.1" 200 241
                    ::1 - - [08/Feb/2018:09:19:34 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/admincp_lnav_btn_default.png HTTP/1.1" 200 106
                    ::1 - - [08/Feb/2018:09:19:33 -0800] "GET /vb5_live/admincp/index.php?do=head HTTP/1.1" 200 9466
                    ::1 - - [08/Feb/2018:09:19:33 -0800] "GET /vb5_live/admincp/index.php?do=nav HTTP/1.1" 200 45635
                    ::1 - - [08/Feb/2018:09:19:36 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/cp_help.png HTTP/1.1" 200 595
                    ::1 - - [08/Feb/2018:09:19:33 -0800] "GET /vb5_live/admincp/index.php?do=home HTTP/1.1" 200 22376
                    ::1 - - [08/Feb/2018:09:19:37 -0800] "GET /vb5_live/core/clientscript/vbulletin_cphome_scripts.js?v=533 HTTP/1.1" 200 4703
                    ::1 - - [08/Feb/2018:09:19:37 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/admincp_modtitle_back.png HTTP/1.1" 200 113
                    ::1 - - [08/Feb/2018:09:19:37 -0800] "POST /vb5_live/admincp/newsproxy.php HTTP/1.1" 200 2559
                    127.0.0.1 - - [08/Feb/2018:09:28:05 -0800] "GET /vb5_live/core/cpstyles/global.css?v=534 HTTP/1.1" 200 3277
                    ::1 - - [08/Feb/2018:09:28:05 -0800] "GET /vb5_live/core/cpstyles/vBulletin_5_Default/controlpanel.css?v=534 HTTP/1.1" 200 12416
                    ::1 - - [08/Feb/2018:09:28:00 -0800] "GET /vb5_live/admincp/index.php HTTP/1.1" 200 7914
                    ::1 - - [08/Feb/2018:09:28:55 -0800] "GET /vb5_live/admincp/index.php HTTP/1.1" 200 1346
                    ::1 - - [08/Feb/2018:09:28:55 -0800] "GET /vb5_live/admincp/index.php?do=head HTTP/1.1" 200 9466
                    ::1 - - [08/Feb/2018:09:28:55 -0800] "GET /vb5_live/admincp/index.php?do=nav HTTP/1.1" 200 45635
                    ::1 - - [08/Feb/2018:09:28:55 -0800] "GET /vb5_live/admincp/index.php?do=home HTTP/1.1" 200 23005
                    ::1 - - [08/Feb/2018:09:28:57 -0800] "GET /vb5_live/core/clientscript/vbulletin_cphome_scripts.js?v=534 HTTP/1.1" 200 4703
                    ::1 - - [08/Feb/2018:09:28:57 -0800] "POST /vb5_live/admincp/newsproxy.php HTTP/1.1" 200 2559
                    ::1 - - [08/Feb/2018:09:29:17 -0800] "GET /vb5_live/admincp/index.php HTTP/1.1" 200 1346
                    ::1 - - [08/Feb/2018:09:29:18 -0800] "GET /vb5_live/admincp/index.php?do=head HTTP/1.1" 200 9466
                    ::1 - - [08/Feb/2018:09:29:18 -0800] "GET /vb5_live/admincp/index.php?do=nav HTTP/1.1" 200 45635
                    ::1 - - [08/Feb/2018:09:29:18 -0800] "GET /vb5_live/admincp/index.php?do=home HTTP/1.1" 200 23005
                    ::1 - - [08/Feb/2018:09:29:19 -0800] "POST /vb5_live/admincp/newsproxy.php HTTP/1.1" 200 2559
                    ::1 - - [08/Feb/2018:09:31:04 -0800] "GET /vb5_live/admincp/index.php HTTP/1.1" 200 1346
                    ::1 - - [08/Feb/2018:09:31:08 -0800] "GET /vb5_live/admincp/index.php?do=head HTTP/1.1" 200 9480
                    127.0.0.1 - - [08/Feb/2018:09:31:08 -0800] "GET /vb5_live/admincp/index.php?do=nav HTTP/1.1" 200 45635
                    ::1 - - [08/Feb/2018:09:31:08 -0800] "GET /vb5_live/admincp/index.php?do=home HTTP/1.1" 200 23047
                    ::1 - - [08/Feb/2018:09:31:10 -0800] "POST /vb5_live/admincp/newsproxy.php HTTP/1.1" 200 2559
                    MemcacheD has nothing to do with the number of page views. It just makes page views faster.
                    Translations provided by Google.

                    Wayne Luke
                    The Rabid Badger - a vBulletin Cloud demonstration site.
                    vBulletin 5 API

                    Comment

                    Related Topics

                    Collapse

                    Working...