Forums database errors server shut down due to hundreds of thousands of empty /tmp files being written

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Maltair
    Senior Member
    • Feb 2009
    • 575
    • 5.7.5

    [Bug / Issue] Forums database errors server shut down due to hundreds of thousands of empty /tmp files being written

    I started getting hundreds of database errors such as the below last week on a certain day.

    My email servers and the cpanel itself shut down.

    My host looked into it, and responded as follows:

    "
    I understand you have contacted us regarding the VPS ***.com which was full on inodes.

    I found that there were over 600,000 files in /tmp, which caused the server to reach the 1,000,000 file limit. I have gone ahead and removed most of the files to allow services to come back online.

    Please check your website softwares to determine which website is generating /tmp/vbvurl* files to address the issue that filled up the VPS.
    "

    Question: Why would so many of these empty temp files be created by vbulletin forums? (They started with the two letters VB so I knew they were yours.) If so, why is/did this happen and how may I stop this from happening in the future?

    Also my host added as follows:


    The majority of the files were created within the past week.

    The normal files/folders expected in /tmp are directory .ICE-unix and symlink mysql.sock. It seems the fm*.txt are related to FormMail and vbvurl* files are related to vBulletin. Keep in mind, I am referring to the server's main /tmp and not the user's /home/*****/tmp directory (The contents of /home/*****/tmp are as expected, with nothing I do not expect to see).




    ---

    Database error in vBulletin 5.4.5:



    Invalid SQL:



    (

    SELECT

    a.filedataid,

    n.nodeid, n.parentid, n.contenttypeid,

    a.visible, a.counter, a.posthash, a.filename, a.reportthreadid, a.settings,

    NULL AS height, NULL AS width, NULL AS style,

    a.caption,

    n.displayorder

    FROM node AS n

    JOIN attach AS a ON (a.nodeid = n.nodeid)

    WHERE n.contenttypeid = 24 AND n.parentid IN (279)

    )

    UNION ALL

    (

    SELECT

    p.filedataid,

    n.nodeid, n.parentid, n.contenttypeid,

    NULL AS visible, NULL AS counter, NULL AS posthash, NULL AS filename, NULL AS reportthreadid, NULL AS settings,

    p.height, p.width, p.style,

    p.caption,

    n.displayorder

    FROM node AS n

    JOIN photo AS p ON (p.nodeid = n.nodeid)

    WHERE n.contenttypeid = 23 AND n.parentid IN (279)

    )

    ORDER BY displayorder



    /**fetchNodeAttachments**/;



    MySQL Error : Can't create/write to file '/tmp/#sql_1449_0.MYI' (Errcode: 122 - Internal (unspecified) error in handler)

    Error Number : 1

    Request Date : Friday, December 21st 2018 @ 12:37:33 PM

    Error Date : Friday, December 21st 2018 @ 12:37:33 PM

    Script : https://www.******.com/forums/forum/***/***/page2

    Referrer :

    IP Address : 23.94.XX.XX

    Username : Guest

    Classname : vB_Database_MySQLi

    MySQL Version :





    Stack Trace:

    #0 vB_Database->getErrorData() called in [path]/vb/database.php on line 1203

    #1 vB_Database->halt() called in [path]/vb/database/mysqli.php on line 201

    #2 vB_Database_MySQLi->execute_query() called in [path]/vb/database.php on line 572

    #3 vB_Database->query_read() called in [path]/vb/db/result.php on line 126

    #4 vB_dB_Result->rewind() called in [path]/vb/db/result.php on line 63

    #5 vB_dB_Result->__construct() called in [path]/vb/db/query/stored.php on line 104

    #6 vB_dB_Query_Stored->execSQL() called in [path]/vb/db/assertor.php on line 301

    #7 vB_dB_Assertor->assertQuery() called in [path]/vb/db/assertor.php on line 650

    #8 vB_dB_Assertor->getRows() called in [path]/vb/library/node.php on line 1780

    #9 vB_Library_Node->fetchNodeAttachments() called in [path]/vb/library/node.php on line 1945

    #10 vB_Library_Node->addFullContentInfo() called in [path]/vb/library/node.php on line 1418

    #11 vB_Library_Node->getFullContentforNodes() called in [path]/vb/api/search.php on line 403

    #12 vB_Api_Search->getMoreResults() called in [path]/vb/api/search.php on line 230

    #13 vB_Api_Search->getInitialResults() called in [path]/vb/api/wrapper.php on line 199

    #14 vB_Api_Wrapper->__call() called in /home/*****/public_html/forums/includes/api/interface/collapsed.php on line 101

    #15 Api_Interface_Collapsed->callApi() called in /home/*****/public_html/forums/includes/vb5/template/runtime.php on line 989

    #16 vB5_Template_Runtime:arseDataWithErrors() called in /home/*****/public_html/forums/includes/vb5/template.php(369) : eval()'d code on line 824

    #17 eval() called in /home/*****/public_html/forums/includes/vb5/template.php on line 369

    #18 vB5_Template->render() called in /home/*****/public_html/forums/includes/vb5/template/cache.php on line 134

    #19 vB5_Template_Cache->replacePlaceholders() called in /home/*****/public_html/forums/includes/vb5/template.php on line 391

    #20 vB5_Template->render() called in /home/*****/public_html/forums/includes/vb5/template/cache.php on line 134

    #21 vB5_Template_Cache->replacePlaceholders() called in /home/*****/public_html/forums/includes/vb5/template.php on line 391

    #22 vB5_Template->render() called in /home/*****/public_html/forums/includes/vb5/template/cache.php on line 134

    #23 vB5_Template_Cache->replacePlaceholders() called in /home/*****/public_html/forums/includes/vb5/template.php on line 391

    #24 vB5_Template->render() called in /home/*****/public_html/forums/includes/vb5/template/cache.php on line 134

    #25 vB5_Template_Cache->replacePlaceholders() called in /home/*****/public_html/forums/includes/vb5/template.php on line 391

    #26 vB5_Template->render() called in /home/*****/public_html/forums/includes/vb5/frontend/controller/page.php on line 259

    #27 vB5_Frontend_Controller_Page->index() called in /home/*****/public_html/forums/index.php on line 74
    Last edited by Maltair; Wed 26 Dec '18, 8:35am.
  • Wayne Luke
    vBulletin Technical Support Lead
    • Aug 2000
    • 74123

    #2
    We don't tell MySQL to write and/or delete tmp files. It manages them on its own. These files will be created during normal operations. It seems that the MySQL user doesn't have permission to delete files from the /tmp directory and that needs to be rectified.

    You cannot delete these files while MySQL is running. To resolve the database errors, you will need to restart the MYSQL server. Your hosting provider should have known to do this when deleting MySQL files.
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud demonstration site.
    vBulletin 5 API

    Comment

    • Maltair
      Senior Member
      • Feb 2009
      • 575
      • 5.7.5

      #3
      Thanks! yes I read that, since then Hostgator has added the following, what do you think? Why are there 400K files in the temp?

      "The current permissions on the /tmp directory should permit writing capabilites by MySQL."

      "
      Based on the error message reported, this issue does not seem to be permissions-related, but rather resource-related. As described earlier in the ticket, the issue is primarily related to the number of files being generated in /tmp. It is more likely that this error was caused by the container having hit a limit such as maximum number of files stored, maximum number of files opened, or a filesystem limit like maximum inodes in a single directory.
      In reviewing the server state, I found about 400k files stored in /tmp from the past few days; I've gone ahead and cleared them at this time. It's worth mentioning that after clearing them, about 5 or 6 remained that were not empty - they contain HTML so it seems possible that this is related to a caching mechanism that stores page data after rendering it but fails to reliably delete the data afterwards.

      Additionally, it may be helpful to change the container's PHP handler from CGI to SuPHP - this will cause PHP scripts to be run as the user they belong to rather than "nobody", primarily improving security but also making resource usage easier to track, especially when sites are hosted on separated cPanel users. We can do this for you if you like; it can also be done from root WHM.
      "

      Comment

      • Wayne Luke
        vBulletin Technical Support Lead
        • Aug 2000
        • 74123

        #4
        I have no idea why those files are there. They shouldn't be.

        Any temp files created by vBulletin should be unlinked (deleted) once the system is done using them. If you are using mod_php, then the user that Apache is running under would need to be able to delete files out of the /tmp directory. If you're using mod_fastcgi or another method to call PHP, then the user that calls PHP needs to be able to delete files from the directory. The software relies on the proper permissions being applied to the proper users in order for operating system calls to work properly. If the hosting provider doesn't want software writing the /tmp directory, they need to configure the proper settings in their PHP config.

        For the vbvurl files you can see why they aren't being deleted by editing /core/vb/vburl.php file and removing the @ in front of all the unlink() calls. This will cause the call to trigger an error if it can't delete the file.

        The same goes for MYI and other files with MySQL. That user has to have permissions to manage the files at the operating system level.

        Last edited by Wayne Luke; Wed 26 Dec '18, 7:06pm.
        Translations provided by Google.

        Wayne Luke
        The Rabid Badger - a vBulletin Cloud demonstration site.
        vBulletin 5 API

        Comment

        • Maltair
          Senior Member
          • Feb 2009
          • 575
          • 5.7.5

          #5
          This is how my host responded:

          "
          Based on the error message reported, this issue does not seem to be permissions-related, but rather resource-related. As described earlier in the ticket, the issue is primarily related to the number of files being generated in /tmp. It is more likely that this error was caused by the container having hit a limit such as maximum number of files stored, maximum number of files opened, or a filesystem limit like maximum inodes in a single directory.
          In reviewing the server state, I found about 400k files stored in /tmp from the past few days; I've gone ahead and cleared them at this time. It's worth mentioning that after clearing them, about 5 or 6 remained that were not empty - they contain HTML so it seems possible that this is related to a caching mechanism that stores page data after rendering it but fails to reliably delete the data afterwards.

          Additionally, it may be helpful to change the container's PHP handler from CGI to SuPHP - this will cause PHP scripts to be run as the user they belong to rather than "nobody", primarily improving security but also making resource usage easier to track, especially when sites are hosted on separated cPanel users. We can do this for you if you like; it can also be done from root WHM.
          "

          1) Should I change PHP handler from CGI to SuPHP? I don't know that this will change anything, but may make the issue easier to track.

          2) Why is this happening? 400K more /tmp files stored in past few days?


          Part of why I am posting this publicly...is to determine if this has happened to anyone else?
          Last edited by Maltair; Thu 27 Dec '18, 9:40am.

          Comment

          • Wayne Luke
            vBulletin Technical Support Lead
            • Aug 2000
            • 74123

            #6
            1) If the PHP user cannot delete files out of the TMP directory then changing does nothing. This is a personal choice on your part.

            2) Answered in Post #4.

            Deleting files is definitely a permissions issue or server configuration issue.
            Translations provided by Google.

            Wayne Luke
            The Rabid Badger - a vBulletin Cloud demonstration site.
            vBulletin 5 API

            Comment


            • Wayne Luke
              Wayne Luke commented
              Editing a comment
              The other option would be the unlink function is disabled in PHP. If this is disabled, then tmp files cannot be deleted and they will need to be managed manually.
          • Maltair
            Senior Member
            • Feb 2009
            • 575
            • 5.7.5

            #7
            The host says that "The PHP: unlink is enabled and functioning. "

            Comment

            • Wayne Luke
              vBulletin Technical Support Lead
              • Aug 2000
              • 74123

              #8
              There is a disconnect somewhere. I can't create a large number of temp files on my test server, no matter what I try. They are deleted when the system is done with them.
              Translations provided by Google.

              Wayne Luke
              The Rabid Badger - a vBulletin Cloud demonstration site.
              vBulletin 5 API

              Comment

              Related Topics

              Collapse

              Working...