HTTP 500 errors with /ajax/api/cron/run, some scheduled jobs not running

[sauloon]

Hi,

Some background. I am running

• vBulletin 5.1.1
• Apache 2.2.27
• PHP 5.3.28
• MySQL 5.6.17
• Amazon Linux (latest)

On my site, some scheduled tasks are shown correctly on AdminCP -> Scheduled Task Manager (next run
is in the future) but when I check the AdminCP -> Scheduled Task Log, it shows that the tasks have never been
run (log empty). My site has about 50 page views per minute, so, there are enough user generated requests to
run the VB cron. The following are the tasks affected.
- Daily Cleanup
- Daily Statistics Log
- Hourly Cleanup
- Hourly Cleanup #2
- Infractions Cleanup
- queueprocessor
- Restore Temporarily Banned Users
- Scheduled Unattended Publish

The strange thing is that other scheduled tasks are running fine and shows up in the log.

Given the above, I checked AdminCP -> Statistics & Logs -> Statistics -> Registration Statistics and indeed,
the statistics are not updated even though I have new users who have been verified and moderated.

I checked the Apache access logs and saw that a number of /ajax/api/cron/run are returning with 500 errors.
Most of them are 200 OK though. Correlating the time of the 500 errors with the error logs (where PHP error
log is written), I do not see any PHP errors there.

I am running 2 boards, one is an upgrade from 3.8.2 to 5.1.1 and the other is an upgrade of 5.1.0 to 5.1.1.
Both exhibit the same behavior and symptoms.

I am at a lost as to how to proceed further. Any help is appreciated. In particular, if you are running 5.1.1 with
PHP 5.3.28, do you also notice that the above scheduled tasks do not run and do not show up in the Scheduled
Task Log ?

Thanks in advance for any help!

regards,
sl

[Wayne Luke]

Do you have Mod_Security or Suhosin installed on the server? Are you using any hardened form of PHP? What are the specific entries in the error log?

[sauloon]

Hi,

Do you have Mod_Security or Suhosin installed on the server? Are you using any hardened form of PHP? What are the specific entries in the error log?

Mod_Security was already disabled a few days back and the problem is still there. I do not have Suhosin installed and I have not hardened the PHP. I just used the stock Amazon Linux PHP package.

For the most recent 500 error, here's the access log line

92.12.153.89 - - [13/Jun/2014:17:10:21 +0000] "POST /vb/ajax/api/cron/run HTTP/1.1" 500 4386 <referrer omitted> "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36"

and these are the most recent error logs i.e. there is no corresponding error log entry for the above line.

[Fri Jun 13 17:08:52 2014] [error] [client 27.150.230.12] File does not exist: <documentroot>/js
[Fri Jun 13 17:14:54 2014] [error] [client 216.117.130.243] script '<documentroot>/wp-login.php' not found or unable to stat, referer: http://www.google.com/

I also found some of the following PHP warnings and notices in the error log, but there are no corresponding 500 errors in the access logs.

[Fri Jun 13 17:01:01 2014] [error] [client 54.76.37.94] PHP Warning: include_once(<documentroot>/vb/core/./includes/cron/queueprocessor.php): failed to open stream: No such file or directory in <documentroot>/vb/core/includes/functions_cron.php on line 250, referer: <vb-base-url>

[Fri Jun 13 17:01:01 2014] [error] [client 54.76.37.94] PHP Warning: include_once(): Failed opening '<documentroot>/vb/core/./includes/cron/queueprocessor.php' for inclusion (include_path='.:/usr/share/pear:/usr/share/php') in <documentroot>/vb/core/includes/functions_cron.php on line 250, referer: <vb-base-url>

[Fri Jun 13 17:04:09 2014] [error] [client 178.162.199.101] PHP Notice: Undefined offset: 1 in <documentroot>/vb/core/vb5/route/privatemessage/list.php on line 34

[Fri Jun 13 17:21:01 2014] [error] [client 54.76.37.94] PHP Warning: include_once(<documentroot>/vb/core/./includes/cron/queueprocessor.php): failed to open stream: No such file or directory in <documentroot>/vb/core/includes/functions_cron.php on line 250, referer: <vb-base-url>

[Fri Jun 13 17:21:01 2014] [error] [client 54.76.37.94] PHP Warning: include_once(): Failed opening '<documentroot>/vb/core/./includes/cron/queueprocessor.php' for inclusion (include_path='.:/usr/share/pear:/usr/share/php') in <documentroot>/vb/core/includes/functions_cron.php on line 250, referer: <vb-base-url>

[Fri Jun 13 17:23:36 2014] [error] [client 66.249.69.55] PHP Notice: Undefined index: lastactivity in <documentroot>/vb/core/includes/functions_bigthree.php on line 138

[Fri Jun 13 17:23:36 2014] [error] [client 66.249.69.55] PHP Notice: Undefined index: 642 in /var/www/www.newturfers.com/html/vb/core/vb/library/node.php on line 1362

Appreciate your help. Thanks!

regards,
sl

[Wayne Luke]

Hmmm.. The dot in the file name is probably the issue. Are you using open_basedir in PHP?

[sauloon]

Hmmm.. The dot in the file name is probably the issue. Are you using open_basedir in PHP?

open_basedir has "No Value" on the AdminCP -> View PHP Info. Does this mean I am using it or not?

I also did a find to look for the file. There was no such file in the folder tree. I unzip a fresh distribution copy
of 5.1.1 and also did not find the file.

$ sudo find . -name queue\*
./vb/core/vb/mail/queue.php



regards,
sl

[sauloon]

Hi,

One more thing. When the script runs into a 500 error, it probably returns some error string. In the case of my access log line in my earlier post, this was 4386 bytes. However, there is no way I can see the error message from the browser.

regards,
sl

[stcorp]

I too am experiencing hit and miss activity with vBulletin scheduled tasks. Off the top of my head, I have to run User Promotions as well as Daily Digest, but I am guessing none of the scheduled tasks are working. If I look at the Scheduled Task Log, all activities are logged as run, but this is incorrect as I never receive the Daily Digest until I manually force it, and my users occasionally notify me when their titles have not been promoted.

In the Scheduled Task Manager, all tasks are checked to run, Save Enabled Status has been checked several times and I have even tried rescheduling a job, all with no effect. Is there anything I can check to debug this? Is it possible to manually call /<path>/<to>/core/includes/cron/digestdaily.php (or any other php cron) from the system crontab and bypass vBulletin's scheduler completely?

We are running vBulletin 5.1.1 with PHP 5.4.1.6.

-Pete

[Wayne Luke]

Delete the scheduled task calling queueprocessor.php.

[Jeanius!]

Hi!
What was the solution to this?
I have this happening on my 5.2.2 with the same strange dot in the path.

[20-Jul-2016 20:20:18 UTC] PHP Warning: include_once(/home/<removed>/public_html/forums/core/./includes/cron/queueprocessor.php): failed to open stream: No such file or directory in /home/<removed>/public_html/forums/core/includes/functions_cron.php on line 250
[20-Jul-2016 20:20:18 UTC] PHP Warning: include_once(): Failed opening '/home/<removed>/public_html/forums/core/./includes/cron/queueprocessor.php' for inclusion (include_path='.:/opt/alt/php70/usr/share/pear') in /home/<removed>/public_html/forums/core/includes/functions_cron.php on line 250

The scheduled task that is calling queueprocessor.php has the same relative file path that the other scheduled tasks have (./includes/cron/queueprocessor.php), so not sure what's happening here.