Sitemap Malformed

[DisasterDotCom]

My sitemaps are being created through the vBulletin cron but I'm getting errors in the sitemap on a regular basis - every other generation, if not every generation, seems to have problems unless I manually create it. A failed generation file is about 2k. A fully formed file is about 181k.

It looks like it's the starter file. It's way too small, and the end of it looks like this:

<url>
<loc>http://www.disaster.com/forums/forum/types-of-disasters/personal-disasters/293-hacked</loc>
<lastmod>2014-07-30T15:26:15+00:00</lastmod>
<chan
</urlset>

You can see the unclosed <changefreq> tag, and the output has been truncated, yet the script still finishes enough to put the </urlset> tag on the end.

Any ideas?

[Wayne Luke]

Try lowering the number of URLS Per Page under Settings -> Options -> XML Sitemap

[DisasterDotCom]

Alright.. I've set it to 500 (from 30,000). I'm not sure if that's it though since (I should have mentioned) the starter file, when it's broken, only has 20-30 entries before it truncates. We'll see how it goes.

BTW - support through this forum is now officially faster than support by opening up a ticket... I opened up a ticket at the same time as posting this in the forum, and I still haven't heard back anything on it.

[Mark.B]

Can sometimes work that way depending on who is about and what they're working on.
Tickets will usually be faster though, as a rule.

Personally, I make regular scans of recent forum posts and if something needs a quick answer, I'll post it.

[DisasterDotCom]

Didn't make a difference.. now I have two corrupt files.

BUT.. here's the commonality between them. They are both exactly 9499 characters in length, which means both of them are cutting off at 9,490 characters before adding a closing </urlset> tag.

And, once again, if I manually create the files, they work fine.

[webcms]

We had an issue with sitemaps earlier when special characters like ™™ were included in the post title and Google did not like them.

[DisasterDotCom]

Thanks webcms. In our case, the sitemaps aren't being fully generated.

BTW - I've increased the PHP timeout and memory size to see if that makes a difference. We'll see tonight.

[Wayne Luke]

Your sitemap script is taking more time to generate than is allotted in PHP resulting in these errors:

[25-Aug-2014 02:59:13 America/New_York] PHP Fatal error: Maximum execution time of 30 seconds exceeded in /forums/core/vb5/route.php on line 163
[25-Aug-2014 10:00:26 America/New_York] PHP Fatal error: Maximum execution time of 30 seconds exceeded in /forums/core/vb5/route.php on line 163

We've done some optimization to route generation in 5.1.3. This may resolve the problem. It may not.

You can try increasing the maximum execution time. To test this you can edit ./core/includes/cron/sitemap.php and add a line under:

error_reporting(E_ALL & ~E_NOTICE);

The line to be added would be:

ini_set('max_execution_time', 60);

You can play with the 60 to see what number works for you. Would try 10,000 entries per file. If you get an error, the host may have disabled this function so you would need to contact them about a workaround.

[DisasterDotCom]

I'm hosting it on my own cloud server so I have access to do whatever I want... always handy!

I've set the timeout to 360 in the global Apache PHP ini for the moment... the server isn't heavily loaded (yet). Should see how it runs tonight.

[DisasterDotCom]

Timeout at 360 and memory at 256M increased file size for both to 10166 characters and 10170 characters (starter_1 and starter_2) but they are still truncated. I added 192M (for a total of 256M) of mem and 5.5 minutes (for a total of 6 minutes) of execution timeout for an extra 100 characters... so I'm not thinking that approach is scalable. What should we try next? I just looked in Google Webmaster Tools and my site is flatlining with regards to growth because new threads aren't being indexed - and I'm paying A LOT of money for new content right now.

[DisasterDotCom]

BTW - file sizes are both 9928 on my Ubuntu system when uncompressed.

[Lynne]

And are you still getting errors in the error_logs? What are the exact errors now?

[DisasterDotCom]

The php.log has no timeouts in it or errors even close to the time the files were generated this morning. The apache error log is empty, and the access log shows 200's for all vb5 crons. Support has told me the following at this point:

"At this time there's not a huge amount more we can really suggest"

"As Wayne's mentioned, there are some optimisations in 5.1.3 which may help, but we can't say for certain."

"The support team aren't coders. We can't rewrite or debug parts of the software."

"I'm not aware of any other sites that have run into this...it may be worth logging your findings in the tracker, in case there is an issue with the code."

I'm a bit surprised at this type of response as I'm used to support teams escalating problems to senior support, and then to developers when they run out of options - or at least walking the customer through SOME debugging and testing to try to narrow down where the problem is. I've ONLY been told to increase my memory and timeouts, as a guess, and then I'm being told that nothing more can be done.

If you go through my prior threads you'll see that I've been a champion for vB5 in my short time working with it. I think it's a great product that's evolving and has some features that really set it apart from the competition. HOWEVER, walking away from customer support is a HUGE issue. I would expect a company that is challenged by so many people to say "Let me show you how we can go above and beyond for you to try to fix your problems" as opposed to a more "Not my problem" approach.

And this could absolutely be something on my side... although the server install is pretty stock with almost no tweaks. But I NEED to find the problem.

So, </rant> and </vent> off.. please help me debug this

BTW - just to reiterate - when I click on manual sitemap creation through the control panel it completes successfully. It's through vb5 cron that it's not. I have two questions, in addition to hoping you can throw some ideas my way...

1. How can I manually trigger a sitemap creation from the command line for the time being so that, upon failure, I can at least have a valid sitemap triggered through the system cron
2. What is the difference between the cron sitemap builder and the manual sitemap builder? Somewhere in there is the problem (most likely).

Thanks!

[DisasterDotCom]

Apparently while I was ranting, I got an email:

"I'll escalate your ticket to one of the senior technicians to see if anything else can be suggested."

So that's a start

BUT all help would be greatly appreciated.

[Wayne Luke]

I've followed up in the ticket. There is nothing in the code that would limit the file size. Normally the files are around 50K each with the default settings.