Huge posts seem to kill rebuild searchindex

[Stadler]

I'm rebuilding the searchindex atm.

Well ... I had some problems with one post. It always stopped immediately after starting to reindex this post, so I've checked the post through phpMyAdmin and I found out, that it's ~36k chars long

Well, anyway. That seems to be a bug, that should be fixed.

btw: If someone is having the same problem and wants to check the postlenght, he should run the following query:

SELECT LENGTH(pagetext) AS postlength
FROM post
WHERE postid=xxxxxxx;

mmh ... never mind. Seems to be an issue with memory overload, though I don't get any errors

[Fallout2man]

It did the same thing on 2.x.x for me, I just ended up having to delete the long post (it was just spam anyway) to rebuild the search index.

[Simms]

I'm having the same problem...though mine's "only" 15k characters. Unfortunately, it's NOT spam, and I'd just as soon not have to delete it.

Is there any other way around this?

[Simms]

Well, in the meantime, I worked around it be restarting the rebuild process and specifying the next postid to begin with. (ie, if post #2284 was causing the hang, I entered #2285 in "Post ID to Start at"). At least that way, I can still get the index rebuilt, albeit with a couple of posts not included.

Still better than not rebuilding.

[poolking]

I think this question has already been raised.

[Marc Smith]

Well, in the meantime, I worked around it be restarting the rebuild process and specifying the next postid to begin with. (ie, if post #2284 was causing the hang, I entered #2285 in "Post ID to Start at"). At least that way, I can still get the index rebuilt, albeit with a couple of posts not included.

Still better than not rebuilding.

My problem with your logic is this.... What will this do in the future? Will it now be that you always have to skip this (or many) post(s) in an index rebuild? Not to mention conflicts / repercussions in other functions in the future.

I want to know why it is happening - not just how to get around it. I want no problems in the future and would rather fix the problem now. To do that, I need to know the root cause.

[fury]

I experienced the same thing on my local server when rebuilding the search index. The same thing did not happen when I did it on the live site.

It might have something to do with packet sizes but I doubt it, since what crashed was php.exe.

On my server, the MySQL Max Packet size is 1mb, on the live server, it is 16

[Marc Smith]

I'll have to look at this, but I'm a chicken. I want some more feedback before I 'COMMIT' to upgrade my live server. I DO appreciate your response. It may be a matter of my Mac not being able to digest something.

[Simms]

I'm not saying my logic is foolproof...*I* want to know why it's happening, too.

I just used that as a workaround to get my forum live. The risk of it causing long-term problems seemed pretty low, so it didn't bother me. So I have 6 or 7 posts out of 75,000 that don't get queried on a search...I can live with that, personally. That said, there's nothing wrong with proceeding tentatively.

[Mike Sullivan]

It always stopped immediately after starting to reindex this post, so I've checked the post through phpMyAdmin and I found out, that it's ~36k chars long !

What exactly do you mean "it stopped"? PHP crashed? Or did you get some sort of error message (out of memory, max execution time issues)?

[Simms]

What exactly do you mean "it stopped"? PHP crashed? Or did you get some sort of error message (out of memory, max execution time issues)?

No messages at all on mine...just a blank status bar and no progress bar with Post #xxxxx: ... on the main page itself.

[Russell]

What exactly do you mean "it stopped"? PHP crashed? Or did you get some sort of error message (out of memory, max execution time issues)?

Mine does this aswell - and when indexing 450,000 posts, it doesn't help having it stop once every 5,000....

The page stops with the message of what post is WAS up to...
status bar just says "done.".

Used to do this with 2.x.x aswell... a solution would REALLY be nice, because I haven't been able to rebuild search index completely for a long time.

[fury]

Can you enter that postid in the edit screen and save it without making any changes, and have it successfully return you to the post? This would mean the post was successfully indexed (every post is reindexed on edit) - some posts that crashed my server could be indexed by this method if they weren't too big.

Try it

http://www.yoursite.com/forum/editpo...postid=[insert post id that it stopped at here]

[Stadler]

Same goes for me. It just stops, displaying no error-messages and the main reason doesn't seem to be, that the post is so big, since we're having much bigger posts, that are being indexed without problems.

Testing it locally on windows, my apache crashed, but not on linux.

btw: I've used the following query to find out, how long our biggest posts are:

SELECT postid,LENGTH(pagetext) AS postlength
FROM post
WHERE LENGTH(pagetext) > 10000
ORDER BY postlength DESC

[Marc Smith]

Can you enter that postid in the edit screen and save it without making any changes, and have it successfully return you to the post? This would mean the post was successfully indexed (every post is reindexed on edit) - some posts that crashed my server could be indexed by this method if they weren't too big.

Try it

http://www.yoursite.com/forum/editpo...postid=[insert post id that it stopped at here]

I did try that. I can't edit or delete the post or the thread containing the post. It just hangs. I can go to my live forums and edit or delete the post there.

I'm doing what what was suggested above - start indexing at the post after the one which killed the indexing. So far 8 of 21667 posts are halting indexing. Only about 30000 posts to go.