Successfully impex'ed, but more counts than expected

[briansol]

MY board is up and running, and all is well, but I have ALMOST double posts and threads. I'm short about 20,000 or so to make it double, so it's not like I ran it twice.

I shut my ipb board down with 667.xxx posts, and now i have 1,0xx,xxx posts showing in the footer.

I ran the rebuild count tools on every option in the ACP.

Any ideas?

[Steve Machol]

Honestly it sounds like you double-imported. Impex does not arbitrarily double your post count. I suggest you clear out the data and redo the import.

[briansol]

I would agree with you, but I don't SEE them anywhere in the forum display.

If I imported twice, I SHOULD be seeing double threads and double posts, no?

Everything looks normal, save for the counts.

[Steve Machol]

Okay, but this still means the import was not done correctly, particularly if you followed all the instructions as you said.

[briansol]

edit, I just found some. It appears to show ALL in order, then the repeats (not by date as i would have expected).

Can i just redo the topics and posts step?

[briansol]

to reply to my own question, i guess it can't be done.

Starting over. 8 hours wasted :( damn big boards. lol

[briansol]

Well, i started from scratch, re-did the whole thing, manually checked the database, tables were truncated.

And I got the exact same results.

It's not me, something is wrong with the impex system.

[briansol]

iv'e narrowed it down.

Posts that were moved on IPB with the option of "leave moved link in this thread" show up as double posts. if they were moved twice, it shows up 3 times.

[briansol]

And why did it create a new forum:

Order the forums found here.
This is a default category for forums that need ordering

[briansol]

ok, tons of threads with no first post id.

SELECT threadid
FROM `thread`
WHERE firstpostid = ""

36,971 results.

I'm going to manually delete those, and it should kill my dupe topics.

Deleted rows: 36971 (Query took 10.4355 sec)

voila, dupe threads are gone.
the posts still exist, at least in counts of the posts table, but i don't care. its just space, and that's cheap. but no dupes are showing, and that's important.

so, please look into this. Moved threads that leave the moved-to link, come in as real topics and create dupes.

[briansol]

SQL query: SELECT count( * )
FROM `post`
WHERE parentid =0

Results: 66202


what does the parentid field do? i can't find any documentation on it...

[Steve Machol]

Given all this, please report this in the 3.5 Bug Tracker here:

http://www.vbulletin.com/forum/forumdisplay.php?f=99

[briansol]

bug submitted.

any info on my last question about the parentid?

[Steve Machol]

No sorry, I don't know what the parentid is for nor do I know where that query came from.

[briansol]

That query came from me, trying to figure out how to get rid of these dupe posts. I think i've come up with somethign better though...



SELECT postid
FROM `post`
where userid IN (select userid from user where username like 'imported_%')
(requires mysql 4.1+)

count: 85,751

still seems rather short... i was hoping for 450,000 ish...

I guess all the duplicate posts are NOT only from an imported_xxxxx username.

[briansol]

kill dupe posts:


SQL query: DELETE FROM `post` WHERE userid IN ( SELECT userid
FROM user
WHERE username LIKE 'imported_%'
)


Deleted rows: 85751 (Query took 97.4090 sec)


kill dupe users names:



delete
FROM user
where username like 'imported_%'


Deleted rows: 8696 (Query took 2.0425 sec)


This killed all my duplicate threads and most of my users. I still have a bunch of posts, i'm going to try to run the orphan tool and see what that picks up.

[briansol]

ok, that didin't work. i lost valid posts.

back to the drawing board...

[briansol]

ok, to find the bad threads with the bad poster names...



SELECT right( postusername, locate( '_', reverse( postusername ) ) -1 ) , postusername
FROM `thread`
WHERE postusername LIKE 'imported_%'



Then we can run a replace/update function which i'm wroking on now.

[briansol]

update thread
set postusername = right(postusername,locate('_',reverse(postusername ))-1)
where postusername like 'imported_%'

[briansol]

this also needs to be done on the lastposter field:



update thread
set lastposter = right(lastposter,locate('_',reverse(lastposter))-1)
where lastposter like 'imported_%'

[briansol]

finally, we need to kill the threads that are now blank with no posts...



DELETE
FROM `thread`
WHERE firstpostid NOT
IN (
SELECT postid
FROM post
)

[briansol]

lol- it never ends.

now the topics i just killed if it was the last post, it still shwos on the forum home as the imported_name, so we need to kill those.

but there's no good way to do it without killing an entire forum, or having a blank last poster field.
since this will get updated on the next post, i decided to just leave it be.


So, I lost about 8,000 threads, still have some dupes, but i'm not wasting another 17 hours of my life starting over on this again.

please look into fixing this tool.

[Steve Machol]

At this point I suggest you report this in the 3.5 Bug Tracker here:

http://www.vbulletin.com/forum/forumdisplay.php?f=99

Jerry is going to have to look at this problem.

[briansol]

Good, cuz i give up. lol

I ran into like 1212 more problems.

everyone's password was re-set. no one can login until they request a new password. Since i didn't require email validation, theres probably 50% of people who will never be able to log in again.

i'm shutting it down.

Jerry or steve, if you're available for consulting, let me know, i'll pay.

[Zachery]

What were you importing from?

[briansol]

ipb 2.1.6

[Zachery]

No matter how you import or who does it, passwords won't come over.

[Steve Machol]

Zach's right. Password are not imported. Please read this:

http://www.vbulletin.com/docs/html/impex_passwords?manualversion=30501500

[briansol]

Well, i need a functioning board before i can email people about their passwords.... lol

[briansol]

ok, i re-did this for the 4th time, and now i have no posts or threads coming up, yet my database says otherwise... still have duplicate members...

I'm no slouch when it comes to the web. I've been running websites for 5 years, code php and mysql for a living, and have a more than average apt towards things like this. I couldn't imagine a "normal" user doing this process.

i'm not please at all. i wasted my entire weekend on this, and have had my board closed for 3 days now, am out tons of ad revenue, and i can go on and on.

been 12 hours since i put in a support ticket with no reply other than saying it was assigned to somebody.... i understand it's the weekend, but there needs to be somebody out there to help, no?

i'm frustrated beyond belief with this process and software, and i'm 2 hours away from un-installing it and requesting a refund for my license.

[Steve Machol]

I thought you were going to wait until Jerry could look at your bug report. Honestly I've never seen an import with this many problems and it's going to take the Developer of the system to take a look at this.

[briansol]

I've waited too long already. This move was a huge mistake.

[Zachery]

Have you tried running the import from a clean vBulletin install?

[briansol]

4 times.

[briansol]

Does anybody work here? seriously.

[briansol]

I guess not....

[briansol]

... and another hour passes....

[Steve Machol]

With all due respect we have tried to help you in every way possible. Also we are not available 24 hours a day 7 days a week to instantly respond to every question.

As I stated previously I have never seen so many problems with an import and if you really have done everything suggested then this has to be due to a bug in the import system. Therefore the next step is to wait until Jerry (who is also not available 24/7) has time to read your bug report and act on it.

I have tried to help as much as possible but now it needs to be looked at by the Developer personally. Sorry for the inconvenience but that is the only thing that can be done with this now.

[briansol]

...and with all due respect back, all I got was "report a bug" and "wait for jerry". I realize you're not the person in charge of coding, and i understand that there's not much you can do. Please don't take anything i've said here personal.

I'm not demanding an instant fix. But something like "Hey, we're looking into this" within 24 hours would be great- at least let me know it's being looked at. For all I know (and which is probably true) no one has even looked at it yet.

I don't seem to be the only one having hell with the IPB convertor:
http://www.vbulletin.com/forum/showthread.php?p=1141023#post1141023

I would suggest taking it off of your "supported" list until it's fixed and actually supports the software as it claims to.

[Steve Machol]

I didn't say you were the only one having problems. Howwever your problms are far in excess of anything I've seen before. Sorry I didn't make that clear.

[Jerry]

I'm not demanding an instant fix. But something like "Hey, we're looking into this" within 24 hours would be great- at least let me know it's being looked at. For all I know (and which is probably true) no one has even looked at it yet..

It is being looked at.


Does anybody work here? seriously.

Of course.

[briansol]

Thanks Jerry, I know I was being a pain in the ass earlier, I was just mad.

[briansol]

Any progress, Jerry?

[Jerry]

I've replied to your support ticket.

Also I was unaware that you had closed your source board before testing the import, I would advise re-opening the source board until all the issues are resolved. As per the howto and manual, I always advise never closing the source board until several successful imports have been done.

[briansol]

after another sleepless night, this is resolved. I manually went through 2 million lines of INSERT's and cleaned it up.

[Jerry]

There was no need to do this, ImpEx doesn't dump SQL to a file either so you must of dumped the target database.

For other users reading this, test an import, before closing your source board and finding the issues to be resolved, if any.

[briansol]

no, I pulled my IPB database backup off the server, and manually removed all the moved post topic references by hand, cleaned up all the crap like admin logs and what not.

i spent 8 hours going through 2 million lines of code.

then re-imported that back into IPB
then ran the impex tool on that, and I also set everything to 1/10th the default setting to ensure no memory issues (100 instead of 1000 per cycle, for example).

PM's got screwed up (can't tell who it was from or sent too... just has owners username on it), and birthdays lost the month. but other than that, i can't complain.


I know the hole point is to test, but when you have a huge database, it takes 2 hours just to run a dump, so the board needs to be off.
and then you need to do it all over again, only to possibly run into new errors.

While I agree that testing should be done, for some like me, I just don't think it's feasable and would rather have the couple days down time.

[SimplyBen]

Wow man, sounds like you got the big problems. I did this on my board and have had nearly a 2.2x post count increase to all members. Some of them got around 1.8x extra posts.. very weird.. especially since i have not found any duplicate posts yet.