Import from phpBB: figuring out what failed

[barefootcoder]

Okay, still working on my phpBB 2.x conversion. Things are going fairly well, but I have another issue I'm hoping someone here can help me with.

We have a test area where we've been playing around with vB, learning about all the new features and figuring out what's different. We've also tweaked all the settings and styles and all that happy stuff the way we want them to be on our production site.

Now I'm running conversion tests, into the actual DB that we'll be using when we switch over. What I'd like to do is copy over all the settings that we've tweaked, so we don't have to make all those changes by hand again. Now, copying over the rows from vb_setting works great. But that's only part of the story. I'm also copying over the changes we've made to vb_usergroup, and that also works well. But when I copy over the records we've changed in vb_style, it doesn't work. What happens is that whenever I try to load the main forums page, I just get a blank page. If I don't copy the vb_style records, everything works fine, but of course then we don't have our cool new style stuff that we put in. Does anyone know what I'm doing wrong? I suspect that there's another table I need to copy over that I'm missing, but I'm not sure what it is.

Also, if anyone knows of any other tables I need to copy like this to get all our settings over, that would be great to know. I obviously don't want to copy certain tables (posts, threads, users, etc), so I don't want to just blindly copy everything. But I don't know enough about vB yet to know which tables to copy and which not. I'm not a PHP guru, but I'm definitely very comfortable with SQL, so I've just been putting some queries into some scripts which I run at various points during the conversion.

TIA for any help.

[barefootcoder]

Okay, so during my conversion I have some things that failed. For most categories, it is a pretty small percentage of failures (e.g. users == 2,782 succeeded, 1 failed). Now, let's say that I'd like to take a look at that one failed user and see what went wrong. (Most likely, I don't care, but I figure it's worth at least taking a look.) How do I figure out which user it is that failed?

[barefootcoder]

I'm probably just doing something stupid here.

So I'm converting my users, and I told it "Yes" to convert the avatars. For the path, I put in the "customavatars" dir under my main vB install ... perhaps that's not correct. I sort of expected a bunch of avatars to land there, but they didn't (the dir is empty but for an index.html). So maybe I'm putting in the wrong path.

Anyway, the end result is that some of the avatars convert ... not many, but a few. The ones that converted are correct, they are definitely custom avatars, and they seem perfectly fine. But most of them just didn't come over at all. Any ideas what the problem might be?

[Jerry]

Make a copy of the set up board and then use that to import into.

[Jerry]

So I'm converting my users, and I told it "Yes" to convert the avatars. For the path, I put in the "customavatars" dir under my main vB install ... perhaps that's not correct. I sort of expected a bunch of avatars to land there, but they didn't (the dir is empty but for an index.html). So maybe I'm putting in the wrong path.

Anyway, the end result is that some of the avatars convert ... not many, but a few. The ones that converted are correct, they are definitely custom avatars, and they seem perfectly fine. But most of them just didn't come over at all. Any ideas what the problem might be?

ImpEx knows where to put them, it needs to know where to get them from all the details it asks for is for the source. The ones that would of worked would of been remote ones.

[barefootcoder]

Make a copy of the set up board and then use that to import into.

Well, that's one possibility. But then, of course, I have the opposite problem: I have to figure out what tables to empty out (as opposed to figuring out what tables to fill up). I was originally thinking that there would be fewer tables to copy over than to blank out, but perhaps I was wrong. (Also, I was worried about tables that might need default values instead of being blank, such as smilies.) But perhaps I can try this.

So I should blank out forums, threads, posts, and users ... would that be all? Smilies I might get by with leaving them as is, but I think I probably will need to blank out the DB with the install program and then save that off somewhere. Usergroups I can blow away all but the bottom 7 (or I might be able to leave them as is; I'll have to check and see if there's anything wacky in there). Anything else I need to worry about?


ImpEx knows where to put them, it needs to know where to get them from all the details it asks for is for the source. The ones that would of worked would of been remote ones.

Ah, okay, got it.

Also, you didn't answer my other question (about how to tell which users etc failed). One reason I thought it would be easier to have each issue in its own thread. :)

[barefootcoder]

Okay, I guess we'll try lumping all the issues here in one thread and see how that works.

So, my next issue is that nearly all (all but one, in fact) of the ranks failed to convert. A brief search through the forum yielded this thread (http://www.vbulletin.com/forum/showthread.php?t=141954), which (again) seems to address my problem almost exactly, but has no answer.

Like that thread's author, we use the ranks for determining who is labeled "site supporter", so I need to get that info over. Any thoughts?

[barefootcoder]

Another thing I just noticed:
Welcome to our newest member, a

I think I saw some other posts that mentioned this problem, but obviously I can't search for "a", so I can't find them now. Just for S&G I clicked on "a", but it isn't a valid user (as expected). Any ideas what's going on with this?

[barefootcoder]

Okay, here's one that's a little hard to explain, so I'll just steal the words of the beta tester who reported it:


... text sizes appear to be formatted at a different scale here in the new format as opposed to the old. For example, "size=20" at the old site looked about
THIS BIG
...the same text imported here now looks
THIS BIG.
... The old site had scale ranges from 0 to 28 (or so), while this new template scales only from 1 to 7. Any imported data with a size delimiter of more than 7 gets shown as "7", ...which is pretty honkin' huge. ...Don't know is anything can be done about this, ...but thought I should point it out. Can the 'size' delimiters be reduced during import, ...divided by a factor of 4 maybe? ...

Any thoughts? (BTW, I know I'm posting a lot of things that are going wrong, but really mostly things are going very right. vBulletin and ImpEx are great products.)

[Jerry]

Okay, I guess we'll try lumping all the issues here in one thread and see how that works.

That's how it's done opposed to create loads of threads so we can keep the context of what's going on.



So, my next issue is that nearly all (all but one, in fact) of the ranks failed to convert. A brief search through the forum yielded this thread (http://www.vbulletin.com/forum/showthread.php?t=141954), which (again) seems to address my problem almost exactly, but has no answer.

Like that thread's author, we use the ranks for determining who is labeled "site supporter", so I need to get that info over. Any thoughts?

What is the error given during the ranks module ?



Another thing I just noticed:
Welcome to our newest member, a

I think I saw some other posts that mentioned this problem, but obviously I can't search for "a", so I can't find them now. Just for S&G I clicked on "a", but it isn't a valid user (as expected). Any ideas what's going on with this?

Have you followed all the after import steps to clean up ?

http://www.vbulletin.com/docs/html/impex_cleanup


Okay, here's one that's a little hard to explain, so I'll just steal the words of the beta tester who reported it:

Any thoughts? (BTW, I know I'm posting a lot of things that are going wrong, but really mostly things are going very right. vBulletin and ImpEx are great products.)

You can use cleaner to replace the [size=X] tags afterwards if you want to shrink them.

[barefootcoder]

What is the error given during the ranks module ?

I dunno; all I saw was a bunch of "failed" messages, then it jumped back to the main import screen. How can I see what the error was?


Have you followed all the after import steps to clean up ?

Yep. Definitely did all those steps.


You can use cleaner to replace the [size=X] tags afterwards if you want to shrink them.

Cleaner? What's that?


Still outstanding: how to tell which users failed, and which tables I have to blank out if I just copy my test data DB and start with that. (AFA that last one goes, I think I'll post my table lists I've got so far. Maybe I can do that tomorrow.)

[Jerry]

I dunno; all I saw was a bunch of "failed" messages, then it jumped back to the main import screen. How can I see what the error was?

Set this in the ImpExConfig :



define('step_through', true);


Then you can see what the items are failing on.



Yep. Definitely did all those steps.




Cleaner? What's that?

tools/cleaner.php

It's a simple replacement script that changes one string for another.



Still outstanding: how to tell which users failed, and which tables I have to blank out if I just copy my test data DB and start with that. (AFA that last one goes, I think I'll post my table lists I've got so far. Maybe I can do that tomorrow.)

Blanking which tables for what reason ? There is a link at the top of every ImpEx page to Database Clean up that will remove any data that ImpEx imported.

[barefootcoder]

Set this in the ImpExConfig :



define('step_through', true);
Then you can see what the items are failing on.

Ah, good tip. Thanx.


tools/cleaner.php

It's a simple replacement script that changes one string for another.

Ah, excellent. I'll give that a try.


Blanking which tables for what reason ? There is a link at the top of every ImpEx page to Database Clean up that will remove any data that ImpEx imported.

No, I'm not talking about removing previous import data. I'm talking about the situation I outlined above. I said I wanted to import part of the data (e.g. permissions, styles, vB options) to my conversion test area from my "fooling around" test area. You suggested I instead copy the entire test area over. This, of course, brings along a lot of data that I don't want to copy (e.g. users, forums, threads, posts). None of that data that I don't want is converted data. It's just data from us playing around with a brand new piece of software and discovering what it can and can't do.

So currently what I'm doing is I've made a list of all the vb_* tables, and I've divided them up into 3 groups. One group I copy over verbatim from the test DB. One group I copy from a previously set up, completely blank vB DB. And one group I start with a blank version and then import from test selectively. It's not working too badly. I just wondered if you had any tips on which tables go in which groups.

Or if you have a different idea of how to do it entirely, I would happily entertain that as well. :)

[Jerry]

If you download ImpEx again and ensure that this is set in your config :

define('impexdebug', true);

Then you can browse the impexerror table to see which source data was skipped.

It's very very unadvisiable to copy some tables andnot others between databases.

There are two ways of doing it :

1) Set up the vBulletin board as you want, with styles and mods, then take a back up. Import into that and then set up the imported data and if anything goes wrong revert to the back up.

2) Do the import, then set up the whole board, with styles and mods etc.

[barefootcoder]

If you download ImpEx again and ensure that this is set in your config :

define('impexdebug', true);

Then you can browse the impexerror table to see which source data was skipped.

Ah, excellent tip. Thanx.


It's very very unadvisiable to copy some tables andnot others between databases.

By which I think you mean you won't help me work on that part. :)

But actually, it's working really well for me ATM. As I said, I have a lot of experience with SQL, so it's fairly easy for me to set up things like bringing over forum permissions while converting all forumid's/userid's/usergroupid's. If I knew what every table actually did, it'd be a snap. Unfortunately, while my SQL is solid, my PHP is not so much, so it's hard for me to run through the code and work out all that stuff.


There are two ways of doing it :

Yes, but both present problems for us. While I wasn't looking, our test vB area got used for two things simultaneously:


1) Set up the vBulletin board as you want, with styles and mods, then take a back up. Import into that and then set up the imported data and if anything goes wrong revert to the back up.

1) a play area to try out new things and just do whatever anyone wanted, which engenders a huge amount of crap that cannot be in the post-conversion data, thus we can't use your #1 suggestion, and


2) Do the import, then set up the whole board, with styles and mods etc.

2) a staging area to work on the set up of how the new site will supposedly look and operate, which engenders a huge amount of non-crap which must be in the post-conversion data, thus we can't do your #2 suggestion either. Well, technically we could, but a) it would be a huge amount of work to recreate all that, and b) half of the stuff they did they don't even remember any more, so there's just no way we'd ever get all of it ported over.

Additionally, I'm running conversion tests 3-4 times per week now, in an effort to make sure we shake out every last glitch before we do it for real. So even if we were to decide to do it the hard way, it wouldn't be practical to do that for every conversion test, which would mean that my tests wouldn't be reflecting what would really happen during the conversion, which would in turn mean that they weren't very good tests, right?

So I completely understand your position on this, because I'm deviating from the script and you don't want to be responsible for helping me screw something up (I'm a developer myself, so I can't say I wouldn't take the same stand as you were I in your shoes), but ANAICT I'm just stuck here with the approach I'm taking and I've got to make the best of it.

[Jerry]

By which I think you mean you won't help me work on that part. :)

Basically due to what can go wrong and I don't want to explain it and then have others attempt it and fubar their boards.


So I completely understand your position on this, because I'm deviating from the script and you don't want to be responsible for helping me screw something up (I'm a developer myself, so I can't say I wouldn't take the same stand as you were I in your shoes), but ANAICT I'm just stuck here with the approach I'm taking and I've got to make the best of it.

You can check out ImpExDatabaseCore for "function import_forum", "function import_user" etc and reading through the code might help you understand the table dependencies.

[barefootcoder]

Basically due to what can go wrong and I don't want to explain it and then have others attempt it and fubar their boards.

Okay, fair enough.


You can check out ImpExDatabaseCore for "function import_forum", "function import_user" etc and reading through the code might help you understand the table dependencies.

Thanx. I think that'll be helpful.

Okay, so let's tackle another topic. Conversion of text sizes doesn't work well, because the sizes in phpBB and vB don't even have the same ranges. phpBB is 0 to 30(ish), and vB is 1 to 7. Now, I've gotten one of my beta-testers to come up with a decent conversion chart. If you can point me in the ImpEx code to where this happens, I can probably submit you a patch to make the sizes look a lot closer than they are now.

[Jerry]

Line 106 of impex/systems/000.php

Is the text size handler along with the function below it pixel_size_mapping().

[ElGeorge]

Another thing I just noticed:
Welcome to our newest member, a


I'm currently in the exact same process of doing "Test rounds" for eventually migrating my phpBB 2.0.22 board to VB 3.6.8, and the "Newest member, a" was one of the things that was giving me a headache, specially since it's not easy looking for that specific case on this support forum (because on the restriction of short search terms)

Anyway... I was able to solve using the "Maintnance > Update Counters > Update User Names" option. Give it a try, it should work on your board also.

BTW... I haven't had much luck importing the user's avatars from my phpBB board. I've copied them locally on a directory that's accesible by the web server where VB is installed, but even with that after running the User Import the customavatars of VB does not contains the avatars.

The thing on one of my first tests with the previous VB version 3.6.7 I was able to import the avatars, but now I don't remember if I used a trick or something.

[barefootcoder]

Anyway... I was able to solve using the "Maintnance > Update Counters > Update User Names" option. Give it a try, it should work on your board also.

Excellent! Thanx for the tip!


BTW... I haven't had much luck importing the user's avatars from my phpBB board. I've copied them locally on a directory that's accesible by the web server where VB is installed, but even with that after running the User Import the customavatars of VB does not contains the avatars.

The thing that did it for me was when I realized (or, to be more precise, when Jerry explained to me) that the path it's asking for is not where the vB avatars are, but where the phpBB avatars are. I didn't think the dialog was very clear on that point. And then you have to copy the avatar images by hand into the vB avatar dir anyway, which didn't make much sense to me, but it was trivial to add to one of my conversion scripts.

HTH!

[Jerry]

The thing that did it for me was when I realized (or, to be more precise, when Jerry explained to me) that the path it's asking for is not where the vB avatars are, but where the phpBB avatars are. I didn't think the dialog was very clear on that point. And then you have to copy the avatar images by hand into the vB avatar dir anyway, which didn't make much sense to me, but it was trivial to add to one of my conversion scripts.

HTH!

Yup, when it comes to asking about data 99.99% of the time (I think 100% actually) it's asking about the source as ImpEx knows from vBulletin where the target data is going.