UBB Import question

[Gary Fischman]

When doing an import of a UBB, I understand that it will normally take the "Displayed" name as the new vB login.

However, what happens when there are conflicts? E.g., I have more than 1 "Bob" as a PDN in UBB. How does it handle that? Does it throw an exception and invalidate the entire import? Does it just ignore certain records and provide a fallout report?

[Jerry]

You can choose the display name or the login name. If the display name is chosen and doesn't exist the importer will default to the login name.

If a duplicate name exists, i.e. "Bob" and a second one is encountered the name will be pre-pended with imported_ so it becomes "imported_Bob".

As for a third, Bob it would pre-pend that with imported_ as well so there would be one Bob and two imported_Bob's as it stands.

Which wouldn't break the import, though would need cleaning up afterwards.

I will mod the ImpExDatabase::import_user() function to look for multiple import duplicates.

It will throw and exception within the import_user() function, which is where it handles it, so no, it won't invalidate or crash the import etc.

As for fall out report (logging) that is #2 on the to-do list atm.

[Gary Fischman]

How would I clean up afterwards? Can the login names be changed?
Any possibility of getting a script to find out beforehand where the duplicates are so I can give users a chance to change 'em?

[Jerry]

How would I clean up afterwards? Can the login names be changed?
Any possibility of getting a script to find out beforehand where the duplicates are so I can give users a chance to change 'em?

I'd suggest performing a test import then select the usernames from the vBulletin database that are "imported_*".

[Gary Fischman]

Just tried a test import with your newest Impex.
Threw an exception and stopped when it hit on a UBB PDN with a single quote in it. Mr. O'Connor broke the import. :)

[Jerry]

Just tried a test import with your newest Impex.
Threw an exception and stopped when it hit on a UBB PDN with a single quote in it. Mr. O'Connor broke the import. :)

If by PDN you mean displayname I have added the relevant addslashes().

[Gary Fischman]

Thanks. Yes, PDN meant displayname. (In UBB, they call it a "Public Displayed Name") As for why I'm smiling :) - its because I like vB and I'm making progress. Each time I find a bug, you have it nailed within a few hours. We're getting there and I appreciate it!

[Gary Fischman]

What happens if the source database for the import contains the same email address and/or login name of the Admin doing the import? Is there any risk of knocking myself out of the vB database?

If that were to happen, is there a way to rescue/reset the Admin password?

[Jerry]

What happens if the source database for the import contains the same email address and/or login name of the Admin doing the import? Is there any risk of knocking myself out of the vB database?

If that were to happen, is there a way to rescue/reset the Admin password?

If a users in the source data has the same email address as the admin of the target board, this will cause the source users data to be associated with the admin users, if you are mergin on email address. It won't over write the admin, the admin will just get an importuserid of the source user.

Though, if you re-run the import user module when the admin users has an importuserid, the origional admin user will be deleted as it is classed as an imported user. So either only run it once or manually set the admin users importuserid to 0 before rerunning it.

[Jerry]

I have thought of a few was to "protect" the admin user but none of them have been better than just making customers aware of the issue.

You can restore an admin user using the tools.php script.

[Gary Fischman]

Just completed a 25,000 user import. Seemed to mostly work, though there are some minor issues.

If the users were banned in UBB (they did not have the ability to post notes), they were not pulled in to the vB "Banned by Moderator" group. I tried running the "Import Ban List", but that didn't change anything.

When viewing the list of user groups, it showed that all the imported users were in the "Registered" group. However, when I clicked on some of the User IDs that I know were banned in UBB, their profile indicated that the primary group was "Coppa awaiting moderation" - even though the group list showed that group as having no members.

How do I deal with this?

[Jerry]

Just completed a 25,000 user import. Seemed to mostly work, though there are some minor issues.

If the users were banned in UBB (they did not have the ability to post notes), they were not pulled in to the vB "Banned by Moderator" group. I tried running the "Import Ban List", but that didn't change anything.

In ubb the importer tries to match :

/BanLists/EmailBan.cgi
/BanLists/IPBan.cgi
/BanLists/NameBansFull.cgi

Do they exsist ?


When viewing the list of user groups, it showed that all the imported users were in the "Registered" group. However, when I clicked on some of the User IDs that I know were banned in UBB, their profile indicated that the primary group was "Coppa awaiting moderation" - even though the group list showed that group as having no members.

How do I deal with this?

Well currently the ubb_classic importer dosn't import the custom groups then match the users to the groups, it tries to match the imported users to the default vBulletin groups.

I've had a look around some of the Members/user_groups/groups.cgi test data I've got and it dosn't have a full list of the memebers groups.

I'm thinking that some of it is hard coded.

I'll have a look at the usergroup parsing for the userimport, see if I can create the groups on the fly if they are no there so that you can manage the users better after an import.

[Gary Fischman]

Not sure we're on the same page.
It had no problem importing the email bans, IP bans, etc.

What didn't get pulled over is the fact that I have a few hundred users who permission to post to the forum is shut off. They're not "user groups" in UBB. They're just not allowed to post at all. But once they're pulled in, they're in the same "registered" group as everyone else, so they have permission to post again.

[Jerry]

I'm thinking the answer could be to create a group of "imported banned users" for them and put them in there then you can turn the posting off for that group after the import during the clean up.

[Gary Fischman]

That would work. Or allow the import to specify a group for imported banned users. Either way is fine.

[Jerry]

That would work. Or allow the import to specify a group for imported banned users. Either way is fine.

I'm going to go with creating a separate group I think, as that way you'll be able to tell who are the imported users who are in that group.

i.e. if you imported a 2nd board with the same settings you would know the original banned and the recently imported banned.

[Gary Fischman]

Kewl - let me know when I can try it. :)

[Jerry]

Kewl - let me know when I can try it. :)

Its late on a Friday here now and I don't work weekends usually so I'm guessing early next week :)

[Jerry]

Testing atm.

[Jerry]

Well there are all in the "Custom Usergroups -> Banned Users" group here.

Though I'll try splitting the groups out some more.

[Jerry]

Well I've updated the import so that it creates 5 import groups (Admins, Registered, COPPA, Banned and Other).

It sorts the users into the first 4, then if it can't extract the group or extrapolate the id/name, it places the user in the 'Other' group. From there you can update the groups permissions or move them into other groups.

I've updated the down load so this is now in version : 1.38

[Gary Fischman]

Does it import moderators? If so, what group are they? "Other"?

[Jerry]

Can you have a look at the 00000****.cgi for a moderator and tell me what is on line 5 ?

At the moment, it would be Other unless they are an Admin also.

[Jerry]

Found the moderator information now.

[Jerry]

Does it import moderators? If so, what group are they? "Other"?

Updated the import now. Removed import styles as that is a front end module and will never be implemented in the core.

Have added import moderators now so that it imports the moderators to the forums, though as with all imports you need to check the permissions.

[Gary Fischman]

Can you have a look at the 00000****.cgi for a moderator and tell me what is on line 5 ?

At the moment, it would be Other unless they are an Admin also.

Line 5? Assuming that the first line is line 1 (and not 0):

Write&20,22,35

I assume that's posting permission in forums 20, 22, and 35.

[Jerry]

Line 5? Assuming that the first line is line 1 (and not 0):

Write&20,22,35

I assume that's posting permission in forums 20, 22, and 35.

As far as i can make out thats means they are a moderator for forums, 20, 22 and 35.

That's what the import moderate module looks for at the moment.

[Gary Fischman]

As far as i can make out thats means they are a moderator for forums, 20, 22 and 35.

That's what the import moderate module looks for at the moment.

That's not what it means. I just checked for that particular user. My forum has several sections that are private or write protected, and these numbers are the forum IDs for those sections. This user has the ability to access write protected sections.

The Moderator status is not in his Member directory listing. It comes from vars_mods.cgi which is located in the UBB cgi-bin directory.

[Jerry]

The Moderator status is not in his Member directory listing. It comes from vars_mods.cgi which is located in the UBB cgi-bin directory.

Ok, I see what is going on now.

Will be simple to update that, I'll do it now.

[Jerry]

Ok done that, just have to over write 000.php and 012.php and then re-run the module and it will clear out the old mods and import them correctly.

[Gary Fischman]

Since you ditched the "Import Styles", the menu is now incorrect. There is still an "Import Styles". So, when I clicked on "Import Forums", it told me I had to click "Import Styles" first. So I clicked on "Import Styles", but it actually then imports the forums.

Next: how does it treat imported moderators? I noticed it does not put them into a separate group, so how does it know they're moderators? Individual user permissions?

Finally - after importing the forums (but not threads or posts), I noticed that they are not showing up on the main forum page. Does that happen later in the import or do I need to run a "Repair/Optimize" or some other thing?

[Jerry]

Since you ditched the "Import Styles", the menu is now incorrect. There is still an "Import Styles". So, when I clicked on "Import Forums", it told me I had to click "Import Styles" first. So I clicked on "Import Styles", but it actually then imports the forums.

This comes from the generated menu, I'm going to hazard a guess that you have uploaded the new files but not deleted the session and all the imported data. When you initially choose a system, it reads through all the files getting the dependencies and building the menu, then it doesn't do it again as it is stored in the session.

Deleting the session and all imported data, that will rebuild the menu.


Next: how does it treat imported moderators? I noticed it does not put them into a separate group, so how does it know they're moderators? Individual user permissions?

In the admincp browse the Forums & Moderators section, they should be linked to the forums there.


Finally - after importing the forums (but not threads or posts), I noticed that they are not showing up on the main forum page. Does that happen later in the import or do I need to run a "Repair/Optimize" or some other thing?

That's in the HOWTO, what I believe is going on there is display order. Check out the Forum Manager ensure they aren't nested under the Main Forum and that their imported display value isn't 0.

I'm going to bet that the categories have a display value of 0 and you've not updated them.

As for the moderators, it might be an idea to add them to a custom usergroup as well for manageability.

[Gary Fischman]

I'm going to bet that the categories have a display value of 0 and you've not updated them.



No, they're not set to 0. They start at 101 and increment from there. But I still don't see them. They show up in "Forum manager", but not in "Forum permissions".

Never mind - all I had to do was go to "Forum Manager" and click on "Save display order" without making changes.

[Gary Fischman]

BTW - all users show a "Last visit" of "December 31, 1969". Does this get updated only after they visit?

[Jerry]

No, they're not set to 0. They start at 101 and increment from there. But I still don't see them. They show up in "Forum manager", but not in "Forum permissions".

Never mind - all I had to do was go to "Forum Manager" and click on "Save display order" without making changes.

So it was the forum cache from another import..

[QUOTE=Gary Fischmanall users show a "Last visit" of "December 31, 1969". Does this get updated only after they visit?[/QUOTE]

Yes.

[Gary Fischman]

OK, this migration is starting to shape up.
After all these test imports, I'd like to do a final "cleanup" prior to my real migration. I'd like to do the following:

Purge all users
Purge all Forums, Threads, and Posts
Purge calendar
Leave all styles and functional settings in place.


I figure I'll have to take the above steps manually. Is there anything else I should do to "clean up" prior to a final import?

BTW - since one of my servers just crashed, I'm going to postpone this work until I finish burning in a new machine, so I'm looking at doing this migration first week of February.

[Jerry]

Purge all users
Purge all Forums, Threads, and Posts



The link at the top of each impex page, to clean up database will do that.



Purge calendar
Leave all styles and functional settings in place.


I figure I'll have to take the above steps manually. Is there anything else I should do to "clean up" prior to a final import?


Yes the calandar will have to be manual. You can download the styles, maybe do that, then reinstall vBulletin then upload the styles ?

Main thing I would do, that you are headding towards it a clean dB to start from.

Possiably, a fresh install then configureing it how you would like, make a back up of that and set that as a base to go from.

[Gary Fischman]

Not that simple. I've made dozens, if not hundreds of configuration tweaks and changes. It'll be difficult to track them all down if I re-install.

Any way to do some manual housecleaning prior to the import?

[Jerry]

Not that simple. I've made dozens, if not hundreds of configuration tweaks and changes. It'll be difficult to track them all down if I re-install.


Ah ok, well anything import will have a import_XXXX_id where XXXX is the table name, the links at the top of the impex page to "remove all imported data" will get everything that's been imported and leave the rest.


Any way to do some manual housecleaning prior to the import?


Well that's going to be everything that impex doesn't get, i.e. anything you have manually put in I'm guessing if I understand you correctly.

Out of importer interest how many posts is your ubb.classic board up to now ?

[Gary Fischman]

Around a million, I guess. Not sure. It was up to 1.5M, but I purged quite a bit of old posts. How long do you expect the import will take?

[Jerry]

On a P4 2.4 with 1 Gig of ram I would expect 40-60 mins depending on what else the server is doing, then quite a while to rebuild the search index, 8 hours at a guess ?

The board can be run while it is rebuilding though it just won't be 100% searching untill it is done.