Vbulletin Translation and encoding issue

[sajid09]

I am using two languages in the vbulletin i.e english and arabic... the problem is when i paste arabic in the arabic editor for new thread... the two tables that is vb_post and vb_postparsed the saved 'title' and 'pagetext' are unicode characters. wherease, when i paste the same arabic content in english editor, it is saved as htmlentities.

I want to have htmlentities in the tables no matter in which ever language the thread is being added, right now its not happening for me....


I am unable to find the solution please help.

[Steve Machol]

Honestly I have no idea how to change that. I'm not sure if this will help but see this thread for methods to troubleshoot character set problems:

http://www.vbulletin.com/forum/showthread.php?t=259250

[sajid09]

find the snapshot of DB

Steve thanks for the reply, but i think it didn't worked out wel for me, kindly check the screenshot im attaching for my db i mentioned the two section in the image, one which is desired and one which is not desired i.e

When arabic posted on an English forum ---> Shows first half of image
When arabic posted on an Arabic Forum --> Shows Second half of the image

My only concern is, where is the point from where i could figure out vbulletin is differentiating and inserting content in the threads, postparsed and post table.... There must a check or something which seprating the output...


What i have tested and found is that when text entered from charset=ISO-8859-1 format, which is basically english version of forum, no matter i paste arabic in the editor its wel encoded [DESIRED]

but when i change the language of the forum it set charset=windows-1256, Which ends up producing unicode and stuff when i paste an arabic text in the new thread or post [NOT DESIRED].

Any one who solve this problem is a GENIUS

Please help its real urgent

[sajid09]

Solved

use iconv("windows-1256", "UTF-8",$text);

by default iconv is available in php5.x

you can also download php_iconv.dll or iconv.dll and place it in php/extensions

your problem will be solved, but make sure your browser encoding in charset=utf-8 and if you are quering from database its not setting charset like mysql_set_charset('utf8');


Rest its working like charm

This is basically used where you want to show vbthread/vbpost on a non-vbpage like your site front-end.

[Luciano]

I think this may still make problems...
because what happens if you add a german, russian or chinese text in your "arabic" forum, Or if a english user.. answers an arabic post in arabic?
then it would also run through iconv("windows-1256", "UTF-8",$text);
giving unpredictable results...as the user sends the characters to database in ISO-8859-1 format

What I do.. ( but you have to do it from beginning) I set all languages to UTF-8 encoding..(even english), I set the database to UTF-8...
so all languages work... and everything, even english is stored in UTF-8.

the only thing to do extra is to save the master language file in UTF-8 in your text editor before installing vbulletin...
you then dont need html entities anymore, because browser switches to utf8 in every language..

thats my 2 cents..
Luc

[sajid09]

I think this may still make problems...
because what happens if you add a german, russian or chinese text in your "arabic" forum, Or if a english user.. answers an arabic post in arabic?
then it would also run through iconv("windows-1256", "UTF-8",$text);
giving unpredictable results...as the user sends the characters to database in ISO-8859-1 format

What I do.. ( but you have to do it from beginning) I set all languages to UTF-8 encoding..(even english), I set the database to UTF-8...
so all languages work... and everything, even english is stored in UTF-8.

the only thing to do extra is to save the master language file in UTF-8 in your text editor before installing vbulletin...
you then dont need html entities anymore, because browser switches to utf8 in every language..

thats my 2 cents..
Luc

Hey luc thnx for the reply, i have came across with same solution, but problem was if i make arabic language as utf-8 then it will show unreadable characters in the forum and everything seems to messed up, although my database collaition is set to utf.

To make things working as it is, i let the main language in its original format, and used the conversion for that specific non-vb page where result was required and it worked.

But i'll definitly look for the solution you provided, incase i faced the problem in future...Thanks for the post...

MY NEXT AIM IS TO MAKE A IT SO THAT IF SOMEBODY LOG IN FROM FRONT END OF THE SITE, HE AUTOMATICALLY LOG INTO FORUM, AND IF HE LOGOUT FROM FORUM HE IS LOGGED OUT FROM FRONT-END TOO ..

IT WILLL BE GREAT IF YOU HELP ME OUT IN THAT...

[Luciano]

Well its more complicated than just setting to utf-8
Because if you change to utf8.. ALL older posts will not be utf8, so they will not be readable... it will only work on newposts from the moment on where your database is utf8 AND your charset is UTF-8 AND your language files have been saved AND imported in UTF-8...
that is why it should only be done on a fresh install..
AND you must also convert the english AND the arabic language file to utf-8... BEFORE installing..
to do that,

• open the arabic language xml in your text editor (I use edit plus but there are free ones out there) you must be able to see the arabic characters in the xml file...
• then save as.. (for example change the file name from arabic.xml to arabic-utf8.xml) AND set the encoding to utf8 (this option should appear when you do save as in your editor)
• then open your forum/install directory.. Do the same with your vbulletin-language.xml BUT do not rename it.. just save as with encoding UTF-8 (overwrite it)
• go to phpmyadmin and check that your database... AND your tables have collation: utf8_general_ci (create a NEW table before you install)
  Some people have told me that for some languages it is better to use utf8_unicode_ci (but i never ran into problems with utf8_general_ci)
• Make sure you have this line activated in your config file: $config['Mysqli']['charset'] = 'utf8';
• I have never been able to make it work without mysqli enabled.. so enable mysqli (it will also help performance)
• Now only you install vbulletin to the table you have created!!!
• Then, that is important, once it is running.. go to your language manager -> edit and set the HTML Character Set encoding for ALL installed languages to UTF-8 (capital letters) (for english also)

This works for me, though I have never tried with arabic, it works with chinese/english

Note that per definition, you need to do this on a fresh install..
because if you have different charsets for different languages.. IF a user has english as default language, he will only NOT be able to see an arabic post display correctly. Why? because the page will have iso charset.. and the post he is viewing will have another encoding..


Luc

[sajid09]

Well its more complicated than just setting to utf-8
Because if you change to utf8.. ALL older posts will not be utf8, so they will not be readable... it will only work on newposts from the moment on where your database is utf8 AND your charset is UTF-8 AND your language files have been saved AND imported in UTF-8...
that is why it should only be done on a fresh install..
AND you must also convert the english AND the arabic language file to utf-8... BEFORE installing..
to do that,

• open the arabic language xml in your text editor (I use edit plus but there are free ones out there) you must be able to see the arabic characters in the xml file...
• then save as.. (for example change the file name from arabic.xml to arabic-utf8.xml) AND set the encoding to utf8 (this option should appear when you do save as in your editor)
• then open your forum/install directory.. Do the same with your vbulletin-language.xml BUT do not rename it.. just save as with encoding UTF-8 (overwrite it)
• go to phpmyadmin and check that your database... AND your tables have collation: utf8_general_ci (create a NEW table before you install)
  Some people have told me that for some languages it is better to use utf8_unicode_ci (but i never ran into problems with utf8_general_ci)
• Make sure you have this line activated in your config file: $config['Mysqli']['charset'] = 'utf8';
• I have never been able to make it work without mysqli enabled.. so enable mysqli (it will also help performance)
• Now only you install vbulletin to the table you have created!!!
• Then, that is important, once it is running.. go to your language manager -> edit and set the HTML Character Set encoding for ALL installed languages to UTF-8 (capital letters) (for english also)

This works for me, though I have never tried with arabic, it works with chinese/english

Note that per definition, you need to do this on a fresh install..
because if you have different charsets for different languages.. IF a user has english as default language, he will only NOT be able to see an arabic post display correctly. Why? because the page will have iso charset.. and the post he is viewing will have another encoding..


Luc

Thanks Luc, will do that as i'll move to new hosting... you replied on time

[tgralex]

I had similar problems with converting to UTF8 format. And for conversion and saving a single file you could use Notepad++. I found it quite good, useful and it is free. Here is the link: Notepad++

[Luciano]

I had similar problems with converting to UTF8 format. And for conversion and saving a single file you could use Notepad++. I found it quite good, useful and it is free. Here is the link: Notepad++

Yes i tested it and its nice.. but.. to convert to utf-8 you have no option when doing save as...
you need to go to menu-->format choose encoding there, and then only do a save as..
Luc

[kyrgyz]

Well its more complicated than just setting to utf-8
Because if you change to utf8.. ALL older posts will not be utf8, so they will not be readable... it will only work on newposts from the moment on where your database is utf8 AND your charset is UTF-8 AND your language files have been saved AND imported in UTF-8...
that is why it should only be done on a fresh install..
AND you must also convert the english AND the arabic language file to utf-8... BEFORE installing..
to do that,

• open the arabic language xml in your text editor (I use edit plus but there are free ones out there) you must be able to see the arabic characters in the xml file...
• then save as.. (for example change the file name from arabic.xml to arabic-utf8.xml) AND set the encoding to utf8 (this option should appear when you do save as in your editor)
• then open your forum/install directory.. Do the same with your vbulletin-language.xml BUT do not rename it.. just save as with encoding UTF-8 (overwrite it)
• go to phpmyadmin and check that your database... AND your tables have collation: utf8_general_ci (create a NEW table before you install)
  Some people have told me that for some languages it is better to use utf8_unicode_ci (but i never ran into problems with utf8_general_ci)
• Make sure you have this line activated in your config file: $config['Mysqli']['charset'] = 'utf8';
• I have never been able to make it work without mysqli enabled.. so enable mysqli (it will also help performance)
• Now only you install vbulletin to the table you have created!!!
• Then, that is important, once it is running.. go to your language manager -> edit and set the HTML Character Set encoding for ALL installed languages to UTF-8 (capital letters) (for english also)

This works for me, though I have never tried with arabic, it works with chinese/english

Note that per definition, you need to do this on a fresh install..
because if you have different charsets for different languages.. IF a user has english as default language, he will only NOT be able to see an arabic post display correctly. Why? because the page will have iso charset.. and the post he is viewing will have another encoding..


Luc

Do you think the same steps must be followed on a fresh vB4 Suite install?

[Dody]

The problem is using the wrong language pack. What you need to do is to ensure that the language pack you download is well formed and fit your installation defaults. For arabic I will post a well-working lanuage pack (once vb4.0 goes gold) that will be worikng perfectly without these issues.

[Luciano]

Do you think the same steps must be followed on a fresh vB4 Suite install?

I have no idea!!
but give me the install files.. and i can test it out for you

[kyrgyz]

So, you are not a beta tester either. That sucks.

I read a thread by AlexanderT about lack of UTF8 support in vB4 per his correspondence with vBulletin. UTF8 support vanished from the list of vB4 features. His thread was deleted as well. I can't find it here anymore. So, I expect that nothing really changed in regards to UTF8.

[vishnubhatia]

VB 3.8.4, uploading Arabic - UTF-8 langauge converts the characters to ????

Hi,

Has anyone been able to upload the Arbic UTF-8 language pack successfully?? I have installed my DB in UTF-8 General CI, changed the config.php file, converted the master language file to utf-8 and after all this when I am trying to upload the arabic language file which is created in UTF-8 by PHP Dev (http://www.vbulletin.com/forum/showt...guage-packages), all the phrases are converted to ?????... anyone who can help me?