Error when saving a module with swedish characters in the title

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Mr B
    Member
    • Mar 2007
    • 97
    • 5.3.x

    [Bug / Issue] Error when saving a module with swedish characters in the title

    We have problems to save modules in Site Builder when giving them titles with Swedish characters like åäö.

    When trying to save a error message is displayed:
    An unexpected error was returned: 'Invalid widget ID specified: 0'

    How can we solve this? The module titles is shown on our custom homepage in Site Builder so its important to get it solved.

    vBulletin version 5.4.4
  • Wayne Luke
    vBulletin Technical Support Lead
    • Aug 2000
    • 73981

    #2
    I cannot recreate this issue using the latest version of vBulletin 5 and a UTF8MB4 database. Can you provide more information including all steps to recreate?
    Click image for larger version

Name:	2018-10-16_11-00-31.png
Views:	145
Size:	118.9 KB
ID:	4400564



    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud demonstration site.
    vBulletin 5 API

    Comment

    • Mr B
      Member
      • Mar 2007
      • 97
      • 5.3.x

      #3
      Originally posted by Wayne Luke
      I cannot recreate this issue using the latest version of vBulletin 5 and a UTF8MB4 database. Can you provide more information including all steps to recreate?
      Click image for larger version

Name:	2018-10-16_11-00-31.png
Views:	145
Size:	118.9 KB
ID:	4400564


      Our database is a "latin1 - default collation" InnoDB running on MySQL 5.7.20, PHP 7.2.7 and IIS 10 running.

      Im adding a search module and call it "Köp & sälj" and try to save it. Then the error message appears.

      Can it be the database collation that is faulty? I also having another åäö problem related to ordinary search. Ive posted in this thread about it:


      Maybe related?

      Comment

      • Wayne Luke
        vBulletin Technical Support Lead
        • Aug 2000
        • 73981

        #4
        Yes. You're using UTF-8 characters so you should be using a UTF-8 character set and collation in your database. We recommend using the UTF8MB4 character set and corresponding collation (utf8mb4_general_ci or utf8mb4_swedish_ci in your case). If the vBulletin installer creates the database for you, it will use UTF8MB4 with the utf8mb4_genera_ci collation. UTF8MB4 normalizes the number of bytes per character at 4 each and allows new UTF-8 functionality such as the Emoticons popular on mobile devices.

        The character set controls how characters appear and are sent from the database. The collation affects sort orders when running queries and therefore searches in the software. In addition to the database, you need to set your Language options in the AdminCP to they use UTF-8 (new installs should do this) as opposed to ISO-8859-1 (upgrades from older versions) and you may have to set the Locale to a UTF8 compatible Locale under Languages & Phrases -> Language Manager -> Your Language -> Edit.
        Translations provided by Google.

        Wayne Luke
        The Rabid Badger - a vBulletin Cloud demonstration site.
        vBulletin 5 API

        Comment

        • Mr B
          Member
          • Mar 2007
          • 97
          • 5.3.x

          #5
          Originally posted by Wayne Luke
          Yes. You're using UTF-8 characters so you should be using a UTF-8 character set and collation in your database. We recommend using the UTF8MB4 character set and corresponding collation (utf8mb4_general_ci or utf8mb4_swedish_ci in your case). If the vBulletin installer creates the database for you, it will use UTF8MB4 with the utf8mb4_genera_ci collation. UTF8MB4 normalizes the number of bytes per character at 4 each and allows new UTF-8 functionality such as the Emoticons popular on mobile devices.

          The character set controls how characters appear and are sent from the database. The collation affects sort orders when running queries and therefore searches in the software. In addition to the database, you need to set your Language options in the AdminCP to they use UTF-8 (new installs should do this) as opposed to ISO-8859-1 (upgrades from older versions) and you may have to set the Locale to a UTF8 compatible Locale under Languages & Phrases -> Language Manager -> Your Language -> Edit.
          Thanks! I will put up a testvb tomorrow and test it out. Do you know if i need to rebuild search index afterwards? The process takes around 2 days on my board.

          Comment

          • Wayne Luke
            vBulletin Technical Support Lead
            • Aug 2000
            • 73981

            #6
            I recommend rebuilding the search engine. You should use the searchindex.php file located in the do_not_upload folder. You run it from the command line and let it go. It doesn't need to be monitored and won't time out.

            If that still takes too long, you should look into using Sphinx search. The instructions are in the do_not_upload folder.
            Translations provided by Google.

            Wayne Luke
            The Rabid Badger - a vBulletin Cloud demonstration site.
            vBulletin 5 API

            Comment

            • Mr B
              Member
              • Mar 2007
              • 97
              • 5.3.x

              #7
              Originally posted by Wayne Luke
              I recommend rebuilding the search engine. You should use the searchindex.php file located in the do_not_upload folder. You run it from the command line and let it go. It doesn't need to be monitored and won't time out.

              If that still takes too long, you should look into using Sphinx search. The instructions are in the do_not_upload folder.
              I ran into major issues when trying to make a testvb installation to test the solution for the search problem. Please also take a look at https://www.vbulletin.com/forum/foru...in-site-broken

              I do not dare to make the database changes if i dont have tested it in an test environment before and the search issue is a major thing that must be solved.

              Comment

              • Mr B
                Member
                • Mar 2007
                • 97
                • 5.3.x

                #8
                Originally posted by Wayne Luke
                Yes. You're using UTF-8 characters so you should be using a UTF-8 character set and collation in your database. We recommend using the UTF8MB4 character set and corresponding collation (utf8mb4_general_ci or utf8mb4_swedish_ci in your case). If the vBulletin installer creates the database for you, it will use UTF8MB4 with the utf8mb4_genera_ci collation. UTF8MB4 normalizes the number of bytes per character at 4 each and allows new UTF-8 functionality such as the Emoticons popular on mobile devices.

                The character set controls how characters appear and are sent from the database. The collation affects sort orders when running queries and therefore searches in the software. In addition to the database, you need to set your Language options in the AdminCP to they use UTF-8 (new installs should do this) as opposed to ISO-8859-1 (upgrades from older versions) and you may have to set the Locale to a UTF8 compatible Locale under Languages & Phrases -> Language Manager -> Your Language -> Edit.
                Hmm, it seems to be a VERY complex task to convert the DB, tables and columns from latin1 to utf8mb4. Our forum have been upgraded over a lot of years and latin1 have always worked. Is this solvable in another way? We also have issues searching for words or phrases with åäö in them. All words/phrases with åäö is missing in "words" database table so they cant be searched…

                Is it some easy fix for this? Or can the conversation from latin1 to utf8mb4 be done in some easy way? Please help with this.

                Comment

                • Wayne Luke
                  vBulletin Technical Support Lead
                  • Aug 2000
                  • 73981

                  #9
                  Yes, keep your database as Latin1 and make sure that the new database uses the same character set and collation as the old database. I can't guarantee that you'll be able to use anything other than English though.

                  If you had a support contract, we could log into your server and do the conversion for you. It takes about an hour to complete. If you migrate to vBulletin Cloud, we will do the conversion as part of the Migration process.
                  Translations provided by Google.

                  Wayne Luke
                  The Rabid Badger - a vBulletin Cloud demonstration site.
                  vBulletin 5 API

                  Comment

                  • Mr B
                    Member
                    • Mar 2007
                    • 97
                    • 5.3.x

                    #10
                    Originally posted by Wayne Luke
                    Yes, keep your database as Latin1 and make sure that the new database uses the same character set and collation as the old database. I can't guarantee that you'll be able to use anything other than English though.

                    If you had a support contract, we could log into your server and do the conversion for you. It takes about an hour to complete. If you migrate to vBulletin Cloud, we will do the conversion as part of the Migration process.
                    I think that we have support contract but we have problems to give telnet access etc.

                    I did a clean test install of vBulletin 5.4.4 with utf8mb4 and utf8mb4_swedish_ci and the problem with åäö in module names is gone.

                    But, the search function still not working! Actually you have the same problem here on the vBulletin forum. If i post some Swedish fish names here like gädda gös färna etc they will not be searchable. If you exclude the åäö from the searchword like gdda gs frna they will be searchable.

                    In the DB table "words" they will appear like gdda gs frna (they missing the åäö). The posts shows the correct characters (the node data seems ok) but the indexing of the search words do not work. Why is the searchindex missing the åäö parts of Swedish words?

                    See attached images:

                    Comment

                    • Wayne Luke
                      vBulletin Technical Support Lead
                      • Aug 2000
                      • 73981

                      #11
                      Search Engine Issue




                      I need the software to fully support UTF-8 before I can push these issues more. That work is currently ongoing. The biggest issue is upgrades because vBulletin 3 and 4 tried to fake UTF-8 support by converting these characters to ASCII HTML Equivalents. So if you're upgrading, we'll need a tool to convert all these entities into actual characters. Not a small endeavor while maintaining database integrity. Every single post as has to be scanned, modified, and resaved.
                      Translations provided by Google.

                      Wayne Luke
                      The Rabid Badger - a vBulletin Cloud demonstration site.
                      vBulletin 5 API

                      Comment

                      • Mr B
                        Member
                        • Mar 2007
                        • 97
                        • 5.3.x

                        #12
                        Originally posted by Wayne Luke
                        Search Engine Issue

                        https://tracker.vbulletin.com/vbulle...ssues/VBV-1333
                        https://tracker.vbulletin.com/vbulle...sues/VBV-17331

                        I need the software to fully support UTF-8 before I can push these issues more. That work is currently ongoing. The biggest issue is upgrades because vBulletin 3 and 4 tried to fake UTF-8 support by converting these characters to ASCII HTML Equivalents. So if you're upgrading, we'll need a tool to convert all these entities into actual characters. Not a small endeavor while maintaining database integrity. Every single post as has to be scanned, modified, and resaved.
                        But i have done a fresh new install (not an upgrade) in /testvb/ folder to test if old data is the issue. No old data in that test. Only one test post containing åäö i the posts words. The same issue appears there. How can we proceed?

                        Comment

                        • Wayne Luke
                          vBulletin Technical Support Lead
                          • Aug 2000
                          • 73981

                          #13
                          You can't until either VBV-1333 or VBV-17331 is implemented. You can probably use a custom Google Search for your site instead of the built-in vBulletin search.
                          Translations provided by Google.

                          Wayne Luke
                          The Rabid Badger - a vBulletin Cloud demonstration site.
                          vBulletin 5 API

                          Comment

                          • Mr B
                            Member
                            • Mar 2007
                            • 97
                            • 5.3.x

                            #14
                            Originally posted by Wayne Luke
                            You can't until either VBV-1333 or VBV-17331 is implemented. You can probably use a custom Google Search for your site instead of the built-in vBulletin search.
                            What??? Can i help you to proceed with the vbvs? This is vital for my site. Google search is not an alternative.

                            Comment

                            • Mr B
                              Member
                              • Mar 2007
                              • 97
                              • 5.3.x

                              #15
                              Originally posted by Wayne Luke
                              You can't until either VBV-1333 or VBV-17331 is implemented. You can probably use a custom Google Search for your site instead of the built-in vBulletin search.
                              The first vbv is from 2012 and the second from 2018. When can we see a solution for this?

                              Comment

                              Related Topics

                              Collapse

                              Working...