Using norwegian characters in name of user (utf8 vs ascii-8bit) gives error (Bug #440)

What versions of ruby, rails and rake are you using?

Ruby version is 1.9.2 (x86_64)

1. gem list

• LOCAL GEMS ***

actionmailer (2.3.5)
actionpack (2.3.5)
activerecord (2.3.5)
activeresource (2.3.5)
activesupport (2.3.5)
daemon_controller (0.2.6)
fastthread (1.0.7)
i18n (0.4.2)
mysql (2.8.1)
passenger (3.0.7)
rack (1.0.1)
rails (2.3.5)
rake (0.9.0, 0.8.7)

Ok, I'd wager the problem comes from ruby 1.9 (the ChiliProject 1.x series is not tested against 1.9, ChiliProject 2.0 is tested against 1.9 and should mostly work, but could still have some bugs like that). Anyway, I don't think it's a problem with the name of your user but some problem with strings having different encodings in the lang_options_for_select method. Could you either try your current install on ruby 1.8 or upgrade to ChiliProject 2.0.0RC if you want/need to use ruby 1.9?

So I just did a new installation (on the side, on the same server, with the same setup),
of what's in HEAD in git, and got the same error:

ActionView::TemplateError (incompatible character encodings: ASCII-8BIT and UTF-8) on line #9 of app/views/users/_form.rhtml:
6: <p><%= f.text_field :firstname, :required => true %></p>
7: <p><%= f.text_field :lastname, :required => true %></p>
8: <p><%= f.text_field :mail, :required => true %></p>
9: <p><%= f.select :language, lang_options_for_select %></p>
10: <% if Setting.openid? %>
11: <p><%= f.text_field :identity_url  %></p>
12: <% end %>

    app/views/users/_form.rhtml:9:in `_run_rhtml_app47views47users47_form46rhtml_locals_f_form_object'
    app/views/users/_general.rhtml:2:in `block in _run_rhtml_app47views47users47_general46rhtml_locals_general_object_tab'
    app/helpers/application_helper.rb:779:in `labelled_tabular_form_for'
    app/views/users/_general.rhtml:1:in `_run_rhtml_app47views47users47_general46rhtml_locals_general_object_tab'
    app/views/common/_tabs.rhtml:24:in `block in _run_rhtml_app47views47common47_tabs46rhtml_locals_object_tabs'
    app/views/common/_tabs.rhtml:23:in `each'
    app/views/common/_tabs.rhtml:23:in `_run_rhtml_app47views47common47_tabs46rhtml_locals_object_tabs'
    app/helpers/application_helper.rb:223:in `render_tabs'
    app/views/users/edit.rhtml:8:in `_run_rhtml_app47views47users47edit46rhtml'
    <internal:prelude>:10:in `synchronize'
    passenger (3.0.7) lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
    passenger (3.0.7) lib/phusion_passenger/abstract_request_handler.rb:513:in `accept_and_process_next_request'
    passenger (3.0.7) lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
    passenger (3.0.7) lib/phusion_passenger/classic_rails/application_spawner.rb:321:in `start_request_handler'
    passenger (3.0.7) lib/phusion_passenger/classic_rails/application_spawner.rb:275:in `block in handle_spawn_application'
    passenger (3.0.7) lib/phusion_passenger/utils.rb:479:in `safe_fork'
    passenger (3.0.7) lib/phusion_passenger/classic_rails/application_spawner.rb:270:in `handle_spawn_application'
    passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:180:in `start'
    passenger (3.0.7) lib/phusion_passenger/classic_rails/application_spawner.rb:149:in `start'
    passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:219:in `block (2 levels) in spawn_rails_application'
    passenger (3.0.7) lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
    passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:214:in `block in spawn_rails_application'
    passenger (3.0.7) lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
    <internal:prelude>:10:in `synchronize'
    passenger (3.0.7) lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
    passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:213:in `spawn_rails_application'
    passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:132:in `spawn_application'
    passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
    passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    passenger (3.0.7) helper-scripts/passenger-spawn-server:99:in `<main>'

With this, I can easily do tests since I can pull tests from git-tree and apply to the
installation without much extra work :)

Try my github branch.
https://github.com/marutosi/redmine/tree/ruby-1.9

• http://www.redmine.org/attachments/5498/Ruby-1.9-Encoding.default_external.diff
• http://www.redmine.org/attachments/5886/application_controller.diff

Thomas Langaas:

I can't reproduce this on 1.4.0 using postgres. The user's name accepted and displayed the UTF character properly.

Toshi MARUYAMA:

Can you add tests for your patches?

Works fine on the demo site (mysql)

Eric Davis wrote:

Toshi MARUYAMA:

Can you add tests for your patches?

http://www.redmine.org/projects/redmine/repository/revisions/5307/entry/trunk/test/unit/lib/redmine/i18n_test.rb#L150

Eric Davis wrote:

I just tested this with ChiliProject 2.0.0RC1 and couldn't reproduce the error at all. I'll try 1.4.0 in a minute.

The other thing it could be is your database/data. I see you have the mysql gem installed so I'm assuming you are using MySQL (I'm testing in Postgres). In your config/database.yml do you have " encoding: utf8" set?

If nothing else, can you post a copy of your user's data here. Run the following command. Change 3 to the id of your user in the new install (and feel free to scramble the hashed_password and salt fields before pasting here).

[...]

encoding: utf8 is set, yes

Here's the output from the command (I scrambled the password), this is from the git head-version I have installed:

# ./script/runner -e production "puts User.find(3).inspect"  
#<User id: 3, login: "tlan", hashed_password: ".......", firstname: "Thomas", lastname: "Lang\xC3\xA5s", mail: "thomas.langas@fxitech.com", admin: true, status: 1, last_login_on: nil, language: "en", auth_source_id: nil, created_on: "2011-06-02 00:20:42", updated_on: "2011-06-02 00:20:42", type: "User", identity_url: nil, mail_notification: "only_my_events">

Do you understand the meaning of "incompatible character encodings: ASCII-8BIT and UTF-8"?

$ irb
ruby-1.9.2-p136 :001 > s1 = "Lang\xC3\xA5s".force_encoding("ASCII-8BIT")
 => "Lang\xC3\xA5s" 
ruby-1.9.2-p136 :002 > s2 = "Lang\xC3\xA5s".force_encoding("UTF-8")
 => "Langås" 
ruby-1.9.2-p136 :003 > s3 = s1 + s2
Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
    from (irb):3
    from /home/xxxxx/.rvm/rubies/ruby-1.9.2-p136/bin/irb:16:in `<main>'

Eric Davis wrote:

Toshi MARUYAMA:

Can you add tests for your patches?

http://ci.chiliproject.org/job/chiliproject_unstable/DATABASE=mysql,RUBY=ruby-1.9.2-p180/lastCompletedBuild/testReport/%28root%29/RepositoryGitTest/test_log_utf8/

RepositoryGitTest.test_log_utf8

<"Felix Schäfer <felix _ fachschaften.org>"> expected but was
<"Felix Sch\xC3\xA4fer <felix _ fachschaften.org>">. (Test::Unit::AssertionFailedError)
test/unit/repository_git_test.rb:264:in `test_log_utf8'

Toshi MARUYAMA wrote:

Do you understand the meaning of "incompatible character encodings: ASCII-8BIT and UTF-8"?

Are you wondering if I understand? Yes, I know what it means, but I'm not very fluent with
Ruby :|

$ irb
ruby-1.9.2-p136 :001 > s = "Felix Schäfer" 
 => "Felix Schäfer" 
ruby-1.9.2-p136 :002 > s1 = s.dup.force_encoding("ASCII-8BIT")
 => "Felix Sch\xC3\xA4fer" 
ruby-1.9.2-p136 :003 > s2 = s.dup.force_encoding("UTF-8")
 => "Felix Schäfer" 
ruby-1.9.2-p136 :004 > s1 == s2
 => false 

$ irb
ruby-1.9.2-p136 :001 > s = "Eric Davis" 
 => "Eric Davis" 
ruby-1.9.2-p136 :002 > s1 = s.dup.force_encoding("ASCII-8BIT")
 => "Eric Davis" 
ruby-1.9.2-p136 :003 > s2 = s.dup.force_encoding("UTF-8")
 => "Eric Davis" 
ruby-1.9.2-p136 :004 > s1 == s2
 => true 

Toshi MARUYAMA:

We appreicate your help here but I need to officially ask you to change your behavior.

1. Posting multiple updates to an issue with only examples or links to redmine.org is not helpful and is close to being considering spamming.

2. When we set a status to "Needs more information" there is a reason for it. Changing it back to "Open" is not helpful and is causing more confusion (#440 note 12). The same goes for reopening closed issues that are about other things (#437 note 4).

Thank you for understanding.

Eric Davis
ChiliProject project lead

Did anything happen to this, I suspect I have some additional errors in other parts of Chiliproject due to the same error. When trying to add my git repo to a project, I get this:

ArgumentError (invalid byte sequence in US-ASCII):
  lib/redmine/scm/adapters/git_adapter.rb:147:in `block (2 levels) in revisions'
  lib/redmine/scm/adapters/git_adapter.rb:146:in `each_line'
  lib/redmine/scm/adapters/git_adapter.rb:146:in `block in revisions'
  lib/redmine/scm/adapters/abstract_adapter.rb:205:in `call'
  lib/redmine/scm/adapters/abstract_adapter.rb:205:in `block in shellout'
  lib/redmine/scm/adapters/abstract_adapter.rb:203:in `popen'
  lib/redmine/scm/adapters/abstract_adapter.rb:203:in `shellout'
  lib/redmine/scm/adapters/abstract_adapter.rb:189:in `shellout'
  lib/redmine/scm/adapters/git_adapter.rb:140:in `revisions'
  app/models/repository/git.rb:62:in `fetch_changesets'
  app/controllers/repositories_controller.rb:77:in `show'
  <internal:prelude>:10:in `synchronize'
  passenger (3.0.7) lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
  passenger (3.0.7) lib/phusion_passenger/abstract_request_handler.rb:513:in `accept_and_process_next_request'
  passenger (3.0.7) lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
  passenger (3.0.7) lib/phusion_passenger/classic_rails/application_spawner.rb:321:in `start_request_handler'
  passenger (3.0.7) lib/phusion_passenger/classic_rails/application_spawner.rb:275:in `block in handle_spawn_application'
  passenger (3.0.7) lib/phusion_passenger/utils.rb:479:in `safe_fork'
  passenger (3.0.7) lib/phusion_passenger/classic_rails/application_spawner.rb:270:in `handle_spawn_application'
  passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:180:in `start'
  passenger (3.0.7) lib/phusion_passenger/classic_rails/application_spawner.rb:149:in `start'
  passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:219:in `block (2 levels) in spawn_rails_application'
  passenger (3.0.7) lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
  passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:214:in `block in spawn_rails_application'
  passenger (3.0.7) lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
  <internal:prelude>:10:in `synchronize'
  passenger (3.0.7) lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
  passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:213:in `spawn_rails_application'
  passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:132:in `spawn_application'
  passenger (3.0.7) lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
  passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  passenger (3.0.7) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  passenger (3.0.7) helper-scripts/passenger-spawn-server:99:in `<main>'

Which gives me a 500-error on the webpage for viewing the repo for that project.

If this is not the same error, give me some more hints on what to provide and I'll open a new bugreport with this.

Thomas Langaas wrote:

If this is not the same error, give me some more hints on what to provide and I'll open a new bugreport with this.

That's a different error, we pulled in some incomplete code from Redmine. If you can open a new bug report for it, that would be great.

I'll look into the bug report for this actual issue in a minute. Been focusing on getting ChiliProject 2.0.0 released, now that it's out I have a bit to debug this.

Toshi MARUYAMA wrote:

http://www.redmine.org/issues/4050

• http://www.redmine.org/attachments/5498/Ruby-1.9-Encoding.default_external.diff
• http://www.redmine.org/attachments/5886/application_controller.diff

For what it's worth, I actually tested those two on my 2.0 installation, but it didn't fix the problem.

Thomas Langaas wrote:

Toshi MARUYAMA wrote:

http://www.redmine.org/issues/4050

• http://www.redmine.org/attachments/5498/Ruby-1.9-Encoding.default_external.diff
• http://www.redmine.org/attachments/5886/application_controller.diff

For what it's worth, I actually tested those two on my 2.0 installation, but it didn't fix the problem.

• http://ci.chiliproject.org/job/chiliproject_master/60/
• http://ci.chiliproject.org/job/chiliproject_master/60/DATABASE=mysql,RUBY=ruby-1.9.2-p180/

mysql2 test passes.
Could you try mysql2 adapter?

Toshi MARUYAMA wrote:

http://www.redmine.org/issues/4050

• http://www.redmine.org/attachments/5498/Ruby-1.9-Encoding.default_external.diff
• http://www.redmine.org/attachments/5886/application_controller.diff

Hey Toshi,

i've been experiencing a similiar UTF-8/ASCII issue. Here's my log:
https://gist.github.com/1226139

It prevented me from inputting cyrillic text into issue fields.

Applying your patches solved the issue.

PS The application_controller.diff didn't want to apply for some reason, so i added the snippet manually.

PPS My versions:

• Ruby 1.9.2p290
• SQlite database
• ChiliProject installed via "git clone git://github.com/chiliproject/chiliproject.git"

FYI:
incompatible character encodings: UTF-8 and CP850
http://www.redmine.org/boards/4/topics/26440

Holger Just wrote:

Could you please post the full stack trace from the log?

Here you are.

production.log excerpt:

Processing ProjectsController#settings (for 10.0.111.3 at 2011-11-04 21:49:16) [GET]
  Parameters: {"controller"=>"projects", "action"=>"settings", "id"=>"vasiliy_pupkin"}
Rendering template within layouts/base
Rendering projects/settings

ActionView::TemplateError (incompatible character encodings: ASCII-8BIT and UTF-8) on line #66 of app/views/layouts/base.rhtm
l:
63:
64:     <div id="content">
65:                             <%= render_flash_messages %>
66:         <%= yield %>
67:         <%= call_hook :view_layouts_base_content %>
68:                             <div style="clear:both;"></div>
69:     </div>

    app/views/layouts/base.rhtml:66:in `concat'
    app/views/layouts/base.rhtml:66:in `_run_rhtml_app47views47layouts47base46rhtml'
    <internal:prelude>:10:in `synchronize'
    passenger (3.0.9) lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
    passenger (3.0.9) lib/phusion_passenger/abstract_request_handler.rb:513:in `accept_and_process_next_request'
    passenger (3.0.9) lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
    passenger (3.0.9) lib/phusion_passenger/classic_rails/application_spawner.rb:321:in `start_request_handler'
    passenger (3.0.9) lib/phusion_passenger/classic_rails/application_spawner.rb:275:in `block in handle_spawn_application'
    passenger (3.0.9) lib/phusion_passenger/utils.rb:479:in `safe_fork'
    passenger (3.0.9) lib/phusion_passenger/classic_rails/application_spawner.rb:270:in `handle_spawn_application'
    passenger (3.0.9) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    passenger (3.0.9) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    passenger (3.0.9) lib/phusion_passenger/abstract_server.rb:180:in `start'
    passenger (3.0.9) lib/phusion_passenger/classic_rails/application_spawner.rb:149:in `start'
    passenger (3.0.9) lib/phusion_passenger/spawn_manager.rb:219:in `block (2 levels) in spawn_rails_application'
    passenger (3.0.9) lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
    passenger (3.0.9) lib/phusion_passenger/spawn_manager.rb:214:in `block in spawn_rails_application'
    passenger (3.0.9) lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
    <internal:prelude>:10:in `synchronize'
    passenger (3.0.9) lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
    passenger (3.0.9) lib/phusion_passenger/spawn_manager.rb:213:in `spawn_rails_application'
    passenger (3.0.9) lib/phusion_passenger/spawn_manager.rb:132:in `spawn_application'
    passenger (3.0.9) lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
    passenger (3.0.9) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    passenger (3.0.9) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    passenger (3.0.9) helper-scripts/passenger-spawn-server:99:in `<main>'

Rendering /var/vhosts/test.chewy.lolmaus.ru/public/500.html (500 Internal Server Error)

Hey Felix, i'm currently snowed under work and studies, and i no longer control the VPS i had used to test 2.4.0.

If i'm lucky, i'll be able to test it two weeks later.

Andrey Mikhaylov wrote:

Hey Felix, i'm currently snowed under work and studies, and i no longer control the VPS i had used to test 2.4.0.

If i'm lucky, i'll be able to test it two weeks later.

Thanks for your answer, I'll wait a little longer then :-)