ChiliProject is not maintained anymore. Please be advised that there will be no more updates.

We do not recommend that you setup new ChiliProject instances and we urge all existing users to migrate their data to a maintained system, e.g. Redmine. We will provide a migration script later. In the meantime, you can use the instructions by Christian Daehn.

Repositories encodings are broken (Bug #433)


Added by Toshi MARUYAMA at 2011-05-28 12:00 am. Updated at 2011-06-06 12:52 am.


Status:Open Start date:2011-05-27
Priority:Normal Due date:
Assignee:- % Done:

0%

Category:SCM
Target version:-
Remote issue URL: Affected version:unstable

Description


repo-setting.png (17.7 kB) Toshi MARUYAMA, 2011-05-28 12:00 am

redmine-fs.png (30.8 kB) Toshi MARUYAMA, 2011-05-28 12:00 am

chili-fs.png (20.5 kB) Toshi MARUYAMA, 2011-05-28 12:00 am

nul.png (24.7 kB) Toshi MARUYAMA, 2011-05-30 03:51 am


Associated revisions

Revision 200842ba
Added by Jean-Philippe Lang at 2008-02-27 09:50 pm

Propagates time tracking to the parent project (closes #433). Time report enhancements.

git-svn-id: http://redmine.rubyforge.org/svn/trunk@1176 e93f8b46-1217-0410-a6f0-8f06a7374b81

History

Updated by Eric Davis at 2011-05-29 06:01 pm

Please provide a description about the bug when posting it.

  • Priority changed from High to Normal
  • Category set to SCM
  • Status changed from Open to Needs more information

Updated by Toshi MARUYAMA at 2011-05-30 02:57 am

You should backout 9baa3104.

  • Status changed from Needs more information to Open

Updated by Toshi MARUYAMA at 2011-05-30 03:46 am

$ python
>>> s = "japanese" 
>>> u = s.decode("utf-8").encode("utf-16")
>>> u
'\xff\xfej\x00a\x00p\x00a\x00n\x00e\x00s\x00e\x00'

Updated by Toshi MARUYAMA at 2011-05-30 03:51 am

UTF-16 has '\0'.

  • File nul.png added

Updated by Toshi MARUYAMA at 2011-05-30 03:57 am

$ python
>>> s = "日本語" 
>>> u = s.decode("utf-8").encode("utf-16")
>>> u
'\xff\xfe\xe5e,g\x9e\x8a'

Updated by Toshi MARUYAMA at 2011-05-30 04:01 am

to_utf8_for_attachments() at source:app/helpers/attachments_helper.rb@13de4cc4#L31 returns "e,g" in "日本語" of UTF-16.

Updated by Toshi MARUYAMA at 2011-05-30 04:07 am

to_utf8() at source:app/helpers/repositories_helper.rb@2e1bcb2a#L119 converts UTF-16 to UTF-8.

Updated by Toshi MARUYAMA at 2011-06-03 12:11 am

$ irb
ruby-1.8.7-p330 :001 > require 'iconv'
 => true 
ruby-1.8.7-p330 :002 > Iconv.conv("UTF-16","UTF-8","日本語")
 => "\377\376\345e,g\236\212" 
ruby-1.8.7-p330 :003 > Iconv.conv("UTF-16","UTF-8","japanese")
 => "\377\376j\000a\000p\000a\000n\000e\000s\000e\000" 

Updated by Eric Davis at 2011-06-03 06:06 pm

Fixed in my ticket/release-v2.0.0/444-scm-encoding branch and merged into release-v2.0.0 in 3b64c60.

  • Target version set to 2.0.0
  • Assignee set to Eric Davis
  • Status changed from Open to Closed

Updated by Toshi MARUYAMA at 2011-06-03 10:29 pm

Not fixed.
Why you don't backout 9baa310486d782b ?
Do you understand 'UTF-16'?

  • Status changed from Closed to Open

Updated by Toshi MARUYAMA at 2011-06-03 10:33 pm

Did you run and confirm filesystem test repository?

Updated by Eric Davis at 2011-06-03 10:57 pm

Toshi MARUYAMA wrote:

Not fixed.
Did you run and confirm filesystem test repository?

1. It has been confirmed fixed according to my own tests and also confirmed on the CI server

Why you don't backout 9baa310486d782b ?

2. 9baa310486d782b has nothing to do with this. Both assert_tag and assert_select are testing the same thing.

Do you understand 'UTF-16'?

3. Yes

Status changed from Closed to Open

4. Please do not reopen issues without contributing any more information. This is the second time today I've had to say this today. https://www.chiliproject.org/issues/440#note-21

  • Status changed from Open to Closed

Updated by Toshi MARUYAMA at 2011-06-03 11:06 pm

Description images are on ChiliProject.
It is easy to reproduce.
UTF-16 has '\0', so it is easy to understand ChiliProject has 'MOJI-BAKE' (broken encoding).

Before you changed filesytem functional test, output has '\0'.
So, it is easy to understand why test fails.

Your modification filesystem test is wrong.

  • Status changed from Closed to Open

Updated by Toshi MARUYAMA at 2011-06-03 11:11 pm

Your recent revisions are about ChangeSets.
Filesystem repository does not have ChangeSets.

Filesystem repository reads file directly for equivalent "svn/hg/git cat".

Do you really understand character encoding and internal ChiliProject/Redmine?

Updated by Toshi MARUYAMA at 2011-06-03 11:36 pm

I changed this lines as following.
source:test/functional/repositories_filesystem_controller_test.rb@a428fa60#L81

japanese -> english

    def test_show_utf16
      with_settings :repositories_encodings => 'UTF-16' do
        get :entry, :id => PRJ_ID, :path => ['japanese', 'utf-16.txt']
        assert_response :success

        assert_select "tr" do
          assert_select "th.line-num" do
            assert_select "a", :text => /2/
          end
          assert_select "td", :content => /english/
        end

      end

But, test passes.

Your test is wrong.

You don't understand 'UTF-16'.

Updated by Toshi MARUYAMA at 2011-06-04 01:18 am

Eric Davis wrote:

4. Please do not reopen issues without contributing any more information. This is the second time today I've had to say this today. https://www.chiliproject.org/issues/440#note-21

I give you more information in English.
You don't understand my information.

Updated by Toshi MARUYAMA at 2011-06-04 04:57 am

Eric Davis wrote:

Do you understand 'UTF-16'?

3. Yes

You don't understand UTF-16.

$ rake test:scm:setup:filesystem
./filesystem_repository/
./filesystem_repository/test
./filesystem_repository/japanese/
./filesystem_repository/japanese/utf-16.txt
./filesystem_repository/japanese/euc-jp.txt
./filesystem_repository/japanese/big-file.txt
./filesystem_repository/dir/
./filesystem_repository/dir/dirfile
./filesystem_repository/dir/subdir/

$ export LANG=en_US.UTF-8
$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8" 
LC_NUMERIC="en_US.UTF-8" 
LC_TIME="en_US.UTF-8" 
LC_COLLATE="en_US.UTF-8" 
LC_MONETARY="en_US.UTF-8" 
LC_MESSAGES="en_US.UTF-8" 
LC_PAPER="en_US.UTF-8" 
LC_NAME="en_US.UTF-8" 
LC_ADDRESS="en_US.UTF-8" 
LC_TELEPHONE="en_US.UTF-8" 
LC_MEASUREMENT="en_US.UTF-8" 
LC_IDENTIFICATION="en_US.UTF-8" 
LC_ALL=

$ cat tmp/test/filesystem_repository/japanese/utf-16.txt  \
    | iconv -f utf-16 -t utf-8 | cat -n
     1    UTF-16
     2    japanese
     3    日本語
     4    にほんご
     5    ニホンゴ
     6    日本語
     7    にほんご
     8    ニホンゴ
     9    日本語
    10    にほんご
    11    ニホンゴ
    12    

$ cat tmp/test/filesystem_repository/japanese/utf-16.txt  | od -a
0000000 del   ~   U nul   T nul   F nul   - nul   1 nul   6 nul  cr nul
0000020  nl nul   j nul   a nul   p nul   a nul   n nul   e nul   s nul
0000040   e nul  cr nul  nl nul   e   e   ,   g  rs  nl  cr nul  nl nul
0000060   k   0   {   0 dc3   0   T   0  cr nul  nl nul   K   0   [   0
0000100   s   0   4   0  cr nul  nl nul   e   e   ,   g  rs  nl  cr nul
0000120  nl nul   k   0   {   0 dc3   0   T   0  cr nul  nl nul   K   0
0000140   [   0   s   0   4   0  cr nul  nl nul   e   e   ,   g  rs  nl
0000160  cr nul  nl nul   k   0   {   0 dc3   0   T   0  cr nul  nl nul
0000200   K   0   [   0   s   0   4   0  cr nul  nl nul  cr nul  nl nul
0000220

Updated by Toshi MARUYAMA at 2011-06-04 05:34 am

Eric Davis wrote:

2. 9baa310486d782b has nothing to do with this. Both assert_tag and assert_select are testing the same thing.

Your test is wrong.

This test passes.

    def test_show_eucjp
      with_settings :repositories_encodings => 'UTF-8' do
        get :entry, :id => PRJ_ID, :path => ['japanese', 'euc-jp.txt']
        assert_response :success
        assert_select "td", {:count=>1, :text=>"japanese"}
      end
    end

But, this test fails.

    def test_show_utf16
      with_settings :repositories_encodings => 'UTF-16' do
        get :entry, :id => PRJ_ID, :path => ['japanese', 'utf-16.txt']
        assert_response :success
        assert_select "td", {:count=>1, :text=>"japanese"}
      end
    end
  1) Failure:
test_show_utf16(RepositoriesFilesystemControllerTest)
    [test/functional/repositories_filesystem_controller_test.rb:93:in `test_show_utf16'
     test/test_helper.rb:89:in `with_settings'
     test/functional/repositories_filesystem_controller_test.rb:90:in `test_show_utf16']:
<"japanese"> expected but was
<"U\000T\000F\000-\0001\0006\000\r\000">.
<false> is not true.

UTF-16 is not compatible with ASCII.
EUC-JP and UTF-8 are compatible with ASCII.

You don't understand UTF-16.

Updated by Eric Davis at 2011-06-05 05:20 pm

  • Target version deleted (2.0.0)
  • Assignee deleted (Eric Davis)

Updated by deleted deleted at 2011-06-05 07:30 pm

@Toshi: It's quite over the top imho to (re-)open THREE times a ticket despite having been asked not to do so, resp. having been asked for more information (which 'You should backout 9baa3104.' certainly is not). Please try not to clutter issues with updates and finally, writing "Your test is wrong. You don't understand 'UTF-16'." is very unfriendly and really not helpful. - thank you

Updated by Toshi MARUYAMA at 2011-06-06 12:52 am

Hans-Peter Suter wrote:

Please try not to clutter issues with updates and finally, writing "Your test is wrong. You don't understand 'UTF-16'." is very unfriendly and really not helpful. - thank you

It is easy to know why RepositoriesFilesystemControllerTest.test_show_utf16 fails.

http://ci.chiliproject.org/job/chiliproject_unstable/DATABASE=mysql,RUBY=ruby-1.8.7-p334/12/testReport/junit/(root)/RepositoriesFilesystemControllerTest/test_show_utf16/

regression

Also available in: Atom PDF