<i18n dev> Reading Linux filenames in a way that will map back the same on open?
Xueming Shen
Xueming.Shen at Sun.COM
Sat Sep 13 14:57:08 PDT 2008
Martin, don't trap people into using -Dfile.encoding, always treat it as
a read only property:-)
I believe initializeEncoding(env) gets invoked before -Dxyz=abc
overwrites the default one,
beside the "jnu encoding" is introduced in 6.0, so we no longer look
file.encoding since, I believe
you "ARE" the reviewer:-)
Dan, I kind of feel, switch the locale to a sio8859-1 locale in your
wrapper, for example
LC_ALL=en_US.ISO8859-1; export LC_ALL; java -Xmx512M -jar equivs.jar
equivs.main
should work, if it does not, can you try to run
LC_ALL=en_US.ISO8859-1; export LC_ALL; java Foo
with Foo.java
System.out.println("sun.jnu.encoding=" +
System.getProperty("sun.jnu.encoding"));
System.out.println("file.encoding=" + System.getProperty("file.encoding"));
System.out.println("default locale=" + java.util.Locale.getDefault());
Let us know the result?
sherman
Martin Buchholz wrote:
> On Wed, Sep 10, 2008 at 17:50, Dan Stromberg <strombrg at gmail.com> wrote:
>
>> Would you believe that I'm getting file not found errors even with
>> ISO-8859-1?
>>
>
> The software world is full of suprises.
>
> Try
> export LANG=C LC_ALL=C LC_CTYPE=C
> java ... -Dfile.encoding=ISO-8859-1 ...
>
> You could also be explicit about the
> encoding used when doing any kind of char<->byte
> conversion, e.g. reading from stdin or writing to stdout.
>
> Oh, and this is only traditional Unix systems like
> Linux and Solaris. Windows and MacOSX
> (at least should) act very differently in this area.
>
> Martin
>
>
>> (Naoto: My program doesn't know what encoding to expect - I'm afraid I
>> probably have different applications writing filenames in different
>> encodings on my Ubuntu system. I'd been thinking I wanted to treat
>> filenames as just a sequence of bytes, and let the terminal emulator
>> interpret the encoding (hopefully) correctly on output).
>>
>>
>>
>> This gives two file not found tracebacks:
>>
>> export LC_ALL='ISO-8859-1'
>> export LC_CTYPE="$LC_ALL"
>> export LANG="$LC_ALL"
>>
>> find 'test-files' -type f -print | java -Xmx512M -Dfile.encoding=ISO-8859-1
>> -jar equivs.jar equivs.main
>>
>> find ~/Sound/Music -type f -print | java -Xmx512M -Dfile.encoding=ISO-8859-1
>> -jar equivs.jar equivs.main
>>
>>
>>
>> I'm reading the filenames like (please forgive the weird indentation) :
>>
>> try {
>>
>> while((line = stdin.readLine()) != null)
>> {
>> // System.out.println(line);
>> // System.out.flush();
>> lst.add(new Sortable_file(line));
>> }
>> }
>> catch(java.io.IOException e)
>> {
>> System.err.println("**** exception " + e);
>> e.printStackTrace(); }
>>
>>
>>
>> Where Sortable_file's constructor just looks like:
>>
>> public Sortable_file(String filename)
>> {
>> this.filename = filename;
>> /*
>> Java doesn't have a stat function without doing some fancy stuff, so we
>> skip this
>> optimization. It really only helps with hard links anyway.
>> this.device = -1
>> this.inode = -1
>> */
>> File file = new File(this.filename);
>> this.size = file.length();
>> // It bothers a little that we can't close this, but perhaps it's
>> unnecessary. That'll
>> // be determined in large tests.
>> // file.close();
>> this.have_prefix = false;
>> this.have_hash = false;
>> }
>>
>>
>>
>> ..and the part that actually blows up looks like:
>>
>> private void get_prefix()
>> {
>> byte[] buffer = new byte[128];
>> try
>> {
>> // The next line is the one that gives file not found
>> FileInputStream file = new FileInputStream(this.filename);
>> file.read(buffer);
>> // System.out.println("this.prefix.length " + this.prefix.length);
>> file.close();
>> }
>> catch (IOException ioe)
>> {
>> // System.out.println( "IO error: " + ioe );
>> ioe.printStackTrace();
>> System.exit(1);
>> }
>> this.prefix = new String(buffer);
>> this.have_prefix = true;
>> }
>>
>>
>>
>> Interestingly, it's already tried to get the file's length without an error
>> when it goes to read data from the file and has trouble.
>>
>> I don't -think- I'm doing anything screwy in there - could it be that
>> ISO-8859-1 isn't giving good round-trip conversions in practice? Would this
>> be an attribute of the java runtime in question, or could it be a matter of
>> the locale files on my Ubuntu system being a little off? It would seem the
>> locale files would be a better explanation (or a bug in my program I'm not
>> seeing!), since I get the same errors with both OpenJDK and gcj.
>>
>> Martin Buchholz wrote:
>>
>>> ISO-8859-1 guarantees round-trip conversion between bytes and chars,
>>> guarateeing no loss of data, or getting apparently impossible situations
>>> where the JDK gives you a list of files in a directory, but you get
>>> File not found when you try to open them.
>>>
>>> If you want to show the file names to users, you can always take
>>> your ISO-8859-1 decoded strings, turn them back into byte[],
>>> and decode using UTF-8 later, if you so desired.
>>> (The basic OS interfaces in the JDK are not so flexible.
>>> They are hard-coded to use the one charset specified by file.encoding)
>>>
>>> Martin
>>>
>>> On Wed, Sep 10, 2008 at 14:54, Naoto Sato <Naoto.Sato at sun.com> wrote:
>>>
>>>> Why ISO-8859-1? CJK filenames are guaranteed to fail in that case. I'd
>>>> rather choose UTF-8, as the default encoding on recent Unix/Linux are all
>>>> UTF-8 so the filenames are likely in UTF-8.
>>>>
>>>> Naoto
>>>>
>>>> Martin Buchholz wrote:
>>>>
>>>>> Java made the decision to use String as an abstraction
>>>>> for many OS-specific objects, like filenames (or environment variables).
>>>>> Most of the time this works fine, but occasionally you can notice
>>>>> that the underlying OS (in the case of Unix) actually uses
>>>>> arbitrary byte arrays as filenames.
>>>>>
>>>>> It would have been much more confusing to provide an interface
>>>>> to filenames that is sometimes a sequence of char, sometimes a
>>>>> sequence of byte.
>>>>>
>>>>> So this is unlikely to change.
>>>>>
>>>>> But if all you want is reliable reversible conversion,
>>>>> using java -Dfile.encoding=ISO-8859-1
>>>>> should do the trick.
>>>>>
>>>>> Martin
>>>>>
>>>>> On Tue, Sep 9, 2008 at 17:39, Dan Stromberg <dstromberglists at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Sorry if this is the wrong list for this question. I tried asking it
>>>>>> on comp.lang.java, but didn't get very far there.
>>>>>>
>>>>>> I've been wanting to expand my horizons a bit by taking one of my
>>>>>> programs and rewriting it into a number of other languages. It
>>>>>> started life in python, and I've recoded it into perl
>>>>>> (http://stromberg.dnsalias.org/~strombrg/equivalence-classes.html).
>>>>>> Next on my list is java. After that I'll probably do Haskell and
>>>>>> Eiffel/Sather.
>>>>>>
>>>>>> So the python and perl versions were pretty easy, but I'm finding that
>>>>>> the java version has a somewhat solution-resistant problem with
>>>>>> non-ASCII filenames.
>>>>>>
>>>>>> The program just reads filenames from stdin (usually generated with
>>>>>> the *ix find command), and then compares those files, dividing them up
>>>>>> into equal groups.
>>>>>>
>>>>>> The problem with the java version, which manifests both with OpenJDK
>>>>>> and gcj, is that the filenames being read from disk are 8 bit, and the
>>>>>> filenames opened by the OpenJDK JVM or gcj-compiled binary are 8 bit,
>>>>>> but as far as the java language is concerned, those filenames are made
>>>>>> up of 16 bit characters. That's fine, but going from 8 to 16 bit and
>>>>>> back to 8 bit seems to be non-information-preserving in this case,
>>>>>> which isn't so fine - I can clearly see the program, in an strace,
>>>>>> reading with one sequence of bytes, but then trying to open
>>>>>> another-though-related sequence of bytes. To be perfectly clear: It's
>>>>>> getting file not found errors.
>>>>>>
>>>>>> By playing with $LC_ALL, $LC_CTYPE and $LANG, it appears I can get the
>>>>>> program to handle files with one encoding, but not another. I've
>>>>>> tried a bunch of values in these variables, including ISO-8859-1, C,
>>>>>> POSIX, UTF-8, and so on.
>>>>>>
>>>>>> Is there such a thing as a filename encoding that will map 8 bit
>>>>>> filenames to 16 bit characters, but only using the low 8 bits of those
>>>>>> 16, and then map back to 8 bit filenames only using those low 8 bits
>>>>>> again?
>>>>>>
>>>>>> Is there some other way of making a Java program on Linux able to read
>>>>>> filenames from stdin and later open those filenames?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>> --
>>>> Naoto Sato
>>>>
>>>>
>>
More information about the i18n-dev
mailing list