<i18n dev> RFR: 8306031: Update IANA Language Subtag Registry to Version 2023-04-13 [v2]

[Naoto Sato]

On Wed, 19 Apr 2023 20:07:32 GMT, Justin Lu <jlu at openjdk.org> wrote:

>> Update the registry and accompanying tests with the **IANA 4/13/2022** update.
>> 
>> This update introduces the case where an IANA entry can have a preferred value, but that preferred value has a preferred value as well.
>> 
>> This causes unexpected failures in JDK tests because of how locale equivalencies are created.
>> 
>> eg: `ar-ajp` has a preferred value of `ajp` but `ajp` has a preferred value of `apc`
>> 
>> Normally, when the JDK is built, _LocaleEquivlalentMaps.java_ generates the following
>> 
>> 
>> ...
>> singleEquivMap.put("ar-ajp", "ajp");
>> singleEquivMap.put("ajp", "ar-ajp");
>> ...
>> multiEquivsMap.put("ajp", new String[] {"apc", "ar-apc"});
>> multiEquivsMap.put("apc", new String[] {"ajp", "ar-apc"});
>> multiEquivsMap.put("ar-apc", new String[] {"apc", "ajp"});
>> ...
>> 
>> 
>> When `LocaleMatcher.parse(ACCEPT_LANGUAGE)` is called with `ACCEPT_LANGUAGE` containing `apc` and `ajp` in that order, the following occurs:
>> 
>> `apc` is found, `apc` is added, all of `apc's` equivalencies are added: `ajp` and `ar-apc`
>> 
>> When parse iterates to `ajp`, it finds that it is already added to the list, and does not add it's equivalency `ar-ajp`.
>> 
>> To address this, the build process must be adjusted so that the equivalencies are built as 
>> 
>> 
>> ...
>> multiEquivsMap.put("ajp", new String[] {"apc", "ar-ajp", "ar-apc"});
>> multiEquivsMap.put("apc", new String[] {"ajp", "ar-ajp", "ar-apc"});
>> multiEquivsMap.put("ar-ajp", new String[] {"apc", "ajp", "ar-apc"});
>> multiEquivsMap.put("ar-apc", new String[] {"apc", "ajp", "ar-ajp"});
>> ...
>> 
>> 
>> As, if `ar-ajp` has a preferred value of `ajp`, and `ajp` has a preferred value of `apc`, this technically means that `ar-ajp` is equivalent to `apc` and its equivalencies as well. This way, when `LocaleMatcher.parse(ACCEPT_LANGUAGE)` iterates to `apc`, it will add all of it's equivalencies including `ar-ajp`.
>
> Justin Lu has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Copyright

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java line 2:

> 1: /*
> 2:  * Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.

Cannot comment on unmodified lines, but instead of calculating the initial load itself, `HashMap.newHashMap()` can be used for initializing maps.

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java line 144:

> 142:                 boolean foundInOther = false;
> 143:                 final String finalPref = ","+preferred;
> 144:                 final String inbtwnPref = ","+preferred+",";

This could utilize regex?

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java line 146:

> 144:                 final String inbtwnPref = ","+preferred+",";
> 145:                 // Check if current pref exists inside a value for another pref
> 146:                 List<StringBuilder> doublePrefs = initialLanguageMap.entrySet()

`values()` fits here

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java line 150:

> 148:                                 e.getValue().toString().contains(inbtwnPref)))
> 149:                         .map(Map.Entry::getValue)
> 150:                         .collect(Collectors.toList());

Can replace `collect()` with `toList()`

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13501#discussion_r1171923706
PR Review Comment: https://git.openjdk.org/jdk/pull/13501#discussion_r1171920639
PR Review Comment: https://git.openjdk.org/jdk/pull/13501#discussion_r1171922562
PR Review Comment: https://git.openjdk.org/jdk/pull/13501#discussion_r1171923033