Thanks Tom! The j.u.regex does not have its own direct access to PropList for now, have to use the properties from j..l.Character class. I will have to move those CharacterDateNN classes from the java.lang package (package private) to sun.lang or somewhere that both j.u.Character and j.u.regex can both have access, given we are so late in release, I'm trying to make as less change as possible. Will redo the properties access part in JDK8 to (1) provide more useful properties or all (? :-), maybe with better performance) (2) as you suggested, to implement \X, need to access the property that probably not available via j.l.Character class. The "Noncharacter_code_point" is not difficult to achieve by simply hard coding those code points. -- "You'll certainly want to somehow make that available for Strings, too;" Yes, I remember I "joked" with a language guy about the possibility of adding the literal syntax for Unicode character name for String and "char", for example '\u{LATIN_CAPITAL_LETTER_A}', I got "haha, interesting",:-) It's going to be a language spec change. -Sherman On 4/24/2011 8:47 PM, Tom Christiansen wrote:

Xueming, the docs look good.

On the name of the flag, I have no strong feelings one way or the other. Perhaps between UNICODE_PROPERTIES and UNICODE_CLASSES, I would prefer the second one. The first makes me think of the regular properties like \p{Script=Greek} from RL1.2, not the compat properties from RL1.2a like \w and \s. (I realize that the POSIX-compat stuff overlaps a bit.)

One thing you might want to take a look at longer term is whether all 11 properties listed in RL1.2 will be accessible. I don't know whether you have methods in Character for #7 and #8 below; I seem to recall seeing one but not the other.

RL1.2 Properties To meet this requirement, an implementation shall provide at least a minimal list of properties, consisting of the following:

yes 1 General_Category yes 2 Script yes 3 Alphabetic yes 4 Uppercase yes 5 Lowercase yes 6 White_Space ? 7 Noncharacter_Code_Point ? 8 Default_Ignorable_Code_Point ? 9 ANY yes 10 ASCII ? 11 ASSIGNED

RL1.2a Compatibility Properties To meet this requirement, an implementation shall provide the properties listed in Annex C. Compatibility Properties, with the property values as listed there. Such an implementation shall document whether it is using the Standard Recommendation or POSIX-compatible properties.

Other things...

I've thought a bit about whether it's worth pointing out that \x{h..h.} is the *only* way to (currently?) get non-BMP code points into a bracketed character class, like for example [\x{1D400}-\x{1D419}] to mean MATHEMATICAL BOLD CAPITAL A through MATHEMATICAL BOLD CAPITAL Z.

Reasons for not mentioning it include how rarely (I imagine) that users come across it, and also because this is one of those rare places in Java where you can't treat UTF-16 code units separately and get the same results. This matters for interpolation, because you can never build up a character class by using [ ] to surround the two different 16-bit char units that non-BMP codepoints turn into. You always have to use the indirect \x{h..h} instead. This is rather non-obvious.

However, after consideration I think it probably not worth risking confusing people by talking about it. I'm just very glad it can now be done. Having to figure out pieces of UTF-16 is no fun.

Once the current effort is done and you've had a well-deserved rest, I know you were thinking about \N{NAME} and \X for a future version of the JDK. Both are important, although for quite different reasons.

\N{NAME} is important for helping make regex code more maintainable by being self-documenting, since having to putting raw "magic" numbers instead of symbolic names in code is always bad. You'll certainly want to somehow make that available for Strings, too; not sure how to do that. The regex string escapes and the Java String escapes have already diverged, and I don't know how that happened. For example, the rules for octal escapes differ, and the regex engine supports things that Java proper does not; The "\cA" style comes to mind, but I think there are a few others, too. And now there is "\x{h..h}" too; pity Strings don't know that one.

\X is important because you really do need access to graphemes. Otherwise it is very difficult to write the equivalent of (?:(?=[aeiou])\X), which assuming NFD, will match a grapheme that start with one of those five letters. More importantly, you have to be able to chunk text by graphemes, and you need to do this even if you don't someday make a way to tie \b to the fancier sense like the ICU (?w) UREGEX_UWORD flag provides.

Getting grapheme clusters right is harder than it might appear. A careful reading of UAX#29 is important. There are two kinds of grapheme clusters, either legacy or extended. The extended version is tricky to get right, especially when you don't have access to all the syllable type properties. One problem with the legacy version is that it breaks up things that it shouldn't. We switched to the extended version for the 5.12 release of Perl, as this shows:

$ perl5.10.0 -le 'print "\r\n" =~ /\A\X\z/ ? 1 : 0' 0

$ perl5.12.3 -le 'print "\r\n" =~ /\A\X\z/ ? 1 : 0' 1

Which is the way it really needs to be. For the legacy sense, you can always still code that one up more explicitly:

$ perl5.12.3 -le 'print "\r\n" =~ /\A\p{Grapheme_Base}\p{Grapheme_Extend}*\z/ ? 1 : 0' 0

But I don't think people often want that version; extended is much better.

Giving Java access to the properties needed for either version of grapheme clusters may be a good time to reconsider whether you might wish to redesign how you check whether a code point has any particular Unicode property. There are of course performance issues, but also because the current mechanism does not seem easily extended to support the full complement of Unicode properties (which is a new Level-2 RL).

So if you are going to widen those again so you can have the properties at your fingertips that you need to support grapheme clusters, it might be worth thinking whether to refactor now or not. Performance and the size of tables will someday become an issue, if not now. I do know the Unicode docs mention this concern; I also know that we have people rethinking how Perl grants access to properties, because even though we do give you all of them, it could be done more tidily.

I haven't thought much about what the non-regex interface to graphemes and such should look like. Besides the ICU stuff, something you might want to take a look at, just to see the sorts of things others are doing in the non-regex grapheme arena, is these two classes:

http://search.cpan.org/perldoc?Unicode::GCString http://search.cpan.org/perldoc?Unicode::LineBreak

It's Spartan, but it will give you an idea. I couldn't (well, wouldn't *want* to) do East Asian text segmentation without those, which is something I've done a bit of lately. Like all the good 3rd-party Unicode modules in Perl, those two come from Asia. They're often the driving force behind progress in Unicode support. With all those complicated scripts, you can certainly see why, too.

Hope this helps!

--tom