<AWT Dev> [10] JDK-8148344: Java robot keypress should be able to use extended key code characters as ? ? ?.

Shashidhara Veerabhadraiah shashidhara.veerabhadraiah at oracle.com
Mon Mar 26 05:53:38 UTC 2018


Hi, Please review the updated Webrev containing the requested changes:

http://cr.openjdk.java.net/~sveerabhadra/8148344/webrev.05/

I believe this change implements a new feature of Unicode key input and the corresponding key event generation without affecting the existing ascii key input and key event generations. Hence the regression should be less with this change.

Some notes with respect to implementation:
   1. On linux platform, native api's present only for the extended ascii support and hence no idea of Unicode input in this platform.
   2. On mac platform, I have used the below native api's to implement the Unicode input to a component and key event trigger from the component to the respective key listener.
	https://developer.apple.com/documentation/coregraphics/1456120-cgeventkeyboardgetunicodestring
	https://developer.apple.com/documentation/coregraphics/1456028-cgeventkeyboardsetunicodestring?language=objc
	(Set and get Unicode key is supported via the core graphics event(CGEvent) which is part of the NSEvent structure which is the base structure for event implementation on mac)
   3. On windows platform, I have used the below native api's to perform this job:
	https://msdn.microsoft.com/en-us/library/windows/desktop/ms646310(v=vs.85).aspx
	(SendInput api supports the Unicode input via the flag KEYEVENTF_UNICODE)
	https://msdn.microsoft.com/en-us/library/windows/desktop/ms646271(v=vs.85).aspx
	(Upon VK_PACKET, Get and decode the message via GetMessage() and TranslateMessage(). This produces a WM_CHAR message containing the respective Unicode key)

Thanks and regards,
Shashi

-----Original Message-----
From: Sergey Bylokhov 
Sent: Monday, November 27, 2017 11:43 PM
To: Shashidhara Veerabhadraiah <shashidhara.veerabhadraiah at oracle.com>; Semyon Sadetsky <semyon.sadetsky at oracle.com>; awt-dev at openjdk.java.net
Subject: Re: <AWT Dev> [10] JDK-8148344: Java robot keypress should be able to use extended key code characters as ? ? ?.

Hi, Shashi.
>> Also we should check what events are come to the application when 
>> this new API will be used. For example if 'm' will be pressed what 
>> notifications will be posted to the 
>> application?(keyTyped/keyPressed/keyReleased)

The test is still manual, I suggest to change it to automated and validate the behavior of "keyTyped/keyPressed/keyReleased". I suggest to implement it first and check that it works as expected on macOS and windows. After that we can take a look to the linux platform(For example I think KeyEvent.getExtendedKeyCodeForChar() should work on linux, and we can check can it be used in the proposed fix or not)

On 20/11/2017 21:03, Shashidhara Veerabhadraiah wrote:
> Hi Sergey, Please find the code updates that you asked for. As discussed I have raised an exception for the linux platform as there is no equivalent functions being implemented yet.
> 
> Webrev: http://cr.openjdk.java.net/~sveerabhadra/8148344/webrev.04/
> 
> shashi
> 
> -----Original Message-----
> From: Shashidhara Veerabhadraiah
> Sent: Thursday, November 16, 2017 12:36 PM
> To: Sergey Bylokhov <sergey.bylokhov at oracle.com>; Semyon Sadetsky 
> <semyon.sadetsky at oracle.com>; awt-dev at openjdk.java.net
> Subject: Re: <AWT Dev> [10] JDK-8148344: Java robot keypress should be able to use extended key code characters as ? ? ?.
> 
> Hi Sergey, Below are my replies:
> 
> shashi
> 
> -----Original Message-----
> From: Sergey Bylokhov
> Sent: Tuesday, November 14, 2017 3:32 AM
> To: Shashidhara Veerabhadraiah 
> <shashidhara.veerabhadraiah at oracle.com>; Semyon Sadetsky 
> <semyon.sadetsky at oracle.com>; awt-dev at openjdk.java.net
> Subject: Re: <AWT Dev> [10] JDK-8148344: Java robot keypress should be able to use extended key code characters as ? ? ?.
> 
> Hi, Shashi.
>    115     @Override
>    116     public void keyReleaseUnicode( int key ) {
>    117         // No special functions that implements a unicode key press
>    118         // and release in linux platform. Hence falls back to the
>    119         // default ASCII key press/release functions.
>    120         keyReleaseImpl(key);
>    121     }
> 
> We should do something in this part of the fix, because we cannot use unicode point as a keyCode. It will produce incorrect result.
> [Shashi] As discussed in the meeting, will add a unsupported exception in the following Webrev.
> 
> Also we should check what events are come to the application when this 
> new API will be used. For example if 'm' will be pressed what 
> notifications will be posted to the
> application?(keyTyped/keyPressed/keyReleased)
> [Shashi] It sends out the keyPressed/keyReleased events(WM_KEYUP/WM_KEYDOWN) events. The current code takes into account of the Unicode chars and uses the Unicode functions like ::ToUnicodeEx() to scan the characters.
> 
> On 05/11/2017 21:04, Shashidhara Veerabhadraiah wrote:
>> Hi Sergey, Please find the updated Webrev addressing your comments/requirements.
>>
>> http://cr.openjdk.java.net/~sveerabhadra/8148344/webrev.03/
>>
>> Thanks and regards,
>> Shashi
>>
>> -----Original Message-----
>> From: Shashidhara Veerabhadraiah
>> Sent: Friday, October 27, 2017 6:44 PM
>> To: Sergey Bylokhov <sergey.bylokhov at oracle.com>; Semyon Sadetsky 
>> <semyon.sadetsky at oracle.com>; awt-dev at openjdk.java.net
>> Subject: Re: <AWT Dev> [10] JDK-8148344: Java robot keypress should be able to use extended key code characters as ? ? ?.
>>
>> Hi Sergey, below are my replies:
>>
>> Thanks and regards,
>> shashi
>>
>> -----Original Message-----
>> From: Sergey Bylokhov
>> Sent: Friday, October 27, 2017 11:47 AM
>> To: Shashidhara Veerabhadraiah
>> <shashidhara.veerabhadraiah at oracle.com>; Semyon Sadetsky 
>> <semyon.sadetsky at oracle.com>; awt-dev at openjdk.java.net
>> Subject: Re: <AWT Dev> [10] JDK-8148344: Java robot keypress should be able to use extended key code characters as ? ? ?.
>>
>> Hi, Shashi.
>>     - Do we need to check passed unicode code-point in Robot.java using checkKeycodeArgument(key);?
>> [Shashi] I think yes. If the key value is zero I believe we don't need to process further.
>>     - Can you please clarify how it will work on linux where we will use code-point as a keycode?
>> [Shashi] There is no update being done to the linux source code as I could not find native call which can interpret the Unicode. Hence it will behave same as the original keypress() and keyrelease().
>>     - It would be good if the names of the parameters will be unified/corrected, for example:
>> private native void keyEventUnicode(int javaKeyCode, boolean keydown); "javaKeyCode" - Is it a java key code or a Unicode code-point?
>> [Shashi] Sure will update in the coming pass.
>>
>> Why the test is manual? Isn't an application should  recognize this new functionality automatically? Additionally it would be good to test all key related listeners(keyTyped/keyPressed/keyReleased).
>> [Shashi] We need to be sure that the Unicode key indeed displayed as the standard Unicode. There is no other way to confirm that it is indeed been the same standard Unicode that's been displayed.
>>
>> Debug code in awt_Robot.cpp
>> if(isUnicode) {printf("In unicode func:%d", jkey); [Shashi] Sure will update in the coming pass.
>>
>>
>> Also I would like to propose an idea for discussion: probably it would be better to create only one Robot#type(codePoint) method? What do you think?
>> [Shashi] It can be done but the only problem is that it is kinda different from the keypress() and keyRelease() functions. If that's fine then it can be done. Please confirm based on that I will produce a new Webrev based on that.
>>
>> On 26/10/2017 21:39, Shashidhara Veerabhadraiah wrote:
>>> Hi Sergey\Semyon, Please do the review for the below bug.
>>>
>>> Thanks and regards,
>>>
>>> Shashi
>>>
>>> *From:* Shashidhara Veerabhadraiah
>>> *Sent:* Thursday, September 21, 2017 2:14 PM
>>> *To:* Sergey Bylokhov <sergey.bylokhov at oracle.com>; Semyon Sadetsky 
>>> <semyon.sadetsky at oracle.com>; awt-dev at openjdk.java.net
>>> *Subject:* Re: <AWT Dev> [10] JDK-8148344: Java robot keypress 
>>> should be able to use extended key code characters as ? ? ?.
>>>
>>> Hi All, Please find the updated webrev containing a new test that is 
>>> added to test out the software changes that were made under this 
>>> enhancement.
>>>
>>> http://cr.openjdk.java.net/~sveerabhadra/8148344/webrev.02/
>>>
>>> Thanks and regards,
>>>
>>> Shashi
>>>
>>> *From:* Shashidhara Veerabhadraiah
>>> *Sent:* Thursday, September 21, 2017 11:37 AM
>>> *To:* Sergey Bylokhov <sergey.bylokhov at oracle.com 
>>> <mailto:sergey.bylokhov at oracle.com>>; Semyon Sadetsky 
>>> <semyon.sadetsky at oracle.com <mailto:semyon.sadetsky at oracle.com>>;
>>> awt-dev at openjdk.java.net <mailto:awt-dev at openjdk.java.net>
>>> *Subject:* Re: <AWT Dev> [10] JDK-8148344: Java robot keypress 
>>> should be able to use extended key code characters as ? ? ?.
>>>
>>> Hi Sergey, I was able to input the surrogate pairs and got the 
>>> required output as shown below:
>>>
>>> Below is the output after we input the surrogate pairs:
>>>
>>> Thanks and regards,
>>>
>>> Shashi
>>>
>>> -----Original Message-----
>>> From: Sergey Bylokhov
>>> Sent: Thursday, September 14, 2017 11:33 PM
>>> To: Shashidhara Veerabhadraiah 
>>> <shashidhara.veerabhadraiah at oracle.com
>>> <mailto:shashidhara.veerabhadraiah at oracle.com>>; Semyon Sadetsky 
>>> <semyon.sadetsky at oracle.com <mailto:semyon.sadetsky at oracle.com>>;
>>> awt-dev at openjdk.java.net <mailto:awt-dev at openjdk.java.net>
>>> Subject: Re: <AWT Dev> [10] JDK-8148344: Java robot keypress should 
>>> be able to use extended key code characters as ? ? ?.
>>>
>>> The java uses UTF16, I guess this new api should use it also, and we 
>>> should check that the surrogate pairs will be supported.
>>>
>>> On 9/14/17 03:56, Shashidhara Veerabhadraiah wrote:
>>>
>>>    > Hi Sergey, Yes it represents the Unicode code point. The 
>>> encoding is same as the window characteristic which is UTF 8 as implemented in Java.
>>>
>>>    >
>>>
>>>    > Thanks and regards,
>>>
>>>    > Shashi
>>>
>>>    >
>>>
>>>    > -----Original Message-----
>>>
>>>    > From: Sergey Bylokhov
>>>
>>>    > Sent: Wednesday, September 13, 2017 5:22 AM
>>>
>>>    > To: Shashidhara Veerabhadraiah
>>>
>>>    > <shashidhara.veerabhadraiah at oracle.com
>>> <mailto:shashidhara.veerabhadraiah at oracle.com>>; Semyon Sadetsky
>>>
>>>    > <semyon.sadetsky at oracle.com 
>>> <mailto:semyon.sadetsky at oracle.com>>;
>>> awt-dev at openjdk.java.net <mailto:awt-dev at openjdk.java.net>
>>>
>>>    > Subject: Re: <AWT Dev> [10] JDK-8148344: Java robot keypress 
>>> should be able to use extended key code characters as ? ? ?.
>>>
>>>    >
>>>
>>>    > Hi, Shashi.
>>>
>>>    > One initial question:
>>>
>>>    > What is an int parameter of these methods means, is it a 
>>> "Unicode code point"? What encoding utf8/utf16 should be used?
>>>
>>>    >
>>>
>>>    > On 9/8/17 02:50, Shashidhara Veerabhadraiah wrote:
>>>
>>>    >> Hi, I have updated the Webrev to accommodate the comments and 
>>> here is
>>>
>>>    >> the new Webrev:
>>>
>>>    >>
>>>
>>>    >> http://cr.openjdk.java.net/~sveerabhadra/8148344/webrev.01/
>>>
>>>    >>
>>>
>>>    >> I have separated the /_Unicode_/ keys input via java robot as 
>>> a new
>>>
>>>    >> set of /_public_/ api’s (this is in similar fashion as how the
>>>
>>>    >> platform offers the Unicode keys input into the system) and 
>>> this has
>>>
>>>    >> been tested on all the platforms using the test file similar 
>>> to the
>>>
>>>    >> attached file in the bug. A more proper test file would be put 
>>> for
>>>
>>>    >> review in the subsequent reviews.
>>>
>>>    >>
>>>
>>>    >> Thanks and regards,
>>>
>>>    >>
>>>
>>>    >> Shashi
>>>
>>>    >>
>>>
>>>    >> *From:* Sergey Bylokhov
>>>
>>>    >> *Sent:* Wednesday, August 30, 2017 2:33 AM
>>>
>>>    >> *To:* Shashidhara Veerabhadraiah
>>>
>>>    >> <shashidhara.veerabhadraiah at oracle.com
>>> <mailto:shashidhara.veerabhadraiah at oracle.com>>
>>>
>>>    >> *Cc:* awt-dev at openjdk.java.net 
>>> <mailto:awt-dev at openjdk.java.net>
>>>
>>>    >> *Subject:* Re: <AWT Dev> [10] JDK-8148344: Java robot keypress 
>>> should
>>>
>>>    >> be able to use extended key code characters as ? ? ?.
>>>
>>>    >>
>>>
>>>    >> Hi, Shashi.
>>>
>>>    >>
>>>
>>>    >> This is part of this fix, to figure out how it will work for 
>>> external
>>>
>>>    >> applications. As you said this functionally can be useful for 
>>> an
>>>
>>>    >> onscreen keyboards, which virtually can have any possible 
>>> keys, but
>>>
>>>    >> we should check how the applications will react on such keys:
>>>
>>>    >>    - Will the application get some kind of keyPress/Release?
>>>
>>>    >>    - Will the application get some keyCode for such event?
>>>
>>>    >>    - Is it possible to get autorepeat for such keys?(between
>>>
>>>    >> press/release)
>>>
>>>    >>
>>>
>>>    >> Depending from the answers above we can enhance existed robot 
>>> API or
>>>
>>>    >> provide a new one:
>>>
>>>    >> like Robot.keyType(char)/etc
>>>
>>>    >>
>>>
>>>    >> ----- shashidhara.veerabhadraiah at oracle.com
>>> <mailto:shashidhara.veerabhadraiah at oracle.com>
>>>
>>>    >> <mailto:shashidhara.veerabhadraiah at oracle.com> wrote:
>>>
>>>    >>>
>>>
>>>    >>
>>>
>>>    >>>
>>>
>>>    >>
>>>
>>>    >>>
>>>
>>>    >>
>>>
>>>    >> Hi Sergey, I was only able to add short cut keys in the 
>>> Microsoft
>>>
>>>    >> word but not as a system wide short cut key. There was no 
>>> mechanism
>>>
>>>    >> that I could find to add a short cut key for a Unicode char!!
>>> Can you
>>>
>>>    >> please tell me the steps to do the same if you are aware of?
>>>
>>>    >>
>>>
>>>    >> Thanks and regards,
>>>
>>>    >>
>>>
>>>    >> shashi
>>>
>>>    >>
>>>
>>>    >>>
>>>
>>>    >>
>>>
>>>    >> *From:*Sergey Bylokhov
>>>
>>>    >>> *Sent:* Tuesday, August 22, 2017 8:34 PM
>>>
>>>    >>> *To:* Shashidhara Veerabhadraiah
>>>
>>>    >>> <shashidhara.veerabhadraiah at oracle.com
>>>
>>>    >> <mailto:shashidhara.veerabhadraiah at oracle.com>>
>>>
>>>    >>> *Cc:* awt-dev at openjdk.java.net 
>>> <mailto:awt-dev at openjdk.java.net> <mailto:awt-dev at openjdk.java.net>
>>>
>>>    >>> *Subject:* Re: <AWT Dev> [10] JDK-8148344: Java robot 
>>> keypress
>>>
>>>    >>> should be
>>>
>>>    >> able to use extended key code characters as ? ? ?.
>>>
>>>    >>
>>>
>>>    >> Hi, Shashi.
>>>
>>>    >>> Can you check how this Robot API will work when the 
>>> application will have a shortcut for such key? Will such shortcuts 
>>> will work after this fix?
>>>
>>>    >>>
>>>
>>>    >>> ----- shashidhara.veerabhadraiah at oracle.com
>>> <mailto:shashidhara.veerabhadraiah at oracle.com>
>>>
>>>    >> <mailto:shashidhara.veerabhadraiah at oracle.com> wrote:
>>>
>>>    >>>>
>>>
>>>    >>
>>>
>>>    >>>
>>>
>>>    >>
>>>
>>>    >>>
>>>
>>>    >>
>>>
>>>    >> Hi All, Please review fix for the /_enhancement_/ wherein the 
>>> robot
>>>
>>>    >> key press of non-ascii were interpreted as question marks.
>>>
>>>    >>
>>>
>>>    >> Issue: The robot key press events was handling only the ascii 
>>> inputs
>>>
>>>    >> and ignored the other Unicode inputs. Either it was throwing 
>>> illegal
>>>
>>>    >> argument exception in windows or does nothing on the mac for 
>>> those
>>>
>>>    >> Unicode inputs.
>>>
>>>    >>
>>>
>>>    >> Solution and fix: The platform specific api’s was unable 
>>> handle the
>>>
>>>    >> non-ascii inputs. I have modified the api’s to accept the 
>>> non-ascii
>>>
>>>    >> inputs and correspondingly send the message to the window to 
>>> print
>>>
>>>    >> the non-ascii characters as well. Below is the picture of how 
>>> the
>>>
>>>    >> non-ascii inputs are considered and printed onto the window.
>>>
>>>    >>
>>>
>>>    >> The solution spans across windows and mac platform and still 
>>> in
>>>
>>>    >> search of a solution for the Linux platform. The solution 
>>> implements
>>>
>>>    >> key scanning only upon existing valid ascii key was /_not_/ 
>>> found and
>>>
>>>    >> assumes it as Unicode key and sends the event to event queue 
>>> to be
>>>
>>>    >> processed as Unicode keys. Different formats are being used by
>>>
>>>    >> different platform implementation of Unicode. For ex., per the 
>>> below
>>>
>>>    >> Unicode list, in the case of windows and mac, the key input 
>>> can take
>>>
>>>    >> decimal values whereas on Linux it can only take the Code values.
>>>
>>>    >>
>>>
>>>    >> On Linux, I was able to get the KeySym of Unicode keys but was 
>>> unable
>>>
>>>    >> to fake the key event as there was no mechanism available for 
>>> the
>>>
>>>    >> same(which sends the key event to window). Please let me know 
>>> if
>>>
>>>    >> there is any such mechanism available to simulate Unicode key 
>>> events
>>>
>>>    >> on Linux platform. Hence I think to raise a bug for the Linux
>>>
>>>    >> platform and close this JDK-8148344 bug.
>>>
>>>    >>
>>>
>>>    >> Enhancement id: 
>>> https://bugs.openjdk.java.net/browse/JDK-8148344
>>>
>>>    >>
>>>
>>>    >> Webrev:
>>> http://cr.openjdk.java.net/~sveerabhadra/8148344/webrev.00/
>>>
>>>    >>
>>>
>>>    >> Thanks and regards,
>>>
>>>    >>
>>>
>>>    >> Shashi
>>>
>>>    >>
>>>
>>>    >
>>>
>>>    >
>>>
>>>    > --
>>>
>>>    > Best regards, Sergey.
>>>
>>>    >
>>>
>>> --
>>>
>>> Best regards, Sergey.
>>>
>>
>>
>> --
>> Best regards, Sergey.
>>
> 
> 
> --
> Best regards, Sergey.
> 


--
Best regards, Sergey.


More information about the awt-dev mailing list