[OpenJDK 2D-Dev] [9] Review request for 8029339 Custom MultiResolution image support on HiDPI displays

Alexander Scherbatiy alexandr.scherbatiy at oracle.com
Tue Feb 18 15:33:17 UTC 2014

  Hi Jim,

  Let's divide the discussion into two part.

  1. Where it is better to hold resolution variants?
     Putting resolution variants in Image class brings some questions like:
   - Some type of images do not need to have resolution variants
   - Should resolution variants have the same type as the base image?
   - getResolutionVariants() method can return copy of the original list 
so add/removeRV methods should be also added.

   There are pros and cons for placing resolution variants to Image 
class or to a separate intreface.

  2. Using scale factor/image sizes/scaled image sizes to retreive a 
resolution variant.

   May be it is better to have a structure that provide all necessary 
information  to query the resolution variant: scale factor, draw area 
width/height, transformed area width/height?

   For example:
     public interface MultiResolutionImage {

         interface DrawAreaInfo {

             float getScaleFactor();
             float getAreaWidth();
             float getAreaHeight();
             float getTransformedAreaWidth();
             float getTransformedAreaHeight();

         public Image getResolutionVariant(DrawAreaInfo drawAreaInfo) ;
         public List<Image> getResolutionVariants();

   The MultiResolutionImage default implementation could allow to use 
different strategies like scale factor/transfom/OS based
   to query a resolution variant. The OS based strategy can be used by 


On 2/13/2014 4:42 AM, Jim Graham wrote:
> On 2/12/14 5:59 AM, Alexander Scherbatiy wrote:
>> On 2/8/2014 4:19 AM, Jim Graham wrote:
>>> The primary thing that I was concerned about was the presence of
>>> integers in the API when Windows uses non-integer multiples
>>       It would make sense to pass real numbers to the
>> getResolutionVariant() method  if the difference between resolution
>> variants sizes is 1.
>>       It seems that it is not a common case.
> I was thinking of other API that is related to this, such as the API 
> that queries the scaling factor from a SurfaceManager.  I seem to 
> remember some integer return values in that, but Windows might have 
> the answer 1.4 or 1.8, depending on the screen scaling factor that was 
> determined from the UI.
> In terms of the getResolutionVariant() method here, those non-integer 
> screen scaling factors don't directly impact this API.  But, we have 
> some issues with the use of integers there from other sources:
> - That API assumes that the caller will determine the pixel size 
> needed, but the actual media choice is determined with different 
> techniques on Mac and Windows so this means that the caller will have 
> to worry about platform conventions.  Is that the right tradeoff?
> - The technique recommended for Mac involves computing the precise 
> size desired using the current transform, which may be a floating 
> point value, so the integer values used in this API are already 
> approximations and there is no documentation on how to generate the 
> proper integer.  In particular, the current code in SG2D naively uses 
> a cast to integer to determine the values to supply which means a 
> transformed size of W+0.5 will be truncated to W and the lower 
> resolution image will be used. Does that conform to Mac guidelines?  
> Do they require the truncated size to reach W+1 before the next size 
> is used?  Passing in float or double values would sidestep all of that 
> since then the comparisons would be done with full precision, but as 
> long as we can determine a "best practices compatible with all 
> platforms" rule on how to round to integers, then integers are OK there.
> - The Windows document you cite below suggests that the determination 
> should be made by looking at the Screen DPI and choosing the next 
> higher media variant based on that screen DPI. They do not specify 
> choosing media based on the current transform as is done for Mac.  If 
> we stick with supplying values that are used to determine which media 
> to use, then on Windows we should not take the transform into account, 
> but instead query the SurfaceManager for the scale factor and only 
> transform by those values (even if the current transform was manually 
> overridden to identity).
> There are pros and cons to both approaches.
> Mac ensures that you are always using the best media for any given 
> render operation.
> But, Windows ensure more consistency in the face of other scaling.
> The thing to consider is that if you have a 500x500 image with a 
> 1000x1000 variant and you rendering it at 500x500 and then 501x501, 
> that first jump will be fairly jarring as the scaled version of the 
> 1000x1000 will not look precisely like the original 500x500 did.  With 
> @2x images only, this effect is minimized so the advantage of always 
> using "the best media for a given render operation" may outweigh the 
> inconsistency issue.  But, on Windows where the media are 1.4x or 1.8x 
> in size, a downscaled image will start to show more interpolation 
> noise and so the balance of the two choices may shift more towards not 
> wanting a jarring shift.
> We might want one or more of the following:
> - Developer chooses policy (TX_AWARE, DPI_AWARE, ALWAYS_LARGEST, NONE, 
> PLATFORM) where the last policy would use TX_AWARE on Mac and 
> DPI_AWARE on Windows
> - We create our own policy and always use it (TX_AWARE?  or DPI_AWARE?)
> - We create our own policy that dynamically chooses one of the above 
> strategies depending on platform or available media or ???
> - We could create an optional interface for them to install their own 
> algorithm as well.  I think it would work best as a delegate interface 
> that one installs into Image so that it can be used with any image 
> without having to subclass (it wouldn't really have much to do for 
> BufferedImages or VolatileImages, though):
> class Image {
>     void setResolutionHelper(ImageResolutionHelper foo);
>     List<Image> getResolutionVariants();
> }
> or:
> class Graphics {
>      void setResolutionHelper(ImageResolutionHelper foo);
> }
> or - anywhere else it could be installed more centrally (per App 
> context)?
> and the interface would be something like one of these variants:
> interface ImageResolutionHelper {
>     // This version would prevent substituting a random image:
>     // They have to return an index into the List<Image> for that 
> image...
>     public int chooseVariant(Image img, double dpi, number w, number h);
> or:
>     // This version would allow substituting an arbitrary image:
>     public Image getVariant(Image img, double dpi, number w, number h);
> }
> Since they would be in full control of the policy, though, we would 
> unfortunately always have to call this, there would be no more testing 
> in SG2D to see "if" we need to deal with DPI, though perhaps we could 
> document some internal conditions in which we do not call it for 
> common cases (but that would have to be well agreed not to get in the 
> way of reasonable uses of the API and well documented)?
> Note that we would have to do a security audit to make sure that 
> random image substitution couldn't allow any sort of "screen phishing" 
> exploit.
>             ...jim
>>> and also what policy they use for choosing scaled images.
>>> I don't see a mention of taking the current transform into account,
>>> just physical issues like screen DPI and form factor. They talk about
>>> resolution plateaus and in their recommendations section they tell the
>>> developer to use a particular property that tells them the screen
>>> resolution to figure out which image to load if they are loading
>>> manually.  There is no discussion about dynamically loading multiple
>>> versions of the image based on a dynamic program-applied transform
>>> factor as is done on MacOS.
>>> Also, they tell developers to draw images to a specific size rather
>>> than using auto-sizing.  That begs the question as to how they
>>> interpret a call to draw an image just using a location in the
>>> presence of various DPI factors.
>>       There is an interesting doc that describes how to write DPI-aware
>> Win32 applications:
>> http://msdn.microsoft.com/en-us/library/dd464646%28v=vs.85%29.aspx
>>       It is suggested to handle WM_DPICHANGED message,  load the graphic
>> that has slightly greater resolution to the current DPI and use 
>> StretchBlt
>>       to scale down the image.
>>      Thanks,
>>      Alexandr.
>>>             ...jim
>>> On 2/7/14 3:00 AM, Alexander Scherbatiy wrote:
>>>> On 1/22/2014 6:40 AM, Jim Graham wrote:
>>>>> Hi Alexander,
>>>>> Before we get too far down the road on this API, I think we 
>>>>> understand
>>>>> the way in which MacOS processes multi-resolution images for HiDPI
>>>>> screens, but have we investigated the processes that Windows uses
>>>>> under Windows 8?  My impression is that Windows 8 has included a
>>>>> number of new techniques for dealing with the high resolution 
>>>>> displays
>>>>> that it will run on in the Windows tablet and mobile industries and
>>>>> that these will also come into play as 4K displays (already 
>>>>> available)
>>>>> become more common on the desktop.  We should make sure that what we
>>>>> come up with here can provide native compatibility with either
>>>>> platform's policies and standard practices.
>>>>> If you've investigated the MS policies I'd like to see a summary so
>>>>> that we can consider them as we review this API...
>>>>     There is the Windows Guidelines for scaling to pixel density:
>>>> http://msdn.microsoft.com/en-us/library/windows/apps/hh465362.aspx
>>>>     which says that Windows has automatic resource loading that 
>>>> supports
>>>> three version of images scaling (100%, 140%, and 180%)
>>>>    --------------------------------
>>>> Without scaling, as the pixel density of a display device 
>>>> increases, the
>>>> physical sizes of objects on screen get smaller.
>>>> When UI would otherwise be too small to touch and when text gets too
>>>> small to read,
>>>> Windows scales the system and app UI to one of the following scaling
>>>> plateaus:
>>>>      1.0 (100%, no scaling is applied)
>>>>      1.4 (140% scaling)
>>>>      1.8 (180% scaling)
>>>> Windows determines which scaling plateau to use based on the physical
>>>> screen size, the screen resolution, the DPI of the screen, and form
>>>> factor.
>>>> Use resource loading for bitmap images in the app package For bitmap
>>>> images stored
>>>> in the app package, provide a separate image for each scaling
>>>> factor(100%, 140%, and 180%),
>>>> and name your image files using the "scale" naming convention 
>>>> described
>>>> below.
>>>> Windows loads the right image for the current scale automatically.
>>>>    --------------------------------
>>>>   The image name convention for the various scales is:
>>>>     images/logo.scale-100.png
>>>>     images/logo.scale-140.png
>>>>     images/logo.scale-180.png
>>>>    The 'ms-appx:///images/logo.png' uri is used to load the image 
>>>> in an
>>>> application.
>>>>    If we want to support this in the same way as it is done for Mac 
>>>> OS X
>>>>    the WToolkit should return MultiResolution image in case if the
>>>> loaded image has .scale-* qualifiers.
>>>>    The Graphics class can request an image with necessary resolution
>>>> from the MultiResolution image.
>>>>    It seems that nothing should be changed in the MultiResolution
>>>> interface in this case.
>>>>     Thanks,
>>>>     Alexandr.
>>>>>             ...jim
>>>>> On 1/14/14 2:54 AM, Alexander Scherbatiy wrote:
>>>>>> Hello,
>>>>>> Could you review the fix:
>>>>>>    bug: https://bugs.openjdk.java.net/browse/JDK-8029339
>>>>>>    webrev: http://cr.openjdk.java.net/~alexsch/8029339/webrev.00
>>>>>>    This is a proposal to introduce an API that allows to create a
>>>>>> custom
>>>>>> multi resolution image.
>>>>>> I. It seems reasonable that the API should provide two basic
>>>>>> operations:
>>>>>>   1. Get the resolution variant based on the requested image 
>>>>>> width and
>>>>>> height:
>>>>>>      - Image getResolutionVariant(int width, int height)
>>>>>>     Usually the system provides the scale factor which represents 
>>>>>> the
>>>>>> number of pixels corresponding to each linear unit on the display.
>>>>>>     However, it has sense to combine the scale factor and the 
>>>>>> current
>>>>>> transformations to get the actual image size to be displayed.
>>>>>>   2. Get all provided resolution variants:
>>>>>>     - List<Image> getResolutionVariants()
>>>>>>    There are several uses cases:
>>>>>>     - Create a new multi-resolution image based on the given
>>>>>> multi-resolution image.
>>>>>>     - Pass to the native system the multi-resolution image. For
>>>>>> example,
>>>>>> a use can set to the system the custom multi-resolution cursor.
>>>>>> II. There are some possible ways where the new API can be added
>>>>>>   1. java.awt.Image.
>>>>>>    The 2 new methods can be added to the Image class. A user can
>>>>>> override
>>>>>>    the getResolutionVariant() and getResolutionVariants() methods to
>>>>>> provide the resolution variants
>>>>>>    or there can be default implementations of these methods if a 
>>>>>> user
>>>>>> puts resolution variants
>>>>>>    to the list in the sorted order.
>>>>>>    To check that the image has resolution variants the following
>>>>>> statement can be used: image.getResolutionVariants().size() != 1
>>>>>>    The disadvantage is that there is an overhead that the Image 
>>>>>> class
>>>>>> should contain the List object and not all
>>>>>>    images can have resolution variants.
>>>>>>   2. Introduce new MultiResolutionImage interface.
>>>>>>    A user should extend Image class and implement the
>>>>>> MultiResolutionImage interface.
>>>>>>    For example:
>>>>>>    ---------------------
>>>>>>      public class CustomMultiResolutionImage extends BufferedImage
>>>>>>              implements MultiResolutionImage {
>>>>>>          Image highResolutionImage;
>>>>>>          public CustomMultiResolutionImage(BufferedImage baseImage,
>>>>>>                  BufferedImage highResolutionImage) {
>>>>>>              super(baseImage.getWidth(), baseImage.getHeight(),
>>>>>> baseImage.getType());
>>>>>>              this.highResolutionImage = highResolutionImage;
>>>>>>              Graphics g = getGraphics();
>>>>>>              g.drawImage(baseImage, 0, 0, null);
>>>>>>              g.dispose();
>>>>>>          }
>>>>>>          @Override
>>>>>>          public Image getResolutionVariant(int width, int height) {
>>>>>>              return ((width <= getWidth() && height <= getHeight()))
>>>>>>                      ? this : highResolutionImage;
>>>>>>          }
>>>>>>          @Override
>>>>>>          public List<Image> getResolutionVariants() {
>>>>>>              return Arrays.asList(this, highResolutionImage);
>>>>>>          }
>>>>>>      }
>>>>>>    ---------------------
>>>>>>   The current fix adds the MultiResolutionImage interface and public
>>>>>> resolution variant rendering hints.
>>>>>> Thanks,
>>>>>> Alexandr.

More information about the 2d-dev mailing list