[OpenJDK 2D-Dev] <AWT Dev> [9] Review request for 8029339 Custom MultiResolution image support on HiDPI displays

Alexander Scherbatiy alexandr.scherbatiy at oracle.com
Thu Mar 20 14:52:29 UTC 2014


   Could you review the updated version of the fix:

  - The "getResolutionVariant(int width, int height)" method from 
MultiResolutionImage class is changed to
    Image getResolutionVariant(float logicalDPIX, float logicalDPIY, 
float width, float height, AffineTransform transform);

  - sun.awt.image.ImageResolutionHelper class is added. The 
sun.awt.image.MultiResolutionToolkitImage and
     sun.awt.image.MultiResolutionBufferedImage classes are used 
PLATFORM  ImageResolutionHelper.

  The  MultiResolutionImage interface implementation could look like:
public class CustomMultiResolutionImage extends BufferedImage implements 
MultiResolutionImage {

     private final Image[] resolutionVariants;

     public CustomMultiResolutionImage(int baseIndex, Image... images) {
         this.resolutionVariants = images;
         Graphics g = getGraphics();
         g.drawImage(images[baseIndex], 0, 0, null);

     public Image getResolutionVariant(float logicalDPIX, float logicalDPIY,
             float width, float height, AffineTransform transform) {
         return getResolutionVariant(logicalDPIX * width, logicalDPIY * 

     public List<Image> getResolutionVariants() {
         return Arrays.asList(resolutionVariants);

     public Image getResolutionVariant(double width, double height) {
         for (Image image : resolutionVariants) {
             if (width <= image.getWidth(null) && height <= 
image.getHeight(null)) {
                 return image;
         return this;


On 2/27/2014 4:54 PM, Alexander Scherbatiy wrote:
> On 2/22/2014 3:54 AM, Jim Graham wrote:
>> Hi Alexandr,
>> On 2/18/14 7:33 AM, Alexander Scherbatiy wrote:
>>>   Hi Jim,
>>>   Let's divide the discussion into two part.
>>>   1. Where it is better to hold resolution variants?
>>>      Putting resolution variants in Image class brings some 
>>> questions like:
>>>    - Some type of images do not need to have resolution variants
>>>    - Should resolution variants have the same type as the base image?
>>>    - getResolutionVariants() method can return copy of the original 
>>> list
>>> so add/removeRV methods should be also added.
>>>    There are pros and cons for placing resolution variants to Image
>>> class or to a separate intreface.
>> I agree that this could be a separate interface.  In my examples 
>> below I was just sticking them inside an "Image{}" to show where they 
>> lived in the set of involved objects, not a specific recommendation 
>> that they actually be new methods on the base class itself.  I 
>> probably should have put a comment there about that.
>> With respect to add/remove - that is assuming a need for manual 
>> construction of an image set, right?  Forgive me if I'm forgetting 
>> something, but I seem to recall that manual Multi-Res images was 
>> proposed as a way for developers to introduce @2x support themselves, 
>> but if we are internally managing @2x and -DPI variants for them, 
>> then I'm not sure if there is actual developer need to manually 
>> construct their own.  Am I forgetting something?
>    The NSImage has addRepresentation/removeRepresentation methods to 
> work with image representations on Mac OS X.
>    The java.awt.Image class should provide similar functionality to 
> have the possibilities as Cocoa on HiDPI displays.
>>>   2. Using scale factor/image sizes/scaled image sizes to retreive a
>>> resolution variant.
>>>    May be it is better to have a structure that provide all necessary
>>> information  to query the resolution variant: scale factor, draw area
>>> width/height, transformed area width/height?
>>>    For example:
>>>    ---------------------
>>>      public interface MultiResolutionImage {
>>>          interface DrawAreaInfo {
>>>              float getScaleFactor();
>>>              float getAreaWidth();
>>>              float getAreaHeight();
>>>              float getTransformedAreaWidth();
>>>              float getTransformedAreaHeight();
>>>          }
>>>          public Image getResolutionVariant(DrawAreaInfo drawAreaInfo) ;
>>>          public List<Image> getResolutionVariants();
>>>      }
>>>    ---------------------
>> The problem with a constructor is that this is something that is 
>> (potentially) done on every drawImage() call, which means we are 
>> inviting GC into the equation.  If we can come up with a simple "just 
>> a couple/3/4 numbers" way to embed that data into a method call 
>> argument list then we can make this lighter weight.
>> What about simply having floating point (double) dimensions on the 
>> rendered size 
>       There should be a way to choose a resolution variant based on 
> requested drawing size or transformed drawing size.
>       At least a current transformation should be included too.
>> plus a single floating point "logical DPI" for the screen?
>      There is the ID2D1Factory::GetDesktopDpi method which returns 
> dpiX and dpiY.
> http://msdn.microsoft.com/en-us/library/windows/apps/dd371316
>     That means that logicalDPIX/Y can have different values.
>      At least it is described in the 
> http://msdn.microsoft.com/en-us/library/windows/apps/ff684173
>      "To get the DPI setting, call the ID2D1Factory::GetDesktopDpi 
> method. The DPI is returned as two floating-point values, one for the 
> x-axis and one for the y-axis. In theory, these values can differ. 
> Calculate a separate scaling factor for each axis."
>   The getResolutionVariant method could look like:
>    --------------------------------------
>     public Image getResolutionVariant(float logicalDPIX, float 
> logicalDPIY,
>             float widthX, float widthY, AffineTransform transform);
>    --------------------------------------
>>  If the image is known (either passed as an argument or the method is 
>> called on the image), then it can provide the original WH.
>>>    The MultiResolutionImage default implementation could allow to use
>>> different strategies like scale factor/transfom/OS based
>>>    to query a resolution variant. The OS based strategy can be used by
>>> default.
>> For Mac policy, all we need is the transformed dimensions, which can 
>> be passed in as FP for generality.  For Windows policy, all we need 
>> is logical DPI for the screen.   What other information would we 
>> need, or would an algorithm like to use, that can't be computed from 
>> those 2 pieces?
>      The aim is to provide a base class that can be used to create a 
> MultiResolutionImage like:
> http://hg.openjdk.java.net/jdk9/client/jdk/diff/ae53ebce5fa3/src/share/classes/sun/awt/image/MultiResolutionBufferedImage.java 
>      A developer should be able to implement a custom algorithm to 
> query a resolution variant.
>     It can be done by overriding the getResolutionVariant image:
>    -----------------------
>         Image mrImage = new MultiResolutionBufferedImage(){
>             @Override
>             public Image getResolutionVariant(...) {
>                 // Custom logic here
>             }
>         };
>    -----------------------
>    Or it can be done by using resolution variant choosers so a 
> developer can implement custom resolution variant query:
>    -----------------------
> public class MultiResolutionBufferedImage implements 
> MultiResolutionImage{
>     interface ResolutionVariantChooser{
>         Image getResolutionVariant(dpi, size,..., List<Image> 
> resolutionVariants);
>     }
>     ResolutionVariantChooser TRANSFORM_BASED = null;
>     ResolutionVariantChooser DPI_BASED = null;
>     ResolutionVariantChooser rvChooser;
>     @Override
>     public Image getResolutionVariant(dpi, size,...,) {
>         return rvChooser.getResolutionVariant(dpi, size,..., 
> getResolutionVariants());
>     }
> }
>    -----------------------
>   Thanks,
>   Alexandr.
>>             ...jim
>>> Thanks,
>>> Alexandr.
>>> On 2/13/2014 4:42 AM, Jim Graham wrote:
>>>> On 2/12/14 5:59 AM, Alexander Scherbatiy wrote:
>>>>> On 2/8/2014 4:19 AM, Jim Graham wrote:
>>>>>> The primary thing that I was concerned about was the presence of
>>>>>> integers in the API when Windows uses non-integer multiples
>>>>>       It would make sense to pass real numbers to the
>>>>> getResolutionVariant() method  if the difference between resolution
>>>>> variants sizes is 1.
>>>>>       It seems that it is not a common case.
>>>> I was thinking of other API that is related to this, such as the API
>>>> that queries the scaling factor from a SurfaceManager.  I seem to
>>>> remember some integer return values in that, but Windows might have
>>>> the answer 1.4 or 1.8, depending on the screen scaling factor that was
>>>> determined from the UI.
>>>> In terms of the getResolutionVariant() method here, those non-integer
>>>> screen scaling factors don't directly impact this API.  But, we have
>>>> some issues with the use of integers there from other sources:
>>>> - That API assumes that the caller will determine the pixel size
>>>> needed, but the actual media choice is determined with different
>>>> techniques on Mac and Windows so this means that the caller will have
>>>> to worry about platform conventions.  Is that the right tradeoff?
>>>> - The technique recommended for Mac involves computing the precise
>>>> size desired using the current transform, which may be a floating
>>>> point value, so the integer values used in this API are already
>>>> approximations and there is no documentation on how to generate the
>>>> proper integer.  In particular, the current code in SG2D naively uses
>>>> a cast to integer to determine the values to supply which means a
>>>> transformed size of W+0.5 will be truncated to W and the lower
>>>> resolution image will be used. Does that conform to Mac guidelines? Do
>>>> they require the truncated size to reach W+1 before the next size is
>>>> used?  Passing in float or double values would sidestep all of that
>>>> since then the comparisons would be done with full precision, but as
>>>> long as we can determine a "best practices compatible with all
>>>> platforms" rule on how to round to integers, then integers are OK 
>>>> there.
>>>> - The Windows document you cite below suggests that the determination
>>>> should be made by looking at the Screen DPI and choosing the next
>>>> higher media variant based on that screen DPI. They do not specify
>>>> choosing media based on the current transform as is done for Mac.  If
>>>> we stick with supplying values that are used to determine which media
>>>> to use, then on Windows we should not take the transform into account,
>>>> but instead query the SurfaceManager for the scale factor and only
>>>> transform by those values (even if the current transform was manually
>>>> overridden to identity).
>>>> There are pros and cons to both approaches.
>>>> Mac ensures that you are always using the best media for any given
>>>> render operation.
>>>> But, Windows ensure more consistency in the face of other scaling.
>>>> The thing to consider is that if you have a 500x500 image with a
>>>> 1000x1000 variant and you rendering it at 500x500 and then 501x501,
>>>> that first jump will be fairly jarring as the scaled version of the
>>>> 1000x1000 will not look precisely like the original 500x500 did.  With
>>>> @2x images only, this effect is minimized so the advantage of always
>>>> using "the best media for a given render operation" may outweigh the
>>>> inconsistency issue.  But, on Windows where the media are 1.4x or 1.8x
>>>> in size, a downscaled image will start to show more interpolation
>>>> noise and so the balance of the two choices may shift more towards not
>>>> wanting a jarring shift.
>>>> We might want one or more of the following:
>>>> - Developer chooses policy (TX_AWARE, DPI_AWARE, ALWAYS_LARGEST, NONE,
>>>> PLATFORM) where the last policy would use TX_AWARE on Mac and
>>>> DPI_AWARE on Windows
>>>> - We create our own policy and always use it (TX_AWARE?  or 
>>>> DPI_AWARE?)
>>>> - We create our own policy that dynamically chooses one of the above
>>>> strategies depending on platform or available media or ???
>>>> - We could create an optional interface for them to install their own
>>>> algorithm as well.  I think it would work best as a delegate interface
>>>> that one installs into Image so that it can be used with any image
>>>> without having to subclass (it wouldn't really have much to do for
>>>> BufferedImages or VolatileImages, though):
>>>> class Image {
>>>>     void setResolutionHelper(ImageResolutionHelper foo);
>>>>     List<Image> getResolutionVariants();
>>>> }
>>>> or:
>>>> class Graphics {
>>>>      void setResolutionHelper(ImageResolutionHelper foo);
>>>> }
>>>> or - anywhere else it could be installed more centrally (per App
>>>> context)?
>>>> and the interface would be something like one of these variants:
>>>> interface ImageResolutionHelper {
>>>>     // This version would prevent substituting a random image:
>>>>     // They have to return an index into the List<Image> for that
>>>> image...
>>>>     public int chooseVariant(Image img, double dpi, number w, 
>>>> number h);
>>>> or:
>>>>     // This version would allow substituting an arbitrary image:
>>>>     public Image getVariant(Image img, double dpi, number w, number 
>>>> h);
>>>> }
>>>> Since they would be in full control of the policy, though, we would
>>>> unfortunately always have to call this, there would be no more testing
>>>> in SG2D to see "if" we need to deal with DPI, though perhaps we could
>>>> document some internal conditions in which we do not call it for
>>>> common cases (but that would have to be well agreed not to get in the
>>>> way of reasonable uses of the API and well documented)?
>>>> Note that we would have to do a security audit to make sure that
>>>> random image substitution couldn't allow any sort of "screen phishing"
>>>> exploit.
>>>>             ...jim
>>>>>> and also what policy they use for choosing scaled images.
>>>>>> I don't see a mention of taking the current transform into account,
>>>>>> just physical issues like screen DPI and form factor. They talk 
>>>>>> about
>>>>>> resolution plateaus and in their recommendations section they 
>>>>>> tell the
>>>>>> developer to use a particular property that tells them the screen
>>>>>> resolution to figure out which image to load if they are loading
>>>>>> manually.  There is no discussion about dynamically loading multiple
>>>>>> versions of the image based on a dynamic program-applied transform
>>>>>> factor as is done on MacOS.
>>>>>> Also, they tell developers to draw images to a specific size rather
>>>>>> than using auto-sizing.  That begs the question as to how they
>>>>>> interpret a call to draw an image just using a location in the
>>>>>> presence of various DPI factors.
>>>>>       There is an interesting doc that describes how to write 
>>>>> DPI-aware
>>>>> Win32 applications:
>>>>> http://msdn.microsoft.com/en-us/library/dd464646%28v=vs.85%29.aspx
>>>>>       It is suggested to handle WM_DPICHANGED message, load the 
>>>>> graphic
>>>>> that has slightly greater resolution to the current DPI and use
>>>>> StretchBlt
>>>>>       to scale down the image.
>>>>>      Thanks,
>>>>>      Alexandr.
>>>>>>             ...jim
>>>>>> On 2/7/14 3:00 AM, Alexander Scherbatiy wrote:
>>>>>>> On 1/22/2014 6:40 AM, Jim Graham wrote:
>>>>>>>> Hi Alexander,
>>>>>>>> Before we get too far down the road on this API, I think we
>>>>>>>> understand
>>>>>>>> the way in which MacOS processes multi-resolution images for HiDPI
>>>>>>>> screens, but have we investigated the processes that Windows uses
>>>>>>>> under Windows 8?  My impression is that Windows 8 has included a
>>>>>>>> number of new techniques for dealing with the high resolution
>>>>>>>> displays
>>>>>>>> that it will run on in the Windows tablet and mobile industries 
>>>>>>>> and
>>>>>>>> that these will also come into play as 4K displays (already
>>>>>>>> available)
>>>>>>>> become more common on the desktop.  We should make sure that 
>>>>>>>> what we
>>>>>>>> come up with here can provide native compatibility with either
>>>>>>>> platform's policies and standard practices.
>>>>>>>> If you've investigated the MS policies I'd like to see a 
>>>>>>>> summary so
>>>>>>>> that we can consider them as we review this API...
>>>>>>>     There is the Windows Guidelines for scaling to pixel density:
>>>>>>> http://msdn.microsoft.com/en-us/library/windows/apps/hh465362.aspx
>>>>>>>     which says that Windows has automatic resource loading that
>>>>>>> supports
>>>>>>> three version of images scaling (100%, 140%, and 180%)
>>>>>>>    --------------------------------
>>>>>>> Without scaling, as the pixel density of a display device
>>>>>>> increases, the
>>>>>>> physical sizes of objects on screen get smaller.
>>>>>>> When UI would otherwise be too small to touch and when text gets 
>>>>>>> too
>>>>>>> small to read,
>>>>>>> Windows scales the system and app UI to one of the following 
>>>>>>> scaling
>>>>>>> plateaus:
>>>>>>>      1.0 (100%, no scaling is applied)
>>>>>>>      1.4 (140% scaling)
>>>>>>>      1.8 (180% scaling)
>>>>>>> Windows determines which scaling plateau to use based on the 
>>>>>>> physical
>>>>>>> screen size, the screen resolution, the DPI of the screen, and form
>>>>>>> factor.
>>>>>>> Use resource loading for bitmap images in the app package For 
>>>>>>> bitmap
>>>>>>> images stored
>>>>>>> in the app package, provide a separate image for each scaling
>>>>>>> factor(100%, 140%, and 180%),
>>>>>>> and name your image files using the "scale" naming convention
>>>>>>> described
>>>>>>> below.
>>>>>>> Windows loads the right image for the current scale automatically.
>>>>>>>    --------------------------------
>>>>>>>   The image name convention for the various scales is:
>>>>>>>     images/logo.scale-100.png
>>>>>>>     images/logo.scale-140.png
>>>>>>>     images/logo.scale-180.png
>>>>>>>    The 'ms-appx:///images/logo.png' uri is used to load the image
>>>>>>> in an
>>>>>>> application.
>>>>>>>    If we want to support this in the same way as it is done for Mac
>>>>>>> OS X
>>>>>>>    the WToolkit should return MultiResolution image in case if the
>>>>>>> loaded image has .scale-* qualifiers.
>>>>>>>    The Graphics class can request an image with necessary 
>>>>>>> resolution
>>>>>>> from the MultiResolution image.
>>>>>>>    It seems that nothing should be changed in the MultiResolution
>>>>>>> interface in this case.
>>>>>>>     Thanks,
>>>>>>>     Alexandr.
>>>>>>>>             ...jim
>>>>>>>> On 1/14/14 2:54 AM, Alexander Scherbatiy wrote:
>>>>>>>>> Hello,
>>>>>>>>> Could you review the fix:
>>>>>>>>>    bug: https://bugs.openjdk.java.net/browse/JDK-8029339
>>>>>>>>>    webrev: http://cr.openjdk.java.net/~alexsch/8029339/webrev.00
>>>>>>>>>    This is a proposal to introduce an API that allows to create a
>>>>>>>>> custom
>>>>>>>>> multi resolution image.
>>>>>>>>> I. It seems reasonable that the API should provide two basic
>>>>>>>>> operations:
>>>>>>>>>   1. Get the resolution variant based on the requested image
>>>>>>>>> width and
>>>>>>>>> height:
>>>>>>>>>      - Image getResolutionVariant(int width, int height)
>>>>>>>>>     Usually the system provides the scale factor which represents
>>>>>>>>> the
>>>>>>>>> number of pixels corresponding to each linear unit on the 
>>>>>>>>> display.
>>>>>>>>>     However, it has sense to combine the scale factor and the
>>>>>>>>> current
>>>>>>>>> transformations to get the actual image size to be displayed.
>>>>>>>>>   2. Get all provided resolution variants:
>>>>>>>>>     - List<Image> getResolutionVariants()
>>>>>>>>>    There are several uses cases:
>>>>>>>>>     - Create a new multi-resolution image based on the given
>>>>>>>>> multi-resolution image.
>>>>>>>>>     - Pass to the native system the multi-resolution image. For
>>>>>>>>> example,
>>>>>>>>> a use can set to the system the custom multi-resolution cursor.
>>>>>>>>> II. There are some possible ways where the new API can be added
>>>>>>>>>   1. java.awt.Image.
>>>>>>>>>    The 2 new methods can be added to the Image class. A user can
>>>>>>>>> override
>>>>>>>>>    the getResolutionVariant() and getResolutionVariants() 
>>>>>>>>> methods to
>>>>>>>>> provide the resolution variants
>>>>>>>>>    or there can be default implementations of these methods if a
>>>>>>>>> user
>>>>>>>>> puts resolution variants
>>>>>>>>>    to the list in the sorted order.
>>>>>>>>>    To check that the image has resolution variants the following
>>>>>>>>> statement can be used: image.getResolutionVariants().size() != 1
>>>>>>>>>    The disadvantage is that there is an overhead that the Image
>>>>>>>>> class
>>>>>>>>> should contain the List object and not all
>>>>>>>>>    images can have resolution variants.
>>>>>>>>>   2. Introduce new MultiResolutionImage interface.
>>>>>>>>>    A user should extend Image class and implement the
>>>>>>>>> MultiResolutionImage interface.
>>>>>>>>>    For example:
>>>>>>>>>    ---------------------
>>>>>>>>>      public class CustomMultiResolutionImage extends 
>>>>>>>>> BufferedImage
>>>>>>>>>              implements MultiResolutionImage {
>>>>>>>>>          Image highResolutionImage;
>>>>>>>>>          public CustomMultiResolutionImage(BufferedImage 
>>>>>>>>> baseImage,
>>>>>>>>>                  BufferedImage highResolutionImage) {
>>>>>>>>>              super(baseImage.getWidth(), baseImage.getHeight(),
>>>>>>>>> baseImage.getType());
>>>>>>>>>              this.highResolutionImage = highResolutionImage;
>>>>>>>>>              Graphics g = getGraphics();
>>>>>>>>>              g.drawImage(baseImage, 0, 0, null);
>>>>>>>>>              g.dispose();
>>>>>>>>>          }
>>>>>>>>>          @Override
>>>>>>>>>          public Image getResolutionVariant(int width, int 
>>>>>>>>> height) {
>>>>>>>>>              return ((width <= getWidth() && height <= 
>>>>>>>>> getHeight()))
>>>>>>>>>                      ? this : highResolutionImage;
>>>>>>>>>          }
>>>>>>>>>          @Override
>>>>>>>>>          public List<Image> getResolutionVariants() {
>>>>>>>>>              return Arrays.asList(this, highResolutionImage);
>>>>>>>>>          }
>>>>>>>>>      }
>>>>>>>>>    ---------------------
>>>>>>>>>   The current fix adds the MultiResolutionImage interface and 
>>>>>>>>> public
>>>>>>>>> resolution variant rendering hints.
>>>>>>>>> Thanks,
>>>>>>>>> Alexandr.

More information about the 2d-dev mailing list