<AWT Dev> [OpenJDK 2D-Dev] [9] Review request for 8029339 Custom MultiResolution image support on HiDPI displays
Alexander Scherbatiy
alexandr.scherbatiy at oracle.com
Thu Mar 20 14:52:29 UTC 2014
Hello,
Could you review the updated version of the fix:
http://cr.openjdk.java.net/~alexsch/8029339/webrev.01/
- The "getResolutionVariant(int width, int height)" method from
MultiResolutionImage class is changed to
Image getResolutionVariant(float logicalDPIX, float logicalDPIY,
float width, float height, AffineTransform transform);
- sun.awt.image.ImageResolutionHelper class is added. The
sun.awt.image.MultiResolutionToolkitImage and
sun.awt.image.MultiResolutionBufferedImage classes are used
PLATFORM ImageResolutionHelper.
The MultiResolutionImage interface implementation could look like:
------------------------------
public class CustomMultiResolutionImage extends BufferedImage implements
MultiResolutionImage {
private final Image[] resolutionVariants;
public CustomMultiResolutionImage(int baseIndex, Image... images) {
super(images[baseIndex].getWidth(null),
images[baseIndex].getHeight(null),
BufferedImage.TYPE_INT_RGB);
this.resolutionVariants = images;
Graphics g = getGraphics();
g.drawImage(images[baseIndex], 0, 0, null);
g.dispose();
}
@Override
public Image getResolutionVariant(float logicalDPIX, float logicalDPIY,
float width, float height, AffineTransform transform) {
return getResolutionVariant(logicalDPIX * width, logicalDPIY *
height);
}
@Override
public List<Image> getResolutionVariants() {
return Arrays.asList(resolutionVariants);
}
public Image getResolutionVariant(double width, double height) {
for (Image image : resolutionVariants) {
if (width <= image.getWidth(null) && height <=
image.getHeight(null)) {
return image;
}
}
return this;
}
}
------------------------------
Thanks,
Alexandr.
On 2/27/2014 4:54 PM, Alexander Scherbatiy wrote:
> On 2/22/2014 3:54 AM, Jim Graham wrote:
>> Hi Alexandr,
>>
>> On 2/18/14 7:33 AM, Alexander Scherbatiy wrote:
>>> Hi Jim,
>>>
>>> Let's divide the discussion into two part.
>>>
>>> 1. Where it is better to hold resolution variants?
>>> Putting resolution variants in Image class brings some
>>> questions like:
>>> - Some type of images do not need to have resolution variants
>>> - Should resolution variants have the same type as the base image?
>>> - getResolutionVariants() method can return copy of the original
>>> list
>>> so add/removeRV methods should be also added.
>>>
>>> There are pros and cons for placing resolution variants to Image
>>> class or to a separate intreface.
>>
>> I agree that this could be a separate interface. In my examples
>> below I was just sticking them inside an "Image{}" to show where they
>> lived in the set of involved objects, not a specific recommendation
>> that they actually be new methods on the base class itself. I
>> probably should have put a comment there about that.
>>
>> With respect to add/remove - that is assuming a need for manual
>> construction of an image set, right? Forgive me if I'm forgetting
>> something, but I seem to recall that manual Multi-Res images was
>> proposed as a way for developers to introduce @2x support themselves,
>> but if we are internally managing @2x and -DPI variants for them,
>> then I'm not sure if there is actual developer need to manually
>> construct their own. Am I forgetting something?
> The NSImage has addRepresentation/removeRepresentation methods to
> work with image representations on Mac OS X.
> The java.awt.Image class should provide similar functionality to
> have the possibilities as Cocoa on HiDPI displays.
>
>>
>>> 2. Using scale factor/image sizes/scaled image sizes to retreive a
>>> resolution variant.
>>>
>>> May be it is better to have a structure that provide all necessary
>>> information to query the resolution variant: scale factor, draw area
>>> width/height, transformed area width/height?
>>>
>>> For example:
>>> ---------------------
>>> public interface MultiResolutionImage {
>>>
>>> interface DrawAreaInfo {
>>>
>>> float getScaleFactor();
>>> float getAreaWidth();
>>> float getAreaHeight();
>>> float getTransformedAreaWidth();
>>> float getTransformedAreaHeight();
>>> }
>>>
>>> public Image getResolutionVariant(DrawAreaInfo drawAreaInfo) ;
>>> public List<Image> getResolutionVariants();
>>> }
>>> ---------------------
>>
>> The problem with a constructor is that this is something that is
>> (potentially) done on every drawImage() call, which means we are
>> inviting GC into the equation. If we can come up with a simple "just
>> a couple/3/4 numbers" way to embed that data into a method call
>> argument list then we can make this lighter weight.
>>
>> What about simply having floating point (double) dimensions on the
>> rendered size
> There should be a way to choose a resolution variant based on
> requested drawing size or transformed drawing size.
> At least a current transformation should be included too.
>> plus a single floating point "logical DPI" for the screen?
> There is the ID2D1Factory::GetDesktopDpi method which returns
> dpiX and dpiY.
> http://msdn.microsoft.com/en-us/library/windows/apps/dd371316
>
> That means that logicalDPIX/Y can have different values.
> At least it is described in the
> http://msdn.microsoft.com/en-us/library/windows/apps/ff684173
> "To get the DPI setting, call the ID2D1Factory::GetDesktopDpi
> method. The DPI is returned as two floating-point values, one for the
> x-axis and one for the y-axis. In theory, these values can differ.
> Calculate a separate scaling factor for each axis."
>
> The getResolutionVariant method could look like:
> --------------------------------------
> public Image getResolutionVariant(float logicalDPIX, float
> logicalDPIY,
> float widthX, float widthY, AffineTransform transform);
> --------------------------------------
>
>> If the image is known (either passed as an argument or the method is
>> called on the image), then it can provide the original WH.
>>
>>> The MultiResolutionImage default implementation could allow to use
>>> different strategies like scale factor/transfom/OS based
>>> to query a resolution variant. The OS based strategy can be used by
>>> default.
>>
>> For Mac policy, all we need is the transformed dimensions, which can
>> be passed in as FP for generality. For Windows policy, all we need
>> is logical DPI for the screen. What other information would we
>> need, or would an algorithm like to use, that can't be computed from
>> those 2 pieces?
> The aim is to provide a base class that can be used to create a
> MultiResolutionImage like:
> http://hg.openjdk.java.net/jdk9/client/jdk/diff/ae53ebce5fa3/src/share/classes/sun/awt/image/MultiResolutionBufferedImage.java
>
>
> A developer should be able to implement a custom algorithm to
> query a resolution variant.
>
> It can be done by overriding the getResolutionVariant image:
> -----------------------
> Image mrImage = new MultiResolutionBufferedImage(){
> @Override
> public Image getResolutionVariant(...) {
> // Custom logic here
> }
> };
> -----------------------
>
> Or it can be done by using resolution variant choosers so a
> developer can implement custom resolution variant query:
> -----------------------
> public class MultiResolutionBufferedImage implements
> MultiResolutionImage{
>
> interface ResolutionVariantChooser{
> Image getResolutionVariant(dpi, size,..., List<Image>
> resolutionVariants);
> }
> ResolutionVariantChooser TRANSFORM_BASED = null;
> ResolutionVariantChooser DPI_BASED = null;
>
> ResolutionVariantChooser rvChooser;
>
> @Override
> public Image getResolutionVariant(dpi, size,...,) {
> return rvChooser.getResolutionVariant(dpi, size,...,
> getResolutionVariants());
> }
> }
> -----------------------
>
> Thanks,
> Alexandr.
>
>>
>> ...jim
>>
>>> Thanks,
>>> Alexandr.
>>>
>>>
>>> On 2/13/2014 4:42 AM, Jim Graham wrote:
>>>> On 2/12/14 5:59 AM, Alexander Scherbatiy wrote:
>>>>> On 2/8/2014 4:19 AM, Jim Graham wrote:
>>>>>> The primary thing that I was concerned about was the presence of
>>>>>> integers in the API when Windows uses non-integer multiples
>>>>> It would make sense to pass real numbers to the
>>>>> getResolutionVariant() method if the difference between resolution
>>>>> variants sizes is 1.
>>>>> It seems that it is not a common case.
>>>>
>>>> I was thinking of other API that is related to this, such as the API
>>>> that queries the scaling factor from a SurfaceManager. I seem to
>>>> remember some integer return values in that, but Windows might have
>>>> the answer 1.4 or 1.8, depending on the screen scaling factor that was
>>>> determined from the UI.
>>>>
>>>> In terms of the getResolutionVariant() method here, those non-integer
>>>> screen scaling factors don't directly impact this API. But, we have
>>>> some issues with the use of integers there from other sources:
>>>>
>>>> - That API assumes that the caller will determine the pixel size
>>>> needed, but the actual media choice is determined with different
>>>> techniques on Mac and Windows so this means that the caller will have
>>>> to worry about platform conventions. Is that the right tradeoff?
>>>>
>>>> - The technique recommended for Mac involves computing the precise
>>>> size desired using the current transform, which may be a floating
>>>> point value, so the integer values used in this API are already
>>>> approximations and there is no documentation on how to generate the
>>>> proper integer. In particular, the current code in SG2D naively uses
>>>> a cast to integer to determine the values to supply which means a
>>>> transformed size of W+0.5 will be truncated to W and the lower
>>>> resolution image will be used. Does that conform to Mac guidelines? Do
>>>> they require the truncated size to reach W+1 before the next size is
>>>> used? Passing in float or double values would sidestep all of that
>>>> since then the comparisons would be done with full precision, but as
>>>> long as we can determine a "best practices compatible with all
>>>> platforms" rule on how to round to integers, then integers are OK
>>>> there.
>>>>
>>>> - The Windows document you cite below suggests that the determination
>>>> should be made by looking at the Screen DPI and choosing the next
>>>> higher media variant based on that screen DPI. They do not specify
>>>> choosing media based on the current transform as is done for Mac. If
>>>> we stick with supplying values that are used to determine which media
>>>> to use, then on Windows we should not take the transform into account,
>>>> but instead query the SurfaceManager for the scale factor and only
>>>> transform by those values (even if the current transform was manually
>>>> overridden to identity).
>>>>
>>>> There are pros and cons to both approaches.
>>>>
>>>> Mac ensures that you are always using the best media for any given
>>>> render operation.
>>>>
>>>> But, Windows ensure more consistency in the face of other scaling.
>>>>
>>>> The thing to consider is that if you have a 500x500 image with a
>>>> 1000x1000 variant and you rendering it at 500x500 and then 501x501,
>>>> that first jump will be fairly jarring as the scaled version of the
>>>> 1000x1000 will not look precisely like the original 500x500 did. With
>>>> @2x images only, this effect is minimized so the advantage of always
>>>> using "the best media for a given render operation" may outweigh the
>>>> inconsistency issue. But, on Windows where the media are 1.4x or 1.8x
>>>> in size, a downscaled image will start to show more interpolation
>>>> noise and so the balance of the two choices may shift more towards not
>>>> wanting a jarring shift.
>>>>
>>>> We might want one or more of the following:
>>>>
>>>> - Developer chooses policy (TX_AWARE, DPI_AWARE, ALWAYS_LARGEST, NONE,
>>>> PLATFORM) where the last policy would use TX_AWARE on Mac and
>>>> DPI_AWARE on Windows
>>>>
>>>> - We create our own policy and always use it (TX_AWARE? or
>>>> DPI_AWARE?)
>>>>
>>>> - We create our own policy that dynamically chooses one of the above
>>>> strategies depending on platform or available media or ???
>>>>
>>>> - We could create an optional interface for them to install their own
>>>> algorithm as well. I think it would work best as a delegate interface
>>>> that one installs into Image so that it can be used with any image
>>>> without having to subclass (it wouldn't really have much to do for
>>>> BufferedImages or VolatileImages, though):
>>>>
>>>> class Image {
>>>> void setResolutionHelper(ImageResolutionHelper foo);
>>>> List<Image> getResolutionVariants();
>>>> }
>>>>
>>>> or:
>>>>
>>>> class Graphics {
>>>> void setResolutionHelper(ImageResolutionHelper foo);
>>>> }
>>>>
>>>> or - anywhere else it could be installed more centrally (per App
>>>> context)?
>>>>
>>>> and the interface would be something like one of these variants:
>>>>
>>>> interface ImageResolutionHelper {
>>>> // This version would prevent substituting a random image:
>>>> // They have to return an index into the List<Image> for that
>>>> image...
>>>> public int chooseVariant(Image img, double dpi, number w,
>>>> number h);
>>>>
>>>> or:
>>>>
>>>> // This version would allow substituting an arbitrary image:
>>>> public Image getVariant(Image img, double dpi, number w, number
>>>> h);
>>>> }
>>>>
>>>> Since they would be in full control of the policy, though, we would
>>>> unfortunately always have to call this, there would be no more testing
>>>> in SG2D to see "if" we need to deal with DPI, though perhaps we could
>>>> document some internal conditions in which we do not call it for
>>>> common cases (but that would have to be well agreed not to get in the
>>>> way of reasonable uses of the API and well documented)?
>>>>
>>>> Note that we would have to do a security audit to make sure that
>>>> random image substitution couldn't allow any sort of "screen phishing"
>>>> exploit.
>>>>
>>>> ...jim
>>>>
>>>>>> and also what policy they use for choosing scaled images.
>>>>>>
>>>>>> I don't see a mention of taking the current transform into account,
>>>>>> just physical issues like screen DPI and form factor. They talk
>>>>>> about
>>>>>> resolution plateaus and in their recommendations section they
>>>>>> tell the
>>>>>> developer to use a particular property that tells them the screen
>>>>>> resolution to figure out which image to load if they are loading
>>>>>> manually. There is no discussion about dynamically loading multiple
>>>>>> versions of the image based on a dynamic program-applied transform
>>>>>> factor as is done on MacOS.
>>>>>>
>>>>>> Also, they tell developers to draw images to a specific size rather
>>>>>> than using auto-sizing. That begs the question as to how they
>>>>>> interpret a call to draw an image just using a location in the
>>>>>> presence of various DPI factors.
>>>>> There is an interesting doc that describes how to write
>>>>> DPI-aware
>>>>> Win32 applications:
>>>>> http://msdn.microsoft.com/en-us/library/dd464646%28v=vs.85%29.aspx
>>>>> It is suggested to handle WM_DPICHANGED message, load the
>>>>> graphic
>>>>> that has slightly greater resolution to the current DPI and use
>>>>> StretchBlt
>>>>> to scale down the image.
>>>>>
>>>>> Thanks,
>>>>> Alexandr.
>>>>>
>>>>>>
>>>>>> ...jim
>>>>>>
>>>>>> On 2/7/14 3:00 AM, Alexander Scherbatiy wrote:
>>>>>>> On 1/22/2014 6:40 AM, Jim Graham wrote:
>>>>>>>> Hi Alexander,
>>>>>>>>
>>>>>>>> Before we get too far down the road on this API, I think we
>>>>>>>> understand
>>>>>>>> the way in which MacOS processes multi-resolution images for HiDPI
>>>>>>>> screens, but have we investigated the processes that Windows uses
>>>>>>>> under Windows 8? My impression is that Windows 8 has included a
>>>>>>>> number of new techniques for dealing with the high resolution
>>>>>>>> displays
>>>>>>>> that it will run on in the Windows tablet and mobile industries
>>>>>>>> and
>>>>>>>> that these will also come into play as 4K displays (already
>>>>>>>> available)
>>>>>>>> become more common on the desktop. We should make sure that
>>>>>>>> what we
>>>>>>>> come up with here can provide native compatibility with either
>>>>>>>> platform's policies and standard practices.
>>>>>>>>
>>>>>>>> If you've investigated the MS policies I'd like to see a
>>>>>>>> summary so
>>>>>>>> that we can consider them as we review this API...
>>>>>>> There is the Windows Guidelines for scaling to pixel density:
>>>>>>> http://msdn.microsoft.com/en-us/library/windows/apps/hh465362.aspx
>>>>>>> which says that Windows has automatic resource loading that
>>>>>>> supports
>>>>>>> three version of images scaling (100%, 140%, and 180%)
>>>>>>> --------------------------------
>>>>>>> Without scaling, as the pixel density of a display device
>>>>>>> increases, the
>>>>>>> physical sizes of objects on screen get smaller.
>>>>>>> When UI would otherwise be too small to touch and when text gets
>>>>>>> too
>>>>>>> small to read,
>>>>>>> Windows scales the system and app UI to one of the following
>>>>>>> scaling
>>>>>>> plateaus:
>>>>>>>
>>>>>>> 1.0 (100%, no scaling is applied)
>>>>>>> 1.4 (140% scaling)
>>>>>>> 1.8 (180% scaling)
>>>>>>>
>>>>>>> Windows determines which scaling plateau to use based on the
>>>>>>> physical
>>>>>>> screen size, the screen resolution, the DPI of the screen, and form
>>>>>>> factor.
>>>>>>>
>>>>>>> Use resource loading for bitmap images in the app package For
>>>>>>> bitmap
>>>>>>> images stored
>>>>>>> in the app package, provide a separate image for each scaling
>>>>>>> factor(100%, 140%, and 180%),
>>>>>>> and name your image files using the "scale" naming convention
>>>>>>> described
>>>>>>> below.
>>>>>>> Windows loads the right image for the current scale automatically.
>>>>>>> --------------------------------
>>>>>>>
>>>>>>> The image name convention for the various scales is:
>>>>>>> images/logo.scale-100.png
>>>>>>> images/logo.scale-140.png
>>>>>>> images/logo.scale-180.png
>>>>>>>
>>>>>>> The 'ms-appx:///images/logo.png' uri is used to load the image
>>>>>>> in an
>>>>>>> application.
>>>>>>>
>>>>>>> If we want to support this in the same way as it is done for Mac
>>>>>>> OS X
>>>>>>> the WToolkit should return MultiResolution image in case if the
>>>>>>> loaded image has .scale-* qualifiers.
>>>>>>> The Graphics class can request an image with necessary
>>>>>>> resolution
>>>>>>> from the MultiResolution image.
>>>>>>>
>>>>>>> It seems that nothing should be changed in the MultiResolution
>>>>>>> interface in this case.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Alexandr.
>>>>>>>>
>>>>>>>> ...jim
>>>>>>>>
>>>>>>>> On 1/14/14 2:54 AM, Alexander Scherbatiy wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> Could you review the fix:
>>>>>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8029339
>>>>>>>>> webrev: http://cr.openjdk.java.net/~alexsch/8029339/webrev.00
>>>>>>>>>
>>>>>>>>> This is a proposal to introduce an API that allows to create a
>>>>>>>>> custom
>>>>>>>>> multi resolution image.
>>>>>>>>>
>>>>>>>>> I. It seems reasonable that the API should provide two basic
>>>>>>>>> operations:
>>>>>>>>>
>>>>>>>>> 1. Get the resolution variant based on the requested image
>>>>>>>>> width and
>>>>>>>>> height:
>>>>>>>>> - Image getResolutionVariant(int width, int height)
>>>>>>>>>
>>>>>>>>> Usually the system provides the scale factor which represents
>>>>>>>>> the
>>>>>>>>> number of pixels corresponding to each linear unit on the
>>>>>>>>> display.
>>>>>>>>> However, it has sense to combine the scale factor and the
>>>>>>>>> current
>>>>>>>>> transformations to get the actual image size to be displayed.
>>>>>>>>>
>>>>>>>>> 2. Get all provided resolution variants:
>>>>>>>>> - List<Image> getResolutionVariants()
>>>>>>>>>
>>>>>>>>> There are several uses cases:
>>>>>>>>> - Create a new multi-resolution image based on the given
>>>>>>>>> multi-resolution image.
>>>>>>>>> - Pass to the native system the multi-resolution image. For
>>>>>>>>> example,
>>>>>>>>> a use can set to the system the custom multi-resolution cursor.
>>>>>>>>>
>>>>>>>>> II. There are some possible ways where the new API can be added
>>>>>>>>>
>>>>>>>>> 1. java.awt.Image.
>>>>>>>>>
>>>>>>>>> The 2 new methods can be added to the Image class. A user can
>>>>>>>>> override
>>>>>>>>> the getResolutionVariant() and getResolutionVariants()
>>>>>>>>> methods to
>>>>>>>>> provide the resolution variants
>>>>>>>>> or there can be default implementations of these methods if a
>>>>>>>>> user
>>>>>>>>> puts resolution variants
>>>>>>>>> to the list in the sorted order.
>>>>>>>>>
>>>>>>>>> To check that the image has resolution variants the following
>>>>>>>>> statement can be used: image.getResolutionVariants().size() != 1
>>>>>>>>>
>>>>>>>>> The disadvantage is that there is an overhead that the Image
>>>>>>>>> class
>>>>>>>>> should contain the List object and not all
>>>>>>>>> images can have resolution variants.
>>>>>>>>>
>>>>>>>>> 2. Introduce new MultiResolutionImage interface.
>>>>>>>>>
>>>>>>>>> A user should extend Image class and implement the
>>>>>>>>> MultiResolutionImage interface.
>>>>>>>>>
>>>>>>>>> For example:
>>>>>>>>> ---------------------
>>>>>>>>> public class CustomMultiResolutionImage extends
>>>>>>>>> BufferedImage
>>>>>>>>> implements MultiResolutionImage {
>>>>>>>>>
>>>>>>>>> Image highResolutionImage;
>>>>>>>>>
>>>>>>>>> public CustomMultiResolutionImage(BufferedImage
>>>>>>>>> baseImage,
>>>>>>>>> BufferedImage highResolutionImage) {
>>>>>>>>> super(baseImage.getWidth(), baseImage.getHeight(),
>>>>>>>>> baseImage.getType());
>>>>>>>>> this.highResolutionImage = highResolutionImage;
>>>>>>>>> Graphics g = getGraphics();
>>>>>>>>> g.drawImage(baseImage, 0, 0, null);
>>>>>>>>> g.dispose();
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> @Override
>>>>>>>>> public Image getResolutionVariant(int width, int
>>>>>>>>> height) {
>>>>>>>>> return ((width <= getWidth() && height <=
>>>>>>>>> getHeight()))
>>>>>>>>> ? this : highResolutionImage;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> @Override
>>>>>>>>> public List<Image> getResolutionVariants() {
>>>>>>>>> return Arrays.asList(this, highResolutionImage);
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> ---------------------
>>>>>>>>>
>>>>>>>>> The current fix adds the MultiResolutionImage interface and
>>>>>>>>> public
>>>>>>>>> resolution variant rendering hints.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Alexandr.
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
More information about the awt-dev
mailing list