[OpenJDK 2D-Dev] [9] Review request for 8029339 Custom MultiResolution image support on HiDPI displays
Alexander Scherbatiy
alexandr.scherbatiy at oracle.com
Thu Feb 27 12:54:42 UTC 2014
On 2/22/2014 3:54 AM, Jim Graham wrote:
> Hi Alexandr,
>
> On 2/18/14 7:33 AM, Alexander Scherbatiy wrote:
>> Hi Jim,
>>
>> Let's divide the discussion into two part.
>>
>> 1. Where it is better to hold resolution variants?
>> Putting resolution variants in Image class brings some questions
>> like:
>> - Some type of images do not need to have resolution variants
>> - Should resolution variants have the same type as the base image?
>> - getResolutionVariants() method can return copy of the original list
>> so add/removeRV methods should be also added.
>>
>> There are pros and cons for placing resolution variants to Image
>> class or to a separate intreface.
>
> I agree that this could be a separate interface. In my examples below
> I was just sticking them inside an "Image{}" to show where they lived
> in the set of involved objects, not a specific recommendation that
> they actually be new methods on the base class itself. I probably
> should have put a comment there about that.
>
> With respect to add/remove - that is assuming a need for manual
> construction of an image set, right? Forgive me if I'm forgetting
> something, but I seem to recall that manual Multi-Res images was
> proposed as a way for developers to introduce @2x support themselves,
> but if we are internally managing @2x and -DPI variants for them, then
> I'm not sure if there is actual developer need to manually construct
> their own. Am I forgetting something?
The NSImage has addRepresentation/removeRepresentation methods to
work with image representations on Mac OS X.
The java.awt.Image class should provide similar functionality to
have the possibilities as Cocoa on HiDPI displays.
>
>> 2. Using scale factor/image sizes/scaled image sizes to retreive a
>> resolution variant.
>>
>> May be it is better to have a structure that provide all necessary
>> information to query the resolution variant: scale factor, draw area
>> width/height, transformed area width/height?
>>
>> For example:
>> ---------------------
>> public interface MultiResolutionImage {
>>
>> interface DrawAreaInfo {
>>
>> float getScaleFactor();
>> float getAreaWidth();
>> float getAreaHeight();
>> float getTransformedAreaWidth();
>> float getTransformedAreaHeight();
>> }
>>
>> public Image getResolutionVariant(DrawAreaInfo drawAreaInfo) ;
>> public List<Image> getResolutionVariants();
>> }
>> ---------------------
>
> The problem with a constructor is that this is something that is
> (potentially) done on every drawImage() call, which means we are
> inviting GC into the equation. If we can come up with a simple "just
> a couple/3/4 numbers" way to embed that data into a method call
> argument list then we can make this lighter weight.
>
> What about simply having floating point (double) dimensions on the
> rendered size
There should be a way to choose a resolution variant based on
requested drawing size or transformed drawing size.
At least a current transformation should be included too.
> plus a single floating point "logical DPI" for the screen?
There is the ID2D1Factory::GetDesktopDpi method which returns dpiX
and dpiY.
http://msdn.microsoft.com/en-us/library/windows/apps/dd371316
That means that logicalDPIX/Y can have different values.
At least it is described in the
http://msdn.microsoft.com/en-us/library/windows/apps/ff684173
"To get the DPI setting, call the ID2D1Factory::GetDesktopDpi
method. The DPI is returned as two floating-point values, one for the
x-axis and one for the y-axis. In theory, these values can differ.
Calculate a separate scaling factor for each axis."
The getResolutionVariant method could look like:
--------------------------------------
public Image getResolutionVariant(float logicalDPIX, float logicalDPIY,
float widthX, float widthY, AffineTransform transform);
--------------------------------------
> If the image is known (either passed as an argument or the method is
> called on the image), then it can provide the original WH.
>
>> The MultiResolutionImage default implementation could allow to use
>> different strategies like scale factor/transfom/OS based
>> to query a resolution variant. The OS based strategy can be used by
>> default.
>
> For Mac policy, all we need is the transformed dimensions, which can
> be passed in as FP for generality. For Windows policy, all we need is
> logical DPI for the screen. What other information would we need, or
> would an algorithm like to use, that can't be computed from those 2
> pieces?
The aim is to provide a base class that can be used to create a
MultiResolutionImage like:
http://hg.openjdk.java.net/jdk9/client/jdk/diff/ae53ebce5fa3/src/share/classes/sun/awt/image/MultiResolutionBufferedImage.java
A developer should be able to implement a custom algorithm to
query a resolution variant.
It can be done by overriding the getResolutionVariant image:
-----------------------
Image mrImage = new MultiResolutionBufferedImage(){
@Override
public Image getResolutionVariant(...) {
// Custom logic here
}
};
-----------------------
Or it can be done by using resolution variant choosers so a
developer can implement custom resolution variant query:
-----------------------
public class MultiResolutionBufferedImage implements MultiResolutionImage{
interface ResolutionVariantChooser{
Image getResolutionVariant(dpi, size,..., List<Image>
resolutionVariants);
}
ResolutionVariantChooser TRANSFORM_BASED = null;
ResolutionVariantChooser DPI_BASED = null;
ResolutionVariantChooser rvChooser;
@Override
public Image getResolutionVariant(dpi, size,...,) {
return rvChooser.getResolutionVariant(dpi, size,...,
getResolutionVariants());
}
}
-----------------------
Thanks,
Alexandr.
>
> ...jim
>
>> Thanks,
>> Alexandr.
>>
>>
>> On 2/13/2014 4:42 AM, Jim Graham wrote:
>>> On 2/12/14 5:59 AM, Alexander Scherbatiy wrote:
>>>> On 2/8/2014 4:19 AM, Jim Graham wrote:
>>>>> The primary thing that I was concerned about was the presence of
>>>>> integers in the API when Windows uses non-integer multiples
>>>> It would make sense to pass real numbers to the
>>>> getResolutionVariant() method if the difference between resolution
>>>> variants sizes is 1.
>>>> It seems that it is not a common case.
>>>
>>> I was thinking of other API that is related to this, such as the API
>>> that queries the scaling factor from a SurfaceManager. I seem to
>>> remember some integer return values in that, but Windows might have
>>> the answer 1.4 or 1.8, depending on the screen scaling factor that was
>>> determined from the UI.
>>>
>>> In terms of the getResolutionVariant() method here, those non-integer
>>> screen scaling factors don't directly impact this API. But, we have
>>> some issues with the use of integers there from other sources:
>>>
>>> - That API assumes that the caller will determine the pixel size
>>> needed, but the actual media choice is determined with different
>>> techniques on Mac and Windows so this means that the caller will have
>>> to worry about platform conventions. Is that the right tradeoff?
>>>
>>> - The technique recommended for Mac involves computing the precise
>>> size desired using the current transform, which may be a floating
>>> point value, so the integer values used in this API are already
>>> approximations and there is no documentation on how to generate the
>>> proper integer. In particular, the current code in SG2D naively uses
>>> a cast to integer to determine the values to supply which means a
>>> transformed size of W+0.5 will be truncated to W and the lower
>>> resolution image will be used. Does that conform to Mac guidelines? Do
>>> they require the truncated size to reach W+1 before the next size is
>>> used? Passing in float or double values would sidestep all of that
>>> since then the comparisons would be done with full precision, but as
>>> long as we can determine a "best practices compatible with all
>>> platforms" rule on how to round to integers, then integers are OK
>>> there.
>>>
>>> - The Windows document you cite below suggests that the determination
>>> should be made by looking at the Screen DPI and choosing the next
>>> higher media variant based on that screen DPI. They do not specify
>>> choosing media based on the current transform as is done for Mac. If
>>> we stick with supplying values that are used to determine which media
>>> to use, then on Windows we should not take the transform into account,
>>> but instead query the SurfaceManager for the scale factor and only
>>> transform by those values (even if the current transform was manually
>>> overridden to identity).
>>>
>>> There are pros and cons to both approaches.
>>>
>>> Mac ensures that you are always using the best media for any given
>>> render operation.
>>>
>>> But, Windows ensure more consistency in the face of other scaling.
>>>
>>> The thing to consider is that if you have a 500x500 image with a
>>> 1000x1000 variant and you rendering it at 500x500 and then 501x501,
>>> that first jump will be fairly jarring as the scaled version of the
>>> 1000x1000 will not look precisely like the original 500x500 did. With
>>> @2x images only, this effect is minimized so the advantage of always
>>> using "the best media for a given render operation" may outweigh the
>>> inconsistency issue. But, on Windows where the media are 1.4x or 1.8x
>>> in size, a downscaled image will start to show more interpolation
>>> noise and so the balance of the two choices may shift more towards not
>>> wanting a jarring shift.
>>>
>>> We might want one or more of the following:
>>>
>>> - Developer chooses policy (TX_AWARE, DPI_AWARE, ALWAYS_LARGEST, NONE,
>>> PLATFORM) where the last policy would use TX_AWARE on Mac and
>>> DPI_AWARE on Windows
>>>
>>> - We create our own policy and always use it (TX_AWARE? or DPI_AWARE?)
>>>
>>> - We create our own policy that dynamically chooses one of the above
>>> strategies depending on platform or available media or ???
>>>
>>> - We could create an optional interface for them to install their own
>>> algorithm as well. I think it would work best as a delegate interface
>>> that one installs into Image so that it can be used with any image
>>> without having to subclass (it wouldn't really have much to do for
>>> BufferedImages or VolatileImages, though):
>>>
>>> class Image {
>>> void setResolutionHelper(ImageResolutionHelper foo);
>>> List<Image> getResolutionVariants();
>>> }
>>>
>>> or:
>>>
>>> class Graphics {
>>> void setResolutionHelper(ImageResolutionHelper foo);
>>> }
>>>
>>> or - anywhere else it could be installed more centrally (per App
>>> context)?
>>>
>>> and the interface would be something like one of these variants:
>>>
>>> interface ImageResolutionHelper {
>>> // This version would prevent substituting a random image:
>>> // They have to return an index into the List<Image> for that
>>> image...
>>> public int chooseVariant(Image img, double dpi, number w, number
>>> h);
>>>
>>> or:
>>>
>>> // This version would allow substituting an arbitrary image:
>>> public Image getVariant(Image img, double dpi, number w, number h);
>>> }
>>>
>>> Since they would be in full control of the policy, though, we would
>>> unfortunately always have to call this, there would be no more testing
>>> in SG2D to see "if" we need to deal with DPI, though perhaps we could
>>> document some internal conditions in which we do not call it for
>>> common cases (but that would have to be well agreed not to get in the
>>> way of reasonable uses of the API and well documented)?
>>>
>>> Note that we would have to do a security audit to make sure that
>>> random image substitution couldn't allow any sort of "screen phishing"
>>> exploit.
>>>
>>> ...jim
>>>
>>>>> and also what policy they use for choosing scaled images.
>>>>>
>>>>> I don't see a mention of taking the current transform into account,
>>>>> just physical issues like screen DPI and form factor. They talk about
>>>>> resolution plateaus and in their recommendations section they tell
>>>>> the
>>>>> developer to use a particular property that tells them the screen
>>>>> resolution to figure out which image to load if they are loading
>>>>> manually. There is no discussion about dynamically loading multiple
>>>>> versions of the image based on a dynamic program-applied transform
>>>>> factor as is done on MacOS.
>>>>>
>>>>> Also, they tell developers to draw images to a specific size rather
>>>>> than using auto-sizing. That begs the question as to how they
>>>>> interpret a call to draw an image just using a location in the
>>>>> presence of various DPI factors.
>>>> There is an interesting doc that describes how to write
>>>> DPI-aware
>>>> Win32 applications:
>>>> http://msdn.microsoft.com/en-us/library/dd464646%28v=vs.85%29.aspx
>>>> It is suggested to handle WM_DPICHANGED message, load the
>>>> graphic
>>>> that has slightly greater resolution to the current DPI and use
>>>> StretchBlt
>>>> to scale down the image.
>>>>
>>>> Thanks,
>>>> Alexandr.
>>>>
>>>>>
>>>>> ...jim
>>>>>
>>>>> On 2/7/14 3:00 AM, Alexander Scherbatiy wrote:
>>>>>> On 1/22/2014 6:40 AM, Jim Graham wrote:
>>>>>>> Hi Alexander,
>>>>>>>
>>>>>>> Before we get too far down the road on this API, I think we
>>>>>>> understand
>>>>>>> the way in which MacOS processes multi-resolution images for HiDPI
>>>>>>> screens, but have we investigated the processes that Windows uses
>>>>>>> under Windows 8? My impression is that Windows 8 has included a
>>>>>>> number of new techniques for dealing with the high resolution
>>>>>>> displays
>>>>>>> that it will run on in the Windows tablet and mobile industries and
>>>>>>> that these will also come into play as 4K displays (already
>>>>>>> available)
>>>>>>> become more common on the desktop. We should make sure that
>>>>>>> what we
>>>>>>> come up with here can provide native compatibility with either
>>>>>>> platform's policies and standard practices.
>>>>>>>
>>>>>>> If you've investigated the MS policies I'd like to see a summary so
>>>>>>> that we can consider them as we review this API...
>>>>>> There is the Windows Guidelines for scaling to pixel density:
>>>>>> http://msdn.microsoft.com/en-us/library/windows/apps/hh465362.aspx
>>>>>> which says that Windows has automatic resource loading that
>>>>>> supports
>>>>>> three version of images scaling (100%, 140%, and 180%)
>>>>>> --------------------------------
>>>>>> Without scaling, as the pixel density of a display device
>>>>>> increases, the
>>>>>> physical sizes of objects on screen get smaller.
>>>>>> When UI would otherwise be too small to touch and when text gets too
>>>>>> small to read,
>>>>>> Windows scales the system and app UI to one of the following scaling
>>>>>> plateaus:
>>>>>>
>>>>>> 1.0 (100%, no scaling is applied)
>>>>>> 1.4 (140% scaling)
>>>>>> 1.8 (180% scaling)
>>>>>>
>>>>>> Windows determines which scaling plateau to use based on the
>>>>>> physical
>>>>>> screen size, the screen resolution, the DPI of the screen, and form
>>>>>> factor.
>>>>>>
>>>>>> Use resource loading for bitmap images in the app package For bitmap
>>>>>> images stored
>>>>>> in the app package, provide a separate image for each scaling
>>>>>> factor(100%, 140%, and 180%),
>>>>>> and name your image files using the "scale" naming convention
>>>>>> described
>>>>>> below.
>>>>>> Windows loads the right image for the current scale automatically.
>>>>>> --------------------------------
>>>>>>
>>>>>> The image name convention for the various scales is:
>>>>>> images/logo.scale-100.png
>>>>>> images/logo.scale-140.png
>>>>>> images/logo.scale-180.png
>>>>>>
>>>>>> The 'ms-appx:///images/logo.png' uri is used to load the image
>>>>>> in an
>>>>>> application.
>>>>>>
>>>>>> If we want to support this in the same way as it is done for Mac
>>>>>> OS X
>>>>>> the WToolkit should return MultiResolution image in case if the
>>>>>> loaded image has .scale-* qualifiers.
>>>>>> The Graphics class can request an image with necessary resolution
>>>>>> from the MultiResolution image.
>>>>>>
>>>>>> It seems that nothing should be changed in the MultiResolution
>>>>>> interface in this case.
>>>>>>
>>>>>> Thanks,
>>>>>> Alexandr.
>>>>>>>
>>>>>>> ...jim
>>>>>>>
>>>>>>> On 1/14/14 2:54 AM, Alexander Scherbatiy wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Could you review the fix:
>>>>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8029339
>>>>>>>> webrev: http://cr.openjdk.java.net/~alexsch/8029339/webrev.00
>>>>>>>>
>>>>>>>> This is a proposal to introduce an API that allows to create a
>>>>>>>> custom
>>>>>>>> multi resolution image.
>>>>>>>>
>>>>>>>> I. It seems reasonable that the API should provide two basic
>>>>>>>> operations:
>>>>>>>>
>>>>>>>> 1. Get the resolution variant based on the requested image
>>>>>>>> width and
>>>>>>>> height:
>>>>>>>> - Image getResolutionVariant(int width, int height)
>>>>>>>>
>>>>>>>> Usually the system provides the scale factor which represents
>>>>>>>> the
>>>>>>>> number of pixels corresponding to each linear unit on the display.
>>>>>>>> However, it has sense to combine the scale factor and the
>>>>>>>> current
>>>>>>>> transformations to get the actual image size to be displayed.
>>>>>>>>
>>>>>>>> 2. Get all provided resolution variants:
>>>>>>>> - List<Image> getResolutionVariants()
>>>>>>>>
>>>>>>>> There are several uses cases:
>>>>>>>> - Create a new multi-resolution image based on the given
>>>>>>>> multi-resolution image.
>>>>>>>> - Pass to the native system the multi-resolution image. For
>>>>>>>> example,
>>>>>>>> a use can set to the system the custom multi-resolution cursor.
>>>>>>>>
>>>>>>>> II. There are some possible ways where the new API can be added
>>>>>>>>
>>>>>>>> 1. java.awt.Image.
>>>>>>>>
>>>>>>>> The 2 new methods can be added to the Image class. A user can
>>>>>>>> override
>>>>>>>> the getResolutionVariant() and getResolutionVariants()
>>>>>>>> methods to
>>>>>>>> provide the resolution variants
>>>>>>>> or there can be default implementations of these methods if a
>>>>>>>> user
>>>>>>>> puts resolution variants
>>>>>>>> to the list in the sorted order.
>>>>>>>>
>>>>>>>> To check that the image has resolution variants the following
>>>>>>>> statement can be used: image.getResolutionVariants().size() != 1
>>>>>>>>
>>>>>>>> The disadvantage is that there is an overhead that the Image
>>>>>>>> class
>>>>>>>> should contain the List object and not all
>>>>>>>> images can have resolution variants.
>>>>>>>>
>>>>>>>> 2. Introduce new MultiResolutionImage interface.
>>>>>>>>
>>>>>>>> A user should extend Image class and implement the
>>>>>>>> MultiResolutionImage interface.
>>>>>>>>
>>>>>>>> For example:
>>>>>>>> ---------------------
>>>>>>>> public class CustomMultiResolutionImage extends BufferedImage
>>>>>>>> implements MultiResolutionImage {
>>>>>>>>
>>>>>>>> Image highResolutionImage;
>>>>>>>>
>>>>>>>> public CustomMultiResolutionImage(BufferedImage
>>>>>>>> baseImage,
>>>>>>>> BufferedImage highResolutionImage) {
>>>>>>>> super(baseImage.getWidth(), baseImage.getHeight(),
>>>>>>>> baseImage.getType());
>>>>>>>> this.highResolutionImage = highResolutionImage;
>>>>>>>> Graphics g = getGraphics();
>>>>>>>> g.drawImage(baseImage, 0, 0, null);
>>>>>>>> g.dispose();
>>>>>>>> }
>>>>>>>>
>>>>>>>> @Override
>>>>>>>> public Image getResolutionVariant(int width, int
>>>>>>>> height) {
>>>>>>>> return ((width <= getWidth() && height <=
>>>>>>>> getHeight()))
>>>>>>>> ? this : highResolutionImage;
>>>>>>>> }
>>>>>>>>
>>>>>>>> @Override
>>>>>>>> public List<Image> getResolutionVariants() {
>>>>>>>> return Arrays.asList(this, highResolutionImage);
>>>>>>>> }
>>>>>>>> }
>>>>>>>> ---------------------
>>>>>>>>
>>>>>>>> The current fix adds the MultiResolutionImage interface and
>>>>>>>> public
>>>>>>>> resolution variant rendering hints.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Alexandr.
>>>>>>>>
>>>>>>
>>>>
>>
More information about the 2d-dev
mailing list