<AWT Dev> [9] Review request for 8029339 Custom MultiResolution image support on HiDPI displays
Alexander Scherbatiy
alexandr.scherbatiy at oracle.com
Tue Feb 18 07:33:17 PST 2014
Hi Jim,
Let's divide the discussion into two part.
1. Where it is better to hold resolution variants?
Putting resolution variants in Image class brings some questions like:
- Some type of images do not need to have resolution variants
- Should resolution variants have the same type as the base image?
- getResolutionVariants() method can return copy of the original list
so add/removeRV methods should be also added.
There are pros and cons for placing resolution variants to Image
class or to a separate intreface.
2. Using scale factor/image sizes/scaled image sizes to retreive a
resolution variant.
May be it is better to have a structure that provide all necessary
information to query the resolution variant: scale factor, draw area
width/height, transformed area width/height?
For example:
---------------------
public interface MultiResolutionImage {
interface DrawAreaInfo {
float getScaleFactor();
float getAreaWidth();
float getAreaHeight();
float getTransformedAreaWidth();
float getTransformedAreaHeight();
}
public Image getResolutionVariant(DrawAreaInfo drawAreaInfo) ;
public List<Image> getResolutionVariants();
}
---------------------
The MultiResolutionImage default implementation could allow to use
different strategies like scale factor/transfom/OS based
to query a resolution variant. The OS based strategy can be used by
default.
Thanks,
Alexandr.
On 2/13/2014 4:42 AM, Jim Graham wrote:
> On 2/12/14 5:59 AM, Alexander Scherbatiy wrote:
>> On 2/8/2014 4:19 AM, Jim Graham wrote:
>>> The primary thing that I was concerned about was the presence of
>>> integers in the API when Windows uses non-integer multiples
>> It would make sense to pass real numbers to the
>> getResolutionVariant() method if the difference between resolution
>> variants sizes is 1.
>> It seems that it is not a common case.
>
> I was thinking of other API that is related to this, such as the API
> that queries the scaling factor from a SurfaceManager. I seem to
> remember some integer return values in that, but Windows might have
> the answer 1.4 or 1.8, depending on the screen scaling factor that was
> determined from the UI.
>
> In terms of the getResolutionVariant() method here, those non-integer
> screen scaling factors don't directly impact this API. But, we have
> some issues with the use of integers there from other sources:
>
> - That API assumes that the caller will determine the pixel size
> needed, but the actual media choice is determined with different
> techniques on Mac and Windows so this means that the caller will have
> to worry about platform conventions. Is that the right tradeoff?
>
> - The technique recommended for Mac involves computing the precise
> size desired using the current transform, which may be a floating
> point value, so the integer values used in this API are already
> approximations and there is no documentation on how to generate the
> proper integer. In particular, the current code in SG2D naively uses
> a cast to integer to determine the values to supply which means a
> transformed size of W+0.5 will be truncated to W and the lower
> resolution image will be used. Does that conform to Mac guidelines?
> Do they require the truncated size to reach W+1 before the next size
> is used? Passing in float or double values would sidestep all of that
> since then the comparisons would be done with full precision, but as
> long as we can determine a "best practices compatible with all
> platforms" rule on how to round to integers, then integers are OK there.
>
> - The Windows document you cite below suggests that the determination
> should be made by looking at the Screen DPI and choosing the next
> higher media variant based on that screen DPI. They do not specify
> choosing media based on the current transform as is done for Mac. If
> we stick with supplying values that are used to determine which media
> to use, then on Windows we should not take the transform into account,
> but instead query the SurfaceManager for the scale factor and only
> transform by those values (even if the current transform was manually
> overridden to identity).
>
> There are pros and cons to both approaches.
>
> Mac ensures that you are always using the best media for any given
> render operation.
>
> But, Windows ensure more consistency in the face of other scaling.
>
> The thing to consider is that if you have a 500x500 image with a
> 1000x1000 variant and you rendering it at 500x500 and then 501x501,
> that first jump will be fairly jarring as the scaled version of the
> 1000x1000 will not look precisely like the original 500x500 did. With
> @2x images only, this effect is minimized so the advantage of always
> using "the best media for a given render operation" may outweigh the
> inconsistency issue. But, on Windows where the media are 1.4x or 1.8x
> in size, a downscaled image will start to show more interpolation
> noise and so the balance of the two choices may shift more towards not
> wanting a jarring shift.
>
> We might want one or more of the following:
>
> - Developer chooses policy (TX_AWARE, DPI_AWARE, ALWAYS_LARGEST, NONE,
> PLATFORM) where the last policy would use TX_AWARE on Mac and
> DPI_AWARE on Windows
>
> - We create our own policy and always use it (TX_AWARE? or DPI_AWARE?)
>
> - We create our own policy that dynamically chooses one of the above
> strategies depending on platform or available media or ???
>
> - We could create an optional interface for them to install their own
> algorithm as well. I think it would work best as a delegate interface
> that one installs into Image so that it can be used with any image
> without having to subclass (it wouldn't really have much to do for
> BufferedImages or VolatileImages, though):
>
> class Image {
> void setResolutionHelper(ImageResolutionHelper foo);
> List<Image> getResolutionVariants();
> }
>
> or:
>
> class Graphics {
> void setResolutionHelper(ImageResolutionHelper foo);
> }
>
> or - anywhere else it could be installed more centrally (per App
> context)?
>
> and the interface would be something like one of these variants:
>
> interface ImageResolutionHelper {
> // This version would prevent substituting a random image:
> // They have to return an index into the List<Image> for that
> image...
> public int chooseVariant(Image img, double dpi, number w, number h);
>
> or:
>
> // This version would allow substituting an arbitrary image:
> public Image getVariant(Image img, double dpi, number w, number h);
> }
>
> Since they would be in full control of the policy, though, we would
> unfortunately always have to call this, there would be no more testing
> in SG2D to see "if" we need to deal with DPI, though perhaps we could
> document some internal conditions in which we do not call it for
> common cases (but that would have to be well agreed not to get in the
> way of reasonable uses of the API and well documented)?
>
> Note that we would have to do a security audit to make sure that
> random image substitution couldn't allow any sort of "screen phishing"
> exploit.
>
> ...jim
>
>>> and also what policy they use for choosing scaled images.
>>>
>>> I don't see a mention of taking the current transform into account,
>>> just physical issues like screen DPI and form factor. They talk about
>>> resolution plateaus and in their recommendations section they tell the
>>> developer to use a particular property that tells them the screen
>>> resolution to figure out which image to load if they are loading
>>> manually. There is no discussion about dynamically loading multiple
>>> versions of the image based on a dynamic program-applied transform
>>> factor as is done on MacOS.
>>>
>>> Also, they tell developers to draw images to a specific size rather
>>> than using auto-sizing. That begs the question as to how they
>>> interpret a call to draw an image just using a location in the
>>> presence of various DPI factors.
>> There is an interesting doc that describes how to write DPI-aware
>> Win32 applications:
>> http://msdn.microsoft.com/en-us/library/dd464646%28v=vs.85%29.aspx
>> It is suggested to handle WM_DPICHANGED message, load the graphic
>> that has slightly greater resolution to the current DPI and use
>> StretchBlt
>> to scale down the image.
>>
>> Thanks,
>> Alexandr.
>>
>>>
>>> ...jim
>>>
>>> On 2/7/14 3:00 AM, Alexander Scherbatiy wrote:
>>>> On 1/22/2014 6:40 AM, Jim Graham wrote:
>>>>> Hi Alexander,
>>>>>
>>>>> Before we get too far down the road on this API, I think we
>>>>> understand
>>>>> the way in which MacOS processes multi-resolution images for HiDPI
>>>>> screens, but have we investigated the processes that Windows uses
>>>>> under Windows 8? My impression is that Windows 8 has included a
>>>>> number of new techniques for dealing with the high resolution
>>>>> displays
>>>>> that it will run on in the Windows tablet and mobile industries and
>>>>> that these will also come into play as 4K displays (already
>>>>> available)
>>>>> become more common on the desktop. We should make sure that what we
>>>>> come up with here can provide native compatibility with either
>>>>> platform's policies and standard practices.
>>>>>
>>>>> If you've investigated the MS policies I'd like to see a summary so
>>>>> that we can consider them as we review this API...
>>>> There is the Windows Guidelines for scaling to pixel density:
>>>> http://msdn.microsoft.com/en-us/library/windows/apps/hh465362.aspx
>>>> which says that Windows has automatic resource loading that
>>>> supports
>>>> three version of images scaling (100%, 140%, and 180%)
>>>> --------------------------------
>>>> Without scaling, as the pixel density of a display device
>>>> increases, the
>>>> physical sizes of objects on screen get smaller.
>>>> When UI would otherwise be too small to touch and when text gets too
>>>> small to read,
>>>> Windows scales the system and app UI to one of the following scaling
>>>> plateaus:
>>>>
>>>> 1.0 (100%, no scaling is applied)
>>>> 1.4 (140% scaling)
>>>> 1.8 (180% scaling)
>>>>
>>>> Windows determines which scaling plateau to use based on the physical
>>>> screen size, the screen resolution, the DPI of the screen, and form
>>>> factor.
>>>>
>>>> Use resource loading for bitmap images in the app package For bitmap
>>>> images stored
>>>> in the app package, provide a separate image for each scaling
>>>> factor(100%, 140%, and 180%),
>>>> and name your image files using the "scale" naming convention
>>>> described
>>>> below.
>>>> Windows loads the right image for the current scale automatically.
>>>> --------------------------------
>>>>
>>>> The image name convention for the various scales is:
>>>> images/logo.scale-100.png
>>>> images/logo.scale-140.png
>>>> images/logo.scale-180.png
>>>>
>>>> The 'ms-appx:///images/logo.png' uri is used to load the image
>>>> in an
>>>> application.
>>>>
>>>> If we want to support this in the same way as it is done for Mac
>>>> OS X
>>>> the WToolkit should return MultiResolution image in case if the
>>>> loaded image has .scale-* qualifiers.
>>>> The Graphics class can request an image with necessary resolution
>>>> from the MultiResolution image.
>>>>
>>>> It seems that nothing should be changed in the MultiResolution
>>>> interface in this case.
>>>>
>>>> Thanks,
>>>> Alexandr.
>>>>>
>>>>> ...jim
>>>>>
>>>>> On 1/14/14 2:54 AM, Alexander Scherbatiy wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Could you review the fix:
>>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8029339
>>>>>> webrev: http://cr.openjdk.java.net/~alexsch/8029339/webrev.00
>>>>>>
>>>>>> This is a proposal to introduce an API that allows to create a
>>>>>> custom
>>>>>> multi resolution image.
>>>>>>
>>>>>> I. It seems reasonable that the API should provide two basic
>>>>>> operations:
>>>>>>
>>>>>> 1. Get the resolution variant based on the requested image
>>>>>> width and
>>>>>> height:
>>>>>> - Image getResolutionVariant(int width, int height)
>>>>>>
>>>>>> Usually the system provides the scale factor which represents
>>>>>> the
>>>>>> number of pixels corresponding to each linear unit on the display.
>>>>>> However, it has sense to combine the scale factor and the
>>>>>> current
>>>>>> transformations to get the actual image size to be displayed.
>>>>>>
>>>>>> 2. Get all provided resolution variants:
>>>>>> - List<Image> getResolutionVariants()
>>>>>>
>>>>>> There are several uses cases:
>>>>>> - Create a new multi-resolution image based on the given
>>>>>> multi-resolution image.
>>>>>> - Pass to the native system the multi-resolution image. For
>>>>>> example,
>>>>>> a use can set to the system the custom multi-resolution cursor.
>>>>>>
>>>>>> II. There are some possible ways where the new API can be added
>>>>>>
>>>>>> 1. java.awt.Image.
>>>>>>
>>>>>> The 2 new methods can be added to the Image class. A user can
>>>>>> override
>>>>>> the getResolutionVariant() and getResolutionVariants() methods to
>>>>>> provide the resolution variants
>>>>>> or there can be default implementations of these methods if a
>>>>>> user
>>>>>> puts resolution variants
>>>>>> to the list in the sorted order.
>>>>>>
>>>>>> To check that the image has resolution variants the following
>>>>>> statement can be used: image.getResolutionVariants().size() != 1
>>>>>>
>>>>>> The disadvantage is that there is an overhead that the Image
>>>>>> class
>>>>>> should contain the List object and not all
>>>>>> images can have resolution variants.
>>>>>>
>>>>>> 2. Introduce new MultiResolutionImage interface.
>>>>>>
>>>>>> A user should extend Image class and implement the
>>>>>> MultiResolutionImage interface.
>>>>>>
>>>>>> For example:
>>>>>> ---------------------
>>>>>> public class CustomMultiResolutionImage extends BufferedImage
>>>>>> implements MultiResolutionImage {
>>>>>>
>>>>>> Image highResolutionImage;
>>>>>>
>>>>>> public CustomMultiResolutionImage(BufferedImage baseImage,
>>>>>> BufferedImage highResolutionImage) {
>>>>>> super(baseImage.getWidth(), baseImage.getHeight(),
>>>>>> baseImage.getType());
>>>>>> this.highResolutionImage = highResolutionImage;
>>>>>> Graphics g = getGraphics();
>>>>>> g.drawImage(baseImage, 0, 0, null);
>>>>>> g.dispose();
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public Image getResolutionVariant(int width, int height) {
>>>>>> return ((width <= getWidth() && height <= getHeight()))
>>>>>> ? this : highResolutionImage;
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public List<Image> getResolutionVariants() {
>>>>>> return Arrays.asList(this, highResolutionImage);
>>>>>> }
>>>>>> }
>>>>>> ---------------------
>>>>>>
>>>>>> The current fix adds the MultiResolutionImage interface and public
>>>>>> resolution variant rendering hints.
>>>>>>
>>>>>> Thanks,
>>>>>> Alexandr.
>>>>>>
>>>>
>>
More information about the awt-dev
mailing list