Producing a backend for network based display

[Roman Kennke]

Hi Curt,

Sorry for the late reply.

> I'm interested in providing a backend to enable Java programs to be
> run on one machine, but interacted with via a user running a client
> program somewhere across the network.  There are already native
> applications that do this -- usually for the entire desktop, but
> sometimes for individual applications.  I'm talking about things like
> VNC, RDP, and X.  Can anyone provide any pointers to classes,
> interfaces, methods, documentation, etc... that would be relevant?  Is
> such a thing feasible?

Sure, why not. Sound like a cool thing.

First of all, Caciocavallo seems like a good choice, because it reduces
the burden to implement the AWT widgets significantly. Basically, you
start out by implementing a GraphicsEnvironment (is easy enough when you
subclass SunGraphicsEnvironment). Then you need a Toolkit
implementation, I strongly recommend to subclass CacioToolkit. Then you
fill out the required methods as you need. Most importantly, for the
CacioToolkit, you need a PlatformWindow implementation, which leads to
your second question:

> I know that there are many different levels where such a display
> remoting facility could be introduced.

With Cacio you have two options:
1. You implement the PlatformWindow directly. This is particulary useful
if you want to use native windows of the target machine. But then you
need some kind of protocol that transmits information like 'open a
window', 'set bounds of window', etc. Is a bit more work than the 2nd
approach, but probably gives a better user experience.
2. Use the so-called 'managed windows' by only implementing the
PlatformScreen interface. All the window handling and stuff is done by
Cacio in the form of the ManagedWindow class (which in turn implements
the PlatformWindow mentioned above). This approach is most useful when
your target doesn't have windows at all (embedded platforms,...) or when
you don't want to write protocol for that, or as a starting point for
prototyping.

Either way, what you need is the lowest level of the implementation,
which is a Java2D pipeline. Generally speaking, you need a Graphics2D
implementation. You could subclass Graphics2D directly, and have lots of
work, or you reuse SunGraphics2D and implement a Java2D pipeline by
providing a SurfaceData implementation. This in turn can be implemented
in a very primitive way, or very complex. Usually I start out with a
fairly primitive implementation, and then optimize what needs to be
optimized:
1. The most basic thing you need is access to pixel data. Not sure how
to implement this over network, but basically, the Java end needs some
kind of GetPixmap() that pulls a rectangular region of the display over
network. Then the Java side sets pixels in this thing, and pushes back
the resulting pixmap using some kind of PutPixmap() or so. This is more
or less how the X11 based pipeline works over network. I strongly
recommend reading SurfaceData.h in the OpenJDK sources for more
information.
2. When #1 works reliably, you usually want some optimizations:
- Support client (display) side accelerated images. Basically, it's the
same as #1, but for offscreen pixmaps on the client (note that I'm
swapping the meaning of client and server wrt how X11 sees things) side.
- Support blitting. You usually want some protocol that says 'copy this
area of this surface - offscreen or onscreen- to this location on that
surface - the same or a different one. You avoid much much network
traffic, because otherwise you need to pull BOTH surfaces, make the copy
on the Java side, and push stuff back over the network.
- The above should already give you reasonable results. But for futher
optimizations you might want to implement protocol for the most commonly
used primitives like drawing/filling lines, rectangles and some other
stuff.

I hope this gives you a start, please come back when you need more
information.

Cheers, Roman