Truffle and mlvm

[Benoit Daloze]

Hello Charles and Thomas,

On 31 August 2014 00:04, Thomas Wuerthinger <thomas.wuerthinger at oracle.com>
wrote:

> Thanks a lot, Charlie, for this very detailed feedback! In fact, this is
> probably the most comprehensive feedback we’ve received for Truffle so far
> :).
>
> There are some valid points, some points where I’d like to add a comment,
> and some where Graal and Truffle may have been misunderstood. I’ll try to
> address them in a similarly structured form:
>
> Regarding disadvantage 1 “AST is not enough”:
> A. Yes, you need to provide specialisations for your operations for better
> performance. This makes it harder to implement a Truffle AST interpreter
> than a simple AST interpreter. It has however the advantage that it gives
> you more predictable performance for the different usages of your dynamic
> language operation. We are currently working on source code visualisations
> for Truffle ASTs so that users can see what nodes are specialised and to
> what types.
> B. Yes, it is necessary to store your local variables in a Truffle frame
> object. This object can however contain pointers to arbitrary extra data
> necessary for your guest language.
> C. We did a lot of improvements to Truffle, Graal, and also TruffleRuby
> since January. Inlining works without any problems and independent of the
> guest language, also in TruffleRuby.
>
> Regarding disadvantage 2 “Long startup and warmup times”:
> The benefit of the system is absolutely *not* lost when the compiler
> (Graal) and the Truffle guest language interpreter are AOT compiled. It
> gives you decent startup and high peak. The interpreter is immediately
> available in highly optimized machine code. Hot guest language methods are
> dynamically compiled to machine code by the precompiled Graal.
>
> Regarding disadvantage 3 “Limited concurrency”:
> There is no deeper reason why TruffleRuby is single threaded right now.
> For sure none that has to do with the fundamentals of the Truffle approach.
> We are planning to support 100% multi-threading also in TruffleRuby. One of
> the explorations we are currently doing is to support guest language level
> safepoints such that guest language developers themselves can easier deal
> with concurrency without compromising any peak performance.
>

I plan to work on concurrency in Truffle during my PhD in Linz.
Making Truffle thread-safe is a priority and some work is already done for
AST replacements.
I am also interested in supporting different concurrency primitives such as
threads and fibers for the guest languages.

Benoit

Regarding disadvantage 4 “Limited availability”:
> Yes, this is indeed a chicken and egg problem. Truffle is however not as
> closely tied to Graal as you suggest here. I believe that it is fairly
> straightforward to create a Truffle front-end for C2 (or any other compiler
> supporting deoptimization). There are only 3k LOC in Graal that are
> specific to Truffle. I think that they could be ported in a reasonable time
> frame. The Truffle interpreters themselves run on any Java system even if
> it has only very limited features - this is actually an advantage over a
> pure bytecode generation approach. They can also be AOT compiled for
> devices that do not support a full JVM and have strong footprint
> requirements. This would of course be slower execution than in a full
> fledged VM, but it would at least run correctly. I furthermore think that
> it is possible to do the Truffle partial evaluation via bytecode generation
> for backwards compatibility.
>
> Regarding disadvantage 5 "Unclear benefits for real-world applications”:
> This kind of argument can hardly be countered before a system is 100%
> finished and shipped. The term “real world” is also somewhat loosely
> defined. I would very much support the development of a JRuby benchmark
> suite that tries to reflect “real world” as close as possible.
> There is absolutely no reason to believe that a Truffle-based Ruby
> implementation would not have benefits for “real-world applications”. Or
> that it would not be able to run a large application for a long time. It is
> clear that the TruffleRuby prototype needs more completeness work both at
> the language and the library level. We are very happy with the results we
> got so far with Chris working for about a year. We are planning to increase
> the number of people working on this, and would also be grateful for any
> help we can get from the Ruby community.
>
> Regarding Graal:  Did you ever try to benchmark JRuby without Truffle with
> the latest Graal binaries available at
> http://lafo.ssw.uni-linz.ac.at/builds/? We would be looking forward to
> see the peak performance results on a couple of workloads. We are not
> speculating about Graal becoming part of a particular OpenJDK release (as
> experimental or regular option). This is the sovereign decision of the
> OpenJDK community. All we can do is to demonstrate and inform about Graal’s
> performance and stability.
>
> We recognise that there is a long road ahead. But in particular in this
> context, I would like to emphasize that we are looking for more people to
> support this effort for a new language implementation platform. I strongly
> believe that Truffle is the best currently available vehicle to make Ruby
> competitive in terms of performance with node.js. We are happy to try to
> *prove* you wrong - even happier about support of any kind along the road
> ;). I am also looking forward to continue this discussion at JavaOne (as
> part of the TruffleRuby session or elsewhere).
>
> Regards, thomas
>
> On 30 Aug 2014, at 21:21, Charles Oliver Nutter <headius at headius.com>
> wrote:
>
> > Removing all context, so it's clear this is just my opinions and
> thoughts...
> >
> > As most of you know, we've opened up our codebase and incorporated the
> > graciously-donated RubyTruffle directly into JRuby. It's available on
> > JRuby master and we are planning to ship Truffle support with JRuby
> > 9000, our next major version (due out in the next couple months).
> >
> > At the same time, we have been developing our own next-gen IR-based
> > compiler, which will run unmodified on any JVM (with or without
> > invokedynamic, though I still have to implement the "without" side).
> > Why are we doing this when Truffle shows such promise?
> >
> > I'll try to enumerate the benefits and problems of Truffle here.
> >
> > * Benefits of using Truffle
> >
> > 1. Simpler implementation.
> >
> > From day 1, the most obvious benefit of Truffle is that you just have
> > to write an AST interpreter. Anyone who has implemented a programming
> > language can do this easily. This specific benefit doesn't help us
> > implement JRuby, since we already have an AST interpreter, but it did
> > make Chris Seaton's job easier building RubyTruffle initially. This
> > also means a Truffle-based language is more approachable than one with
> > a complicated compiler pipeline of its own.
> >
> > 2. Better communication with the JIT.
> >
> > Truffle, via Graal, has potential to pass much more information on to
> > the JIT. Things like type shape, escaped references, frame access,
> > type specialization, and so on can be communicated directly, rather
> > than hoping and praying they'll be inferred by the shape of bytecodes.
> > This is probably the largest benefit; much of my time optimizing JRuby
> > has been spend trying to "trick" C2 into doing the right thing, since
> > I don't have a direct way to communicate intent.
> >
> > The peak performance numbers for Truffle-based languages have been
> > extremely impressive. If it's possible to get those numbers reasonably
> > quickly and with predictable steady-state behavior in large,
> > heterogeneous codebases, this is definitely the quickest path (on any
> > runtime) to a high-performance language implementation.
> >
> > 3. OSS and pure Java
> >
> > Truffle and Graal are just OpenJDK projects under OpenJDK licenses,
> > and anyone can build, hack, or distribute them. In addition, both
> > Truffle and Graal are 100% Java, so for the first time a plain old
> > Java developer can see (and manipulate) exactly how the JIT works
> > without getting lost in a sea of plus plus.
> >
> > * Problems with Truffle
> >
> > I want to emphasize that regardless of its warts, we love Truffle and
> > Graal and we see great potential here. But we need a dose of reality
> > once in a while, too.
> >
> > 1. AST is not enough.
> >
> > In order to make that AST fly, you can't just implement a dumb generic
> > interpreter. You need to know about (and generously annotate your AST
> > for) many advanced compiler optimization techniques:
> >
> > A. Type specialization plus guarded fallbacks: Truffle will NOT
> > specialize your code for you. You must provide every specialized path
> > in your AST nodes as well as annotating "slow path", "transfer to
> > interpreter", etc.
> >
> > B. Frame access and reification: In order to have cross-call access to
> > frames or to squash frames created for multiple inlined calls, you
> > must use Truffle's representation of a frame. This means loads/stores
> > within your AST must be done against a Truffle object, not against an
> > arbitrary object of your own creation.
> >
> > C. Method invocation and inlining: Up until fairly recently, if you
> > wanted to inline methods you had to essentially build your own call
> > site logic, profiling, deopt paths within your Truffle AST. When I did
> > a little hacking on RubyTruffle around OSS time (December/January) it
> > did *no* inlining of Ruby-to-Ruby calls. I hacked in inlining using
> > existing classes and managed to get it to work, but I was doing all
> > the plumbing myself. I know this has improved in the Truffle codebase
> > since then, but I have my concerns about production readiness when the
> > inlining call site parts of Truffle were just recently added and are
> > still in flux.
> >
> > And there's plenty of other cases. Building a basic language for
> > Truffle is pretty easy (I did a micro-language in about two hours at
> > JVMLS last year), but building a high-performance language for Truffle
> > still takes a fair investment of effort and working knowledge of
> > dynamic compiler optimizations.
> >
> > 2. Long startup and warmup times.
> >
> > As Thomas pointed out in the other thread, because Truffle and Graal
> > are normally run as plain Java libraries, they can actually aggravate
> > startup time issues. Now, not only would all of JRuby have to warm up,
> > but the eventual native code JIT has to warm up too. This is not
> > surprising, really. It is possible to mitigate this by doing some form
> > of AOT against Graal, but for every case I have seen the Truffle/Graal
> > approach makes startup time much, much worse compared to just running
> > atop JVM.
> >
> > Warmup time is also worsened significantly.
> >
> > The AST you create for Truffle must be heavily mutated while running
> > in order to produce a specialized version of that AST. This must
> > happen before the AST is eventually fed into Graal, which means you
> > have a self-modifying interpreter spinning AST objects like mad while
> > executing the early phases of your application. Compare to a dumb
> > interpreter as in JRuby's old AST, where interpreting the AST produces
> > no additional objects other than those necessary for execution of the
> > code.
> >
> > The Truffle approach itself adds overhead too. Until optimized, the
> > fully-reified frame objects, specialization markup (which triggers AST
> > rewriting), deoptimization guards, and so on are all done manually
> > against heap-level data structures. This is in addition to the
> > JVM-level overhead of executing an AST (native frame-per-node, boxing
> > and type-widening, poor inlining profile).
> >
> > Some amount of AOT *might* be applicable here, but the benefit of
> > Truffle and Graal is lost in the AOT case if we're not getting
> > real-world profile information. The Substrate VM has ben brought up to
> > aid startup and warmup too...but that direction produces a
> > closed-world executable based on optimizing all code up front...not
> > exactly what we're looking for in a general-purpose language runtime.
> >
> > 3. Limited concurrency
> >
> > The RubyTruffle runtime currently has to execute code under the
> > watchful eye of a global lock. Yes, you read that right...RubyTruffle
> > is single-threaded right now.
> >
> > I would like to know if there's a deeper reason for this, but the
> > obvious shallow reason is that you can't have multiple threads
> > executing at the same time if they're making thread-unsafe mutations
> > to the executing code. This is similar to the major stumbling block
> > for e.g. Pypy, which rewrites currently-executing assembly
> > instructions at deopt/reopt safe points.
> >
> > I believe once the code has transitioned to native, you can execute
> > that safely across threads...but this is opaque to your Truffle-based
> > language, and it's unclear how you'd manage re-acquiring some sort of
> > lock when transferring back to the interpreter.
> >
> > The fact that concurrency has so far been hand-waved (or so it seems
> > to me from the outside) scares the living hell out of me, especially
> > when there's talk about rolling this stuff into Java 9.
> >
> > Obviously some of this could be mitigated with an immutable AST
> > structure or other thread-friendly tree-transformation algorithm, but
> > making the Truffle AST thread-safe may also make it even more
> > object-heavy during interpretation, aggravating startup time further.
> >
> > 4. Limited availability
> >
> > This is the chicken-and-egg issue. Truffle is just a library, so we
> > can ignore that for the moment (given any JVM, you can run a Truffle
> > language).
> >
> > Graal is required for Truffle to perform well at all. The Truffle
> > interpreter is without a doubt the slowest interpreter we've ever had
> > for JRuby, and that's saying something (there could be startup/warmup
> > effects in play here too). In order for us to go 100% Truffle, we'd
> > need a Graal VM. That limits us to either pre-release or hand-made
> > builds of Graal/OpenJDK. Even if Graal somehow did get into Java 9,
> > we'd still have legions of users on 8, 7, ... even 6 in some cases,
> > though we're probably leaving them behind with JRuby 9000. Ignoring
> > other platforms (non-OpenJDK, Android) and assuming Graal in Java 9,
> > I'd conservatively estimate JRuby could still not go 100% Truffle
> > until 2017 or later.
> >
> > And it gets worse. Graal will probably never exist on other JVMs.
> > Graal will probably never exist in an Android VM. Graal may not even
> > be available in other non-Oracle OpenJDK derivatives for a very long
> > time. We have users on dozens of different platform/JVM combinations,
> > so there's really no practical way for us to abandon our JVM bytecode
> > runtimes in the near future.
> >
> > Now of course if Graal became essential to users, it would be
> > available in more places. We recognize the potential of Truffle and
> > Graal, which is why we've been thrilled to work with Oracle on a
> > RubyTruffle that's part of JRuby. We also recognize that the
> > Truffle/Graal approach has some very compelling features for our
> > users, and that our users may often be comfortable running custom
> > JVMs. We're allowing all flowers to bloom and our users will pick the
> > ones that work for them.
> >
> > 5. Unclear benefits for real-world applications
> >
> > There have been many published microbenchmarks for Truffle-based
> > languages, but very few benchmarks of real-world applications
> > performing significantly better than custom-made VMs (JS versus V8).
> > There have been practically no studies of a Truffle-based language
> > running a large application for a long period of time...and by long I
> > mean server-scale.
> >
> > Chris Seaton has pushed this forward recently for Ruby, getting
> > general-purpose, numeric-heavy libraries to run and optimize very well
> > (a png library and a psd library). Going deeper requires having more
> > of the language's standard libraries to be available, and I believe
> > this is where Chris has spent much of his time (RubyTruffle currently
> > requires mostly-custom versions of JRuby's core classes...versions
> > that Truffle can recognize, specialize, and escape-analyze away).
> >
> > * Conclusion
> >
> > I again want to emphasize that we think Truffle and Graal are really
> > awesome technology. I spent years with my nose smooshed against the
> > glass, watching the Pypy guys add optimizations I wanted and make good
> > on their promise of "just implement an interpreter...we'll do the
> > rest". Finally we have what I wanted: a Pypy for JVM (in Truffle) and
> > an LLVM for JVM (in Graal). These are exciting times indeed.
> >
> > But reality steps in. There's a long road ahead.
> >
> > I think we need to separate the questions about Truffle from questions
> > about Graal. Truffle is ultimately just a library that uses Graal.
> >
> > Graal is promising JIT technology. Graal is simpler than C2 and may be
> > able to match or beat its performance. Graal provides a better way to
> > communicate intent to the JIT. These facts are not in question.
> >
> > However, Graal is not (other than when used as the JVM's JIT) a JVM.
> > Targeting Graal directly acts against the promise of a standard,
> > platform-and-VM-agnostic bytecode -- and that's the promise that
> > brought most of us here. Graal is not yet ready to replace C2, which
> > would mean adding to the size and complexity of Java 9. And Graal is
> > almost completely untested in large production settings.
> >
> > I personally would love to see Graal get into a Java release soon as
> > an experimental feature, but Java 9 seems ambitious but any standard.
> > It *might* be possible/reasonable to include Graal as experimental in
> > 9. Java 10 is certainly feasible for experimental, and may be feasible
> > for product. But even if Graal got into mainstream OpenJDK and Java,
> > there's a very long adoption tail ahead.
> >
> > I'd like to hear more from folks on the Graal and Truffle teams. Prove
> > me wrong :-)
> >
> > - Charlie
> > _______________________________________________
> > mlvm-dev mailing list
> > mlvm-dev at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20141002/dfe3660f/attachment-0001.html>