Simplifying the poisoning mechanism used in HotSpot

[Julian Waters]

Hi all,

Currently, I'm working on refining the poisoning mechanism used in
HotSpot for forbidding calls to certain methods. The current system
works, but it is a little unwieldy, unfortunately. I recall many
issues being involved in its creation, from requiring headers to be
included in the exact right order to not cause clang to explode during
compilation, to complex workarounds and much code being needed to
dodge compiler warnings. Arguably the biggest downside of this system
is the fact that you need to select the exact correct macro to use for
forbidding any methods in the codebase (There are several), and to
select correctly you need to check the system headers, and choosing
the wrong one will cause errors that aren't obvious on how to fix to
surface, which could potentially be confusing.

An older approach I tried before the current system was set up was
defining a method with an identical signature in an inline namespace
called forbidden, and then deleting that method or adding the
deprecated attribute.  The main problems with that approach is that
there is no way to force C++ to choose the inline namespace over the
Standard Library counterpart. This causes an ambiguity error rather
than the desired "You are not allowed to use this method" error, but
more importantly you couldn't control it, meaning there was no way to
permit forbidden methods. The only way you can force C++ to
definitively use the one in the namespace is by hooking each call with
a function macro.

With a simple function macro redirection, you could reroute calls to,
say, malloc, into a permission manager namespace. In that namespace
would be another different malloc, as described above. This definition
would be annotated to deliberately cause compile failures if called.
This yields the desired effect of not allowing any forbidden methods
into the codebase, but does come with one drawback, namely that any
declarations (Not just calls!) of a method with the same name in any
of our code (Think of the methods in the os class for instance) will
be overwritten as well. There's just 1 way to bypass this: Save the
macro to a macro stack (All our compilers support this), undefine the
macro redirection, declare or define the method in our own code, then
pop the redirection off the macro stack to reactivate the poisoning.
The macro deactivation could be stuffed into a header for convenience,
if needed, and then the header could be included at sites where it's
needed. A small system with this underlying principle is already used
in awt.dll for Windows to forbid calls to malloc, calloc and free, and
other codebases do use similar methods to forbid calls to certain
methods.

Question is, does this somewhat more compact system solve more
problems than it creates? It does eliminate several of the drawbacks
of the current system, but comes with problems of its own, as
mentioned above. I'd like to get feedback on this before deciding if
this should be scrapped or improved upon further.

I'm aware I'm not the best with words, so if anyone is confused as to
what I'm talking about, the latest prototype I have is here, so you
can more easily visualize what I mean: https://godbolt.org/z/bGohMx4fc

Thanks for your time and have a great day/week ahead!

best regards,
Julian