Simplifying the poisoning mechanism used in HotSpot

[Julian Waters]

Argh, some extra code I put in there isn't showing up, here's the new
link: https://godbolt.org/z/jPbWcbn7E


On Mon, Aug 18, 2025 at 2:43 AM Julian Waters <tanksherman27 at gmail.com> wrote:
>
> Hi all,
>
> Currently, I'm working on refining the poisoning mechanism used in
> HotSpot for forbidding calls to certain methods. The current system
> works, but it is a little unwieldy, unfortunately. I recall many
> issues being involved in its creation, from requiring headers to be
> included in the exact right order to not cause clang to explode during
> compilation, to complex workarounds and much code being needed to
> dodge compiler warnings. Arguably the biggest downside of this system
> is the fact that you need to select the exact correct macro to use for
> forbidding any methods in the codebase (There are several), and to
> select correctly you need to check the system headers, and choosing
> the wrong one will cause errors that aren't obvious on how to fix to
> surface, which could potentially be confusing.
>
> An older approach I tried before the current system was set up was
> defining a method with an identical signature in an inline namespace
> called forbidden, and then deleting that method or adding the
> deprecated attribute.  The main problems with that approach is that
> there is no way to force C++ to choose the inline namespace over the
> Standard Library counterpart. This causes an ambiguity error rather
> than the desired "You are not allowed to use this method" error, but
> more importantly you couldn't control it, meaning there was no way to
> permit forbidden methods. The only way you can force C++ to
> definitively use the one in the namespace is by hooking each call with
> a function macro.
>
> With a simple function macro redirection, you could reroute calls to,
> say, malloc, into a permission manager namespace. In that namespace
> would be another different malloc, as described above. This definition
> would be annotated to deliberately cause compile failures if called.
> This yields the desired effect of not allowing any forbidden methods
> into the codebase, but does come with one drawback, namely that any
> declarations (Not just calls!) of a method with the same name in any
> of our code (Think of the methods in the os class for instance) will
> be overwritten as well. There's just 1 way to bypass this: Save the
> macro to a macro stack (All our compilers support this), undefine the
> macro redirection, declare or define the method in our own code, then
> pop the redirection off the macro stack to reactivate the poisoning.
> The macro deactivation could be stuffed into a header for convenience,
> if needed, and then the header could be included at sites where it's
> needed. A small system with this underlying principle is already used
> in awt.dll for Windows to forbid calls to malloc, calloc and free, and
> other codebases do use similar methods to forbid calls to certain
> methods.
>
> Question is, does this somewhat more compact system solve more
> problems than it creates? It does eliminate several of the drawbacks
> of the current system, but comes with problems of its own, as
> mentioned above. I'd like to get feedback on this before deciding if
> this should be scrapped or improved upon further.
>
> I'm aware I'm not the best with words, so if anyone is confused as to
> what I'm talking about, the latest prototype I have is here, so you
> can more easily visualize what I mean: https://godbolt.org/z/bGohMx4fc
>
> Thanks for your time and have a great day/week ahead!
>
> best regards,
> Julian