Timer death

[Pawel Veselov]

Greetings,

After debugging an issue in one of my projects, I've realized that the
problem was that a timer task simply died because VM was out of
memory.
The fact that I catch any Throwable around the code that threw the OOM
error didn't particularly help. The error was logged, but the timer
thread still died.
Well, may be it didn't die at this point, I do see the exception
handling code executed (it logged something else from the catch{}
block as well), and I really hope (don't believe) that a thread would
die when normal exception handling was executed and the exception
wasn't re-thrown upstream.

But, I'm not here to ramble about particulates about my specific
situations. When I started thinking about how do I detect this
somehow, I also realized that it's not normally possible to determine
if any specific timer is alive or not. I see two ways to determine
that:

a) Schedule a single empty task on a timer. If you get back an
IllegalStateException, then the timer thread is gone (the code
suggests if the thread leaves the main loop for any reason, any
scheduling will cause IllegalStateException).
b) When the timer is created, schedule a single task, and determine
the timer thread during the task execution (current thread), and then
check whether the thread is running or not.

Now, both of those are hacky, "a" schedules a task for no reason, and
"b" will stop working if timer implementation becomes capable of
scheduling tasks on multiple threads for whatever load-balancing
reasons.

I would suggest adding a method to either retrieve a timer thread
(sort of bad, because this will make the timer implementation to
always use a single thread), or simply a method that tells whether the
timer is alive or not (whether this would now translate into whether
the thread is alive or not), so the program can attempt to recover (or
take any other action, like shooting itself) if the timer stopped
working.

My overall issue with this (I know, kind of late to realize this now),
is that it's really hard to recover from OOMs, or even detect that
they are happening, and all of the sudden your application threads
start to simply disappear, and you need yet another thread that would
go check they are alive and take some action (and who is to say that
thread won't die as well, and you won't notice that either).

java.lang.OutOfMemoryError: Java heap space
	at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
	at com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:382)
	at com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2604)
	at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:487)
	at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2582)
	at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1758)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2172)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2690)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
	at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1465)
        <classified>
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)


Thanks,
  Pawel.