java.net.SocketException: Cannot allocate memory

[Brian Call]

To give a little bit of context... We're developing a new wrist-worn medical monitoring device that sends real-time vitals information such as heart rate, blood oxygen levels, blood pressure, and even full 12 lead ECG waveforms. The device sends it's data to the base station using a TCP connection, and then the base station retransmits the data back to the monitoring stations using UDP. We chose multicast as the solution there because it greatly reduced network utilization, since the outgoing packets could be replicated at the router level and did not require ack(s) for every packet. We can't use multicast from the device itself because we need to guarantee delivery of packets to the base station for storage and indexing. 

We've divided up the data stream into several disparate multicast groups and use a different port for each device. This results in about twenty+ separate groups across all devices and we're hoping to scale up to hundreds of devices by providing a unique port for each device instance. The reason for such granularity is also for network bandwidth conservation. We don't want to transmit all twelve channels of ECG waveforms if the user is only viewing channel two, etc. So we have the viewing station subscribe only the to groups it is interesting in receiving. 

This scheme has worked very well for Windows/Linux based systems and I've had as many as sixty-four simultaneous device running without any issues. It's only on OS X that this memory allocation problem has happened. Granted, we did have to bump up the max_igmp_memberships threshold on linux, but we've been unable to find an analogous setting for OS X. 

Is it possible that group memberships are not being deallocated when the app process exits? Often I've seen the first startup of the app work with the error occurring more frequently at subsequent restarts. Sometimes it fails when joining as little as a single group. 

I've been perusing the XNU source code hoping to find the source of the error message but have not had much luck yet. I'm fixing to try the earlier suggestion of using dtruss to try and determine what error is being returned to the Net.joinOrDrop() native code. 

Is there any way to get access to the jvm source for MacOS? That might yield some interesting insight as well. 

Thanks again everyone for your help!

-Brian 



On Jul 5, 2013, at 12:19 PM, Alan Bateman <Alan.Bateman at oracle.com> wrote:

> On 05/07/2013 18:20, Brian Call wrote:
>> Hi Guys,
>> 
>> I'm seeing an ugly issue crop up on JDK7 for MacOS that I'm at a total loss to diagnose or even begin to fix. At random times (and sometimes every time) I'm seeing this exception:
>> 
>> Caused by: java.net.SocketException: Cannot allocate memory
>> 	at sun.nio.ch.Net.joinOrDrop4(Native Method)
>> 	at sun.nio.ch.Net.join4(Net.java:421)
>> 	at sun.nio.ch.DatagramChannelImpl.innerJoin(DatagramChannelImpl.java:848)
>> 	at sun.nio.ch.DatagramChannelImpl.join(DatagramChannelImpl.java:866)
>> 	at org.jboss.netty.channel.socket.nio.NioDatagramChannel.joinGroup(NioDatagramChannel.java:158)
>> 	at org.jboss.netty.channel.socket.nio.NioDatagramChannel.joinGroup(NioDatagramChannel.java:135)
>> 	at com.soterawireless.h1s.groups.NioGroupSubscriber.joinGroup(NioGroupSubscriber.java:203)
>> 	... 5 more
>> 
>> This same codes works great on both Linux and Windows but fails intermittently on Mac OS 10.8.x. Google is letting me down and stackoverflow isn't being very helpful either...
>> 
>> Does anyone have any insight on what might be causing this join group call to fail in such a manner? I could very easily write a simple test driver to demonstrate the problem so please let me know if that would be helpful.
>> 
> I haven't seen this but I'm curious as to how many multicast groups are involved and whether you are dropping membership when done (not leaking I mean). I ask because there is typically a limit to how much filtering can be done on the NIC and once it goes past that limit then it has to be switched to promiscuous mode. I don't know the Mach / OS X kernel to know what is actually going here but mentioning limits in case there is a resource issue.
> 
> -Alan

Blessings,

Brian Call
brian.call at soterawireless.com

PRIVILEGED AND CONFIDENTIAL COMMUNICATION: This electronic transmission, and any documents attached hereto, may contain confidential and/or legally privileged information.  The information is intended only for use by the recipient named above. If you have received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited.