From bryan.atsatt at oracle.com Fri Jul 13 11:46:58 2007 From: bryan.atsatt at oracle.com (Bryan Atsatt) Date: Fri, 13 Jul 2007 11:46:58 -0700 Subject: Module system notification mechanism In-Reply-To: <4684425F.4060308@sun.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> Message-ID: <4697C8A2.40407@oracle.com> (Sorry, somehow this message got marked read and I only just now saw it.) Comments inline. Stanley M. Ho wrote: > Bryan Atsatt wrote: >> Stanley, I assume you are proposing a general mechanism in which any >> code can register listeners? If so, then I have to agree with Glyn that >> this is not sufficient to implement a module "activator". > > The mechanism I have in mind can be registered only by code that have > the appropriate security permissions, and untrusted code in the sandbox > won't be able to listen to the notification. > > I am still a bit puzzled by why the module's state needs to be coupled > directly to the result from the module's activator because this looks to > me more like a solution to a problem than a requirement. For that model > to work, I think there are couple underlying assumptions, but they might > not be valid in our case: > > 1. If a module's activator is executed before the module is fully > initialized, it can safely register something into some service lookup > mechanism in the system. > 2. The service lookup mechanism will make sure the registered thing from > that module is not usable by other modules unless that module becomes > fully initialized. > 3. If the module's activator fails to execute when the module is being > initialized, the module system knows how to work with the service lookup > mechanism to undo all the unregisterations automatically and correctly. So do you expect that java.util.ServiceLoader will do this? > In this case, the module system probably needs to be highly integrated > with the service lookup mechanism, and that mechanism should be the only > one used by all the modules' activators. However, in practice, multiple > service lookup mechanisms exist and can be used in the Java platform, > and I don't think we should expect the module system will know how to > work with these arbitrary mechanisms to provide the suggested module's > state semantic in conjunction with the module's activator. Also, from > our previous discussions, I think our consensus is to avoid pushing the > service mechanism down to the module layer if possible. I agree that we shouldn't try to push a *service* mechanism into the module layer. But a generic *activator* certainly seems appropriate. Some modules will need to perform initialization prior to use (e.g. process a config file, check some environmental constraints, populate a JNDI context, etc, etc). Successful initialization for such a module is then a pre-condition for use, and any failure should be treated exactly as seriously as a resolution failure. Yes, this *could* be done using the notification mechanism, but that strikes me as a rather hackish approach. This use case is not limited to "container" environments, so, IMO, should be made part of the spec. And it seems like a rather small addition: 1. A simple interface that the module can implement, e.g.: interface ModuleActivator { public void start(Module module) throws Exception; public void release(Module module) throws Exception; } 2. An annotation to declare the activator impl name. 3. Module system instantiates and invokes activator (at end of prep/validation or a new step). An exception here would put the module into ERROR state. Exactly what such an activator does is mostly irrelevant, though it clearly cannot consume the thread; the usual precautions here should suffice. // Bryan From Stanley.Ho at sun.com Mon Jul 16 12:33:14 2007 From: Stanley.Ho at sun.com (Stanley M. Ho) Date: Mon, 16 Jul 2007 12:33:14 -0700 Subject: Module system notification mechanism In-Reply-To: <4697C8A2.40407@oracle.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> Message-ID: <469BC7FA.5080406@sun.com> Hi Bryan, Bryan Atsatt wrote: >> 1. If a module's activator is executed before the module is fully >> initialized, it can safely register something into some service lookup >> mechanism in the system. >> 2. The service lookup mechanism will make sure the registered thing from >> that module is not usable by other modules unless that module becomes >> fully initialized. >> 3. If the module's activator fails to execute when the module is being >> initialized, the module system knows how to work with the service lookup >> mechanism to undo all the unregisterations automatically and correctly. > > So do you expect that java.util.ServiceLoader will do this? No. Actually, the way java.util.ServiceLoader looks up service-providers is through the context classloader at runtime when a service is being requested. Unlike other service lookup mechanisms, it is not necessary to register service-providers with the java.util.ServiceLoader API at all, so the module's activator won't be useful in this case. The scenarios where using module's activator for registration might be applicable are for those modules that want to register with other service lookup mechanisms that are not based on java.util.ServiceLoader. > I agree that we shouldn't try to push a *service* mechanism into the > module layer. > > But a generic *activator* certainly seems appropriate. Some modules will > need to perform initialization prior to use (e.g. process a config file, > check some environmental constraints, populate a JNDI context, etc, > etc). Successful initialization for such a module is then a > pre-condition for use, and any failure should be treated exactly as > seriously as a resolution failure. The solution does not have to tie to the module state. I think what we simply want is to support the use case that a module instance should not be handed out unless certain initialization code is executed successfully. As long as the API for obtaining such module instance supports this semantic, it's orthogonal to how the module state changes internally. > Yes, this *could* be done using the notification mechanism, but that > strikes me as a rather hackish approach. This use case is not limited to > "container" environments, so, IMO, should be made part of the spec. I'm not aware of use cases which are not in the "container" environments. Could you provide some examples? > And it seems like a rather small addition: > > 1. A simple interface that the module can implement, e.g.: > > interface ModuleActivator { > public void start(Module module) throws Exception; > public void release(Module module) throws Exception; > } > > 2. An annotation to declare the activator impl name. > > 3. Module system instantiates and invokes activator (at end of > prep/validation or a new step). An exception here would put the module > into ERROR state. > > Exactly what such an activator does is mostly irrelevant, though it > clearly cannot consume the thread; the usual precautions here should > suffice. I also think that the API impact is small. However, because the module system and the service lookup mechanisms are not integrated tightly together, a not-yet-fully-initialized Module instance might be leaked out to other service lookup callers after it is registered by the ModuleActivator through the service lookup mechanism but before it is fully initialized. Because of this, I don't think there is obvious benefit to provide built-in support for ModuleActivator since the same can be achieved using the notification mechanism, which we'll support anyway. Another extreme approach is to design a very generic service lookup mechanism that is tightly integrated with the module system, and also requires all other service lookup mechanisms to be built on top of this generic one (if these mechanisms are ever used in the module's activator), so they will all provide the appropriate semantic w.r.t. the module system and no uninitialized module instance would be leaked out accidentally. That said, I don't think this is what we want to do. ;-) - Stanley From bryan.atsatt at oracle.com Mon Jul 16 13:41:44 2007 From: bryan.atsatt at oracle.com (Bryan Atsatt) Date: Mon, 16 Jul 2007 13:41:44 -0700 Subject: Module system notification mechanism In-Reply-To: <469BC7FA.5080406@sun.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> Message-ID: <469BD808.90207@oracle.com> Stanley M. Ho wrote: > Hi Bryan, > > Bryan Atsatt wrote: >>> 1. If a module's activator is executed before the module is fully >>> initialized, it can safely register something into some service lookup >>> mechanism in the system. >>> 2. The service lookup mechanism will make sure the registered thing from >>> that module is not usable by other modules unless that module becomes >>> fully initialized. >>> 3. If the module's activator fails to execute when the module is being >>> initialized, the module system knows how to work with the service lookup >>> mechanism to undo all the unregisterations automatically and correctly. >> >> So do you expect that java.util.ServiceLoader will do this? > > No. Actually, the way java.util.ServiceLoader looks up service-providers > is through the context classloader at runtime when a service is being > requested. Unlike other service lookup mechanisms, it is not necessary > to register service-providers with the java.util.ServiceLoader API at > all, so the module's activator won't be useful in this case. > > The scenarios where using module's activator for registration might be > applicable are for those modules that want to register with other > service lookup mechanisms that are not based on java.util.ServiceLoader. > >> I agree that we shouldn't try to push a *service* mechanism into the >> module layer. >> >> But a generic *activator* certainly seems appropriate. Some modules will >> need to perform initialization prior to use (e.g. process a config file, >> check some environmental constraints, populate a JNDI context, etc, >> etc). Successful initialization for such a module is then a >> pre-condition for use, and any failure should be treated exactly as >> seriously as a resolution failure. > > The solution does not have to tie to the module state. I think what we > simply want is to support the use case that a module instance should not > be handed out unless certain initialization code is executed > successfully. As long as the API for obtaining such module instance > supports this semantic, it's orthogonal to how the module state changes > internally. > >> Yes, this *could* be done using the notification mechanism, but that >> strikes me as a rather hackish approach. This use case is not limited to >> "container" environments, so, IMO, should be made part of the spec. > > I'm not aware of use cases which are not in the "container" > environments. Could you provide some examples? I don't have a specific example in mind, but just consider libraries targeted at SE that, before use, must: - Validate a license key, or - Establish a network connection, or - Read a config file, or - Validate the environment (e.g. java version, presence of some Class, etc,), or - Fork threads. While these could all be done lazily, or with explicit calls by the client, I've seen many cases of the "X must be initialized first" problem. In my experience, many of these have been in the EE environment, but only incidentally: some app bundles lib X, which itself has no EE dependencies but must be initialized. I've seen many EE apps that are configured with a servlet *only* so that they can get an init call at startup! > >> And it seems like a rather small addition: >> >> 1. A simple interface that the module can implement, e.g.: >> >> interface ModuleActivator { >> public void start(Module module) throws Exception; >> public void release(Module module) throws Exception; >> } >> >> 2. An annotation to declare the activator impl name. >> >> 3. Module system instantiates and invokes activator (at end of >> prep/validation or a new step). An exception here would put the module >> into ERROR state. >> >> Exactly what such an activator does is mostly irrelevant, though it >> clearly cannot consume the thread; the usual precautions here should >> suffice. > > I also think that the API impact is small. However, because the module > system and the service lookup mechanisms are not integrated tightly > together, a not-yet-fully-initialized Module instance might be leaked > out to other service lookup callers after it is registered by the > ModuleActivator through the service lookup mechanism but before it is > fully initialized. Because of this, I don't think there is obvious > benefit to provide built-in support for ModuleActivator since the same > can be achieved using the notification mechanism, which we'll support > anyway. You are coupling the ideas of activation and services; I'm explicitly trying to de-couple them here. Yes, service registration could be done by an activator, but I'm focused on the more general problem of module *initialization*. Maybe this would be clearer if we changed the interface name: interface ModuleInitializer { public void start(Module module) throws Exception; public void release(Module module) throws Exception; } Seems to me that we can easily eliminate the leakage problem by: 1. Ensuring that initialization occurs before the module moves from RESOLVED to READY state, and 2. Initialization failure moves the module to ERROR state, and 3. Module lifecycle events are only fired when the module enters either READY or ERROR states (and STOPPING/STOPPED, if we add those). If you intend to expose the states leading up to READY/ERROR as events, then yes, leakage is a potential problem. We could get around this by passing only the ModuleDefinition on these events, not the Module itself. > > Another extreme approach is to design a very generic service lookup > mechanism that is tightly integrated with the module system, and also > requires all other service lookup mechanisms to be built on top of this > generic one (if these mechanisms are ever used in the module's > activator), so they will all provide the appropriate semantic w.r.t. the > module system and no uninitialized module instance would be leaked out > accidentally. That said, I don't think this is what we want to do. ;-) Agreed. > > - Stanley > From Stanley.Ho at sun.com Mon Jul 16 18:08:39 2007 From: Stanley.Ho at sun.com (Stanley M. Ho) Date: Mon, 16 Jul 2007 18:08:39 -0700 Subject: Module system notification mechanism In-Reply-To: <469BD808.90207@oracle.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> Message-ID: <469C1697.2090403@sun.com> Hi Bryan, Bryan Atsatt wrote: > ... > I don't have a specific example in mind, but just consider libraries > targeted at SE that, before use, must: > > - Validate a license key, or > - Establish a network connection, or > - Read a config file, or > - Validate the environment (e.g. java version, presence of some Class, > etc,), or > - Fork threads. > > While these could all be done lazily, or with explicit calls by the > client, I've seen many cases of the "X must be initialized first" > problem. In my experience, many of these have been in the EE > environment, but only incidentally: some app bundles lib X, which itself > has no EE dependencies but must be initialized. I've seen many EE apps > that are configured with a servlet *only* so that they can get an init > call at startup! If we take the service registration out of the picture, the module's initialization could probably take place in the custom import policy directly. Maybe in some situations the custom import policy would require more security permissions for the initialization to succeed, but we could work this out. Do you have examples that using custom import policy is not sufficient for performing initialization? The initialization issue around EE apps is a separate issue, and it could be addressed if EE defines its own activation mechanism on top of the module system's notification mechanism. > You are coupling the ideas of activation and services; I'm explicitly > trying to de-couple them here. Yes, service registration could be done > by an activator, but I'm focused on the more general problem of module > *initialization*. My impression around the activation idea was to support the service registration, and this is the primary use case I see how activation is used in many existing systems. If we want to use this mechanism for initialization, then I would like to better understand why using custom import policy is not sufficient. ;-) > Maybe this would be clearer if we changed the interface name: > > interface ModuleInitializer { > public void start(Module module) throws Exception; > public void release(Module module) throws Exception; > } > > Seems to me that we can easily eliminate the leakage problem by: > > 1. Ensuring that initialization occurs before the module moves from > RESOLVED to READY state, and > > 2. Initialization failure moves the module to ERROR state, and > > 3. Module lifecycle events are only fired when the module enters either > READY or ERROR states (and STOPPING/STOPPED, if we add those). > > If you intend to expose the states leading up to READY/ERROR as events, > then yes, leakage is a potential problem. We could get around this by > passing only the ModuleDefinition on these events, not the Module itself. I agreed some modules might want to cleanup themselves when there is no reference to their Module instances. On the other hand, the suggested release() method is invoked after the reference of the Module instance has been released from the module system, but other modules could still hold references to that instance and continue accessing classes through the module's classloader. In other words, it would be inappropriate to trigger clean up when this release() method is invoked. I believe what you really want is to clean up when the Module instance is about to be GCed; however, this is already possible if the module hooks into the garbage collector to trigger the cleanup code after the Module instance becomes unreachable. It is unclear to me what purpose the release() method really serves and if supporting this method is useful. Unless of course, the clean up is performed when the module's classloader is disabled at the same time so nobody else could use the module, but we have already discussed that disabling the classloader is a very special use case that should only happen in the EE container environments. If this is something only needed by some containers, then it is better to let the containers build their own activation mechanisms on top of the module system's notification mechanism, rather than pushing it down into the module system level. Do you have specific examples where the release() method would be useful outside the container environments? - Stanley From glyn_normington at UK.IBM.COM Tue Jul 17 01:20:31 2007 From: glyn_normington at UK.IBM.COM (Glyn Normington) Date: Tue, 17 Jul 2007 09:20:31 +0100 Subject: Module system notification mechanism In-Reply-To: <469C1697.2090403@sun.com> Message-ID: Hi Stanley You wrote on 17/07/2007 02:08:39 AM: > Do you have examples that using custom import > policy is not sufficient for performing initialization? Take the common example where a custom import policy is being avoided because of its disadvantages ([1]), but a module initialiser is still needed. Glyn [1] http://underlap.blogspot.com/2007/06/representations-of-module-dependencies.html Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/jsr277-eg-observer/attachments/20070717/49fc9711/attachment.html From bryan.atsatt at oracle.com Tue Jul 17 10:28:49 2007 From: bryan.atsatt at oracle.com (Bryan Atsatt) Date: Tue, 17 Jul 2007 10:28:49 -0700 Subject: Module system notification mechanism In-Reply-To: <469C1697.2090403@sun.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> Message-ID: <469CFC51.3020109@oracle.com> Custom import policies cannot leverage any other modules; this would be a severe limitation for an initialization mechanism. Stanley M. Ho wrote: > Hi Bryan, > > Bryan Atsatt wrote: >> ... >> I don't have a specific example in mind, but just consider libraries >> targeted at SE that, before use, must: >> >> - Validate a license key, or >> - Establish a network connection, or >> - Read a config file, or >> - Validate the environment (e.g. java version, presence of some Class, >> etc,), or >> - Fork threads. >> >> While these could all be done lazily, or with explicit calls by the >> client, I've seen many cases of the "X must be initialized first" >> problem. In my experience, many of these have been in the EE >> environment, but only incidentally: some app bundles lib X, which itself >> has no EE dependencies but must be initialized. I've seen many EE apps >> that are configured with a servlet *only* so that they can get an init >> call at startup! > > If we take the service registration out of the picture, the module's > initialization could probably take place in the custom import policy > directly. Maybe in some situations the custom import policy would > require more security permissions for the initialization to succeed, but > we could work this out. Do you have examples that using custom import > policy is not sufficient for performing initialization? > > The initialization issue around EE apps is a separate issue, and it > could be addressed if EE defines its own activation mechanism on top of > the module system's notification mechanism. > >> You are coupling the ideas of activation and services; I'm explicitly >> trying to de-couple them here. Yes, service registration could be done >> by an activator, but I'm focused on the more general problem of module >> *initialization*. > > My impression around the activation idea was to support the service > registration, and this is the primary use case I see how activation is > used in many existing systems. If we want to use this mechanism for > initialization, then I would like to better understand why using custom > import policy is not sufficient. ;-) > >> Maybe this would be clearer if we changed the interface name: >> >> interface ModuleInitializer { >> public void start(Module module) throws Exception; >> public void release(Module module) throws Exception; >> } >> >> Seems to me that we can easily eliminate the leakage problem by: >> >> 1. Ensuring that initialization occurs before the module moves from >> RESOLVED to READY state, and >> >> 2. Initialization failure moves the module to ERROR state, and >> >> 3. Module lifecycle events are only fired when the module enters either >> READY or ERROR states (and STOPPING/STOPPED, if we add those). >> >> If you intend to expose the states leading up to READY/ERROR as events, >> then yes, leakage is a potential problem. We could get around this by >> passing only the ModuleDefinition on these events, not the Module itself. > > I agreed some modules might want to cleanup themselves when there is no > reference to their Module instances. On the other hand, the suggested > release() method is invoked after the reference of the Module instance > has been released from the module system, but other modules could still > hold references to that instance and continue accessing classes through > the module's classloader. In other words, it would be inappropriate to > trigger clean up when this release() method is invoked. > > I believe what you really want is to clean up when the Module instance > is about to be GCed; however, this is already possible if the module > hooks into the garbage collector to trigger the cleanup code after the > Module instance becomes unreachable. It is unclear to me what purpose > the release() method really serves and if supporting this method is > useful. Unless of course, the clean up is performed when the module's > classloader is disabled at the same time so nobody else could use the > module, but we have already discussed that disabling the classloader is > a very special use case that should only happen in the EE container > environments. If this is something only needed by some containers, then > it is better to let the containers build their own activation mechanisms > on top of the module system's notification mechanism, rather than > pushing it down into the module system level. > > Do you have specific examples where the release() method would be useful > outside the container environments? > > - Stanley > From Stanley.Ho at sun.com Tue Jul 17 14:48:34 2007 From: Stanley.Ho at sun.com (Stanley M. Ho) Date: Tue, 17 Jul 2007 14:48:34 -0700 Subject: Module system notification mechanism In-Reply-To: <469CFC51.3020109@oracle.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> <469CFC51.3020109@oracle.com> Message-ID: <469D3932.4060000@sun.com> Hi Bryan, Bryan Atsatt wrote: > Custom import policies cannot leverage any other modules; this would be > a severe limitation for an initialization mechanism. There are good reasons why there is such restriction. When the module is being initialized, its imported modules might also be in the process of initialization. If one module accesses another module while both are in the middle of initialization, the result could be problematic. Also, suppose two modules have cyclic dependencies and each has its own initializer, what happens when an initializer is called in one module and that initializer wants to access the other module, while the other module's initializer wants to do the same? We could either prevent this from happening by saying the initializer could only access code in the standard SE modules and the code within its own module, or we could say that the initializer could access the module itself and all its imported modules - but developers need to be aware that there is a possibility that the imported module is only half-initialized when it is called by the importer's initializer. The latter could be very error-prone, and I'm not sure how useful it is to provide it to the developers. On the other hand, if we restrict which modules the initializer could access, then the initializer is not much different from the custom import policy - it's still a piece of code that is executed during module initialization, and exception would cause the module initialization to fail. In fact, the custom import policy was called ModuleInitializer in the original JSR 277 strawman, and the intention is to allow a module to do everything it needs to initialize itself properly. - Stanley From bryan.atsatt at oracle.com Tue Jul 17 23:46:26 2007 From: bryan.atsatt at oracle.com (Bryan Atsatt) Date: Tue, 17 Jul 2007 23:46:26 -0700 Subject: Module system notification mechanism In-Reply-To: <469D3932.4060000@sun.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> <469CFC51.3020109@oracle.com> <469D3932.4060000@sun.com> Message-ID: <469DB742.4000204@oracle.com> Um, I'm not arguing about the restriction in custom import policies, I certainly understand why that is present. I'm saying that an initializer *should* be able to rely on imported modules, so we need a separate mechanism. And that separate mechanism would kick in *after* resolution/validation had completed. And yes, there is potentially a pathological case even here, if two initializers have cyclic dependencies (not the Class kind ;^), but... Seems to me that this has to be discovered and eliminated in development; this is a variation on the "order of static initialization" problem, and one that we can't solve here. Are you just suggesting that we add start()/refresh() type methods to ImportPolicy, rather than invent a new class/annotation? And that we might then want to rename this class? For example, rename ImportPolicy to ModuleInitializer, and add to it: public abstract class ModuleInitializer { public abstract List getImports(...); public abstract void initialize(Module) throws Exception; public abstract void release(Module); } If so, I'm OK with that approach as long as the class loading restrictions during getImports() are made clear and the module state at initialize() is clearly specified. // Bryan Stanley M. Ho wrote: > Hi Bryan, > > Bryan Atsatt wrote: >> Custom import policies cannot leverage any other modules; this would be >> a severe limitation for an initialization mechanism. > > There are good reasons why there is such restriction. When the module is > being initialized, its imported modules might also be in the process of > initialization. If one module accesses another module while both are in > the middle of initialization, the result could be problematic. Also, > suppose two modules have cyclic dependencies and each has its own > initializer, what happens when an initializer is called in one module > and that initializer wants to access the other module, while the other > module's initializer wants to do the same? > > We could either prevent this from happening by saying the initializer > could only access code in the standard SE modules and the code within > its own module, or we could say that the initializer could access the > module itself and all its imported modules - but developers need to be > aware that there is a possibility that the imported module is only > half-initialized when it is called by the importer's initializer. The > latter could be very error-prone, and I'm not sure how useful it is to > provide it to the developers. > > On the other hand, if we restrict which modules the initializer could > access, then the initializer is not much different from the custom > import policy - it's still a piece of code that is executed during > module initialization, and exception would cause the module > initialization to fail. In fact, the custom import policy was called > ModuleInitializer in the original JSR 277 strawman, and the intention is > to allow a module to do everything it needs to initialize itself properly. > > - Stanley > From Stanley.Ho at sun.com Thu Jul 19 19:34:55 2007 From: Stanley.Ho at sun.com (Stanley M. Ho) Date: Thu, 19 Jul 2007 19:34:55 -0700 Subject: Module system notification mechanism In-Reply-To: <469DB742.4000204@oracle.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> <469CFC51.3020109@oracle.com> <469D3932.4060000@sun.com> <469DB742.4000204@oracle.com> Message-ID: <46A01F4F.7040300@sun.com> Hi Bryan, Bryan Atsatt wrote: > Um, I'm not arguing about the restriction in custom import policies, I > certainly understand why that is present. > > I'm saying that an initializer *should* be able to rely on imported > modules, so we need a separate mechanism. And that separate mechanism > would kick in *after* resolution/validation had completed. > > And yes, there is potentially a pathological case even here, if two > initializers have cyclic dependencies (not the Class kind ;^), but... > Seems to me that this has to be discovered and eliminated in > development; this is a variation on the "order of static initialization" > problem, and one that we can't solve here. I think we agreed that in the presence of circular dependencies a module cannot rely on its initializer being called before other code in the module is called. The problem is that a module may not know if it is part of a cyclic dependency chain because that depends on its imports, their imports, etc. I don't think we can assume that this is a pathological case that never occurs in the real world. If a module cannot rely on its initializer to access other modules safely or in the right order, then it is unclear to me if such support would be very useful in practice. Instead, a module could rely on the existing custom import policy or alternatively on other measures, for example explicit checks in the code or infrastructure provided by the container. > Are you just suggesting that we add start()/refresh() type methods to > ImportPolicy, rather than invent a new class/annotation? And that we > might then want to rename this class? For example, rename ImportPolicy > to ModuleInitializer, and add to it: > > public abstract class ModuleInitializer { > public abstract List getImports(...); > public abstract void initialize(Module) throws Exception; > public abstract void release(Module); > } > > If so, I'm OK with that approach as long as the class loading > restrictions during getImports() are made clear and the module state at > initialize() is clearly specified. I'm fine with changing the name of the ImportPolicy class into ModuleInitializer if it makes things more obvious to developers. I think what you suggested is not unreasonable, but I want to make sure this is something we really need to support, especially outside the container environments. Specifically, I'm not yet convinced we should provide an initialize() method that allows access to the imported modules (excluding the standard SE modules) because of the potential problems. Could you provide use cases for this outside a container environment? It is also unclear to me if providing the release() method would be useful outside a container environment as well. As I hinted before, you probably want to do the clean up in the finalizer or something similar when nobody uses the module instance anymore, rather than when the module instance is released from the module system while other existing users could still access it. - Stanley From bryan.atsatt at oracle.com Fri Jul 20 12:11:10 2007 From: bryan.atsatt at oracle.com (Bryan Atsatt) Date: Fri, 20 Jul 2007 12:11:10 -0700 Subject: Module system notification mechanism In-Reply-To: <46A01F4F.7040300@sun.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> <469CFC51.3020109@oracle.com> <469D3932.4060000@sun.com> <469DB742.4000204@oracle.com> <46A01F4F.7040300@sun.com> Message-ID: <46A108CE.7020606@oracle.com> Stanley M. Ho wrote: > Hi Bryan, > > Bryan Atsatt wrote: >> Um, I'm not arguing about the restriction in custom import policies, I >> certainly understand why that is present. >> >> I'm saying that an initializer *should* be able to rely on imported >> modules, so we need a separate mechanism. And that separate mechanism >> would kick in *after* resolution/validation had completed. >> >> And yes, there is potentially a pathological case even here, if two >> initializers have cyclic dependencies (not the Class kind ;^), but... >> Seems to me that this has to be discovered and eliminated in >> development; this is a variation on the "order of static initialization" >> problem, and one that we can't solve here. > > I think we agreed that in the presence of circular dependencies a module > cannot rely on its initializer being called before other code in the > module is called. The problem is that a module may not know if it is > part of a cyclic dependency chain because that depends on its imports, > their imports, etc. I don't think we can assume that this is a > pathological case that never occurs in the real world. Sure, it *can* happen, just like execution order between classes can cause problems. But we still have class initializers in the language; I don't see this any differently. In the unusual case where a problem does occur, it must be fixed by refactoring the code in some fashion. So what? How can this be an argument that class initializers shouldn't exist? Sorry, Stanley, but I just don't buy this argument. > It is also unclear to me if providing the release() method would be > useful outside a container environment as well. As I hinted before, you > probably want to do the clean up in the finalizer or something similar > when nobody uses the module instance anymore, rather than when the > module instance is released from the module system while other existing > users could still access it. ARGH! Stanley, we've been going around in circles on this for years now :^). Yes, the lifecycle problem is made visible more often in container environments, but it is not limited to them: if this spec is going to enable a module to be released during the life of a process, then we've just pulled the problem directly into SE, container or no container. Your argument here is that since we cannot know whether a module's classes are in use *after* it has been released/uninstalled, informing the module that it has been released is pointless, since it cannot do anything meaningful. This is utter nonsense (sorry)! If a module requires some kind of initialization (whether we make it easy or not), it almost certainly needs to cleanup at some point. In your view, the way to accomplish such cleanup is during GC. In the real world, this is rarely sufficient: a single class leak (often caused by JRE classes!) will cause it to fail. And, often worse, the timing issue rears its ugly head: another module/class comes up that needs to acquire the same resources. This is then a race condition over which you have little control. GC is not the answer to all resource management. Proper resource management requires a well-defined lifecycle: threads, server sockets, client sockets, files, file locks, native library resources, external (server) resources, etc. all need to be explicitly managed. We need to at least give modules a well-defined time to release resources (and a module should be free to reject calls into it after those resources have been released). Either we should avoid this problem entirely by eliminating the ability to release modules, OR we should provide a means to solve the problem. Clearly, I am strongly in favor of the latter. And the solution should be symmetrical... It seems irresponsible in the extreme for this EG to waste the opportunity to introduce a standardized lifecycle, with both init and release calls to the module itself. // Bryan BTW: In the ModuleInitializer I suggested, I would provide no-op implementations for all the methods. If you don't care about getImports(), for example, you shouldn't have to do anything in your subclass. From gordon.hirsch at SAS.COM Fri Jul 20 12:35:59 2007 From: gordon.hirsch at SAS.COM (Gordon Hirsch) Date: Fri, 20 Jul 2007 15:35:59 -0400 Subject: Module system notification mechanism In-Reply-To: <46A108CE.7020606@oracle.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> <469CFC51.3020109@oracle.com> <469D3932.4060000@sun.com> <469DB742.4000204@oracle.com> <46A01F4F.7040300@sun.com> <46A108CE.7020606@oracle.com> Message-ID: <46A10E9F.3080904@sas.com> I've been meaning to chime in to agree with Bryan. I think the analogy to is apt. It's clear to me that init()/release() are fundamentally useful. OSGi provides a useful precedent here. As an OSGi user I often implement the BundleActivator start() and/or stop() methods to manage resources that are needed module-wide, or to install/uninstall various listeners, etc. These facilities can be abused, but without them, module implementors are forced to do all initialization lazily which is usually more complex (especially to make it thread-safe). So I'm strongly in favor of the suggested lifecycle hooks. Bryan Atsatt wrote: > Stanley M. Ho wrote: >> Hi Bryan, >> >> Bryan Atsatt wrote: >>> Um, I'm not arguing about the restriction in custom import policies, I >>> certainly understand why that is present. >>> >>> I'm saying that an initializer *should* be able to rely on imported >>> modules, so we need a separate mechanism. And that separate mechanism >>> would kick in *after* resolution/validation had completed. >>> >>> And yes, there is potentially a pathological case even here, if two >>> initializers have cyclic dependencies (not the Class kind ;^), but... >>> Seems to me that this has to be discovered and eliminated in >>> development; this is a variation on the "order of static initialization" >>> problem, and one that we can't solve here. >> >> I think we agreed that in the presence of circular dependencies a module >> cannot rely on its initializer being called before other code in the >> module is called. The problem is that a module may not know if it is >> part of a cyclic dependency chain because that depends on its imports, >> their imports, etc. I don't think we can assume that this is a >> pathological case that never occurs in the real world. > > Sure, it *can* happen, just like execution order between > classes can cause problems. > > But we still have class initializers in the language; I don't see this > any differently. In the unusual case where a problem does occur, it must > be fixed by refactoring the code in some fashion. So what? How can this > be an argument that class initializers shouldn't exist? > > Sorry, Stanley, but I just don't buy this argument. > >> It is also unclear to me if providing the release() method would be >> useful outside a container environment as well. As I hinted before, you >> probably want to do the clean up in the finalizer or something similar >> when nobody uses the module instance anymore, rather than when the >> module instance is released from the module system while other existing >> users could still access it. > > ARGH! Stanley, we've been going around in circles on this for years now > :^). Yes, the lifecycle problem is made visible more often in container > environments, but it is not limited to them: if this spec is going to > enable a module to be released during the life of a process, then we've > just pulled the problem directly into SE, container or no container. > > Your argument here is that since we cannot know whether a module's > classes are in use *after* it has been released/uninstalled, informing > the module that it has been released is pointless, since it cannot do > anything meaningful. > > This is utter nonsense (sorry)! If a module requires some kind of > initialization (whether we make it easy or not), it almost certainly > needs to cleanup at some point. In your view, the way to accomplish such > cleanup is during GC. In the real world, this is rarely sufficient: a > single class leak (often caused by JRE classes!) will cause it to fail. > And, often worse, the timing issue rears its ugly head: another > module/class comes up that needs to acquire the same resources. This is > then a race condition over which you have little control. GC is not the > answer to all resource management. > > Proper resource management requires a well-defined lifecycle: threads, > server sockets, client sockets, files, file locks, native library > resources, external (server) resources, etc. all need to be explicitly > managed. > > We need to at least give modules a well-defined time to release > resources (and a module should be free to reject calls into it after > those resources have been released). > > Either we should avoid this problem entirely by eliminating the ability > to release modules, OR we should provide a means to solve the problem. > Clearly, I am strongly in favor of the latter. And the solution should > be symmetrical... > > It seems irresponsible in the extreme for this EG to waste the > opportunity to introduce a standardized lifecycle, with both init and > release calls to the module itself. > > // Bryan > > BTW: In the ModuleInitializer I suggested, I would provide no-op > implementations for all the methods. If you don't care about > getImports(), for example, you shouldn't have to do anything in your > subclass. From andyp at bea.com Mon Jul 23 02:39:02 2007 From: andyp at bea.com (Andy Piper) Date: Mon, 23 Jul 2007 10:39:02 +0100 Subject: Module system notification mechanism In-Reply-To: <46A10E9F.3080904@sas.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> <469CFC51.3020109@oracle.com> <469D3932.4060000@sun.com> <469DB742.4000204@oracle.com> <46A01F4F.7040300@sun.com> <46A108CE.7020606@oracle.com> <46A10E9F.3080904@sas.com> Message-ID: <6.2.5.6.2.20070723103845.02bd3bd8@bea.com> At 20:35 20/07/2007, Gordon Hirsch wrote: >So I'm strongly in favor of the suggested lifecycle hooks. me too andy Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it. From Stanley.Ho at sun.com Wed Jul 25 03:47:10 2007 From: Stanley.Ho at sun.com (Stanley M. Ho) Date: Wed, 25 Jul 2007 03:47:10 -0700 Subject: Module system notification mechanism In-Reply-To: <46A108CE.7020606@oracle.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> <469CFC51.3020109@oracle.com> <469D3932.4060000@sun.com> <469DB742.4000204@oracle.com> <46A01F4F.7040300@sun.com> <46A108CE.7020606@oracle.com> Message-ID: <46A72A2E.5090406@sun.com> Hi, Let's briefly recap. The arguments for the initializer I heard so far are (correct me if I miss anything): 1. It's not custom import policy! 2. It provides developers a way to perform initialization in a module after validation succeeds but before the module instance is ready. 3. It provides developers a way to be informed when a module is released, so the module could potentially deny future requests, if it chooses to. As I mentioned already, I think what has been suggested so far is not unreasonable. What I was arguing is not whether such support is useful at all by any means, but more specifically whether such support is useful outside the container environment. What I am concerned with in particular are a few issues: a. whether initialize() can access other imported modules As we briefly mentioned, this could be problematic in some situations, e.g. cyclic dependencies. Because of that, I don't think the module system should/could guarantee that an initializer will always be able to access its imported modules. To be more specific, which imported modules are accessible by the initializer will likely be determined by the states of the module instances in the module system and will also depend on the actual module system implementation (and things get much more complicated if there are other module systems in the picture). In some situations, the set of accessible imported modules from the initializer could be none. I suggest that we should not make such guarantee in the module system at all. b. cleanup resources in release() when the module instance is released As we also discussed, this could be problematic if a module cleans up itself while its classloader is still accessible by other modules. It is possible that some developers might want to do implement something like (3) above in their modules, but I don't think this is what most modules outside the containers should do. I suggest we should make the potential issues obvious to developers in the documentation, and also discourage developers to use it as a general cleanup mechanism, unless they know what they are doing of course. If the EG still thinks that the initializer support is important and also if we have consensus on the suggestions I made above, I will incorporate this into the next revision of the specification. - Stanley From bryan.atsatt at oracle.com Thu Jul 26 12:29:58 2007 From: bryan.atsatt at oracle.com (Bryan Atsatt) Date: Thu, 26 Jul 2007 12:29:58 -0700 Subject: Module system notification mechanism In-Reply-To: <46A72A2E.5090406@sun.com> References: <4684078C.6030907@oracle.com> <4684425F.4060308@sun.com> <4697C8A2.40407@oracle.com> <469BC7FA.5080406@sun.com> <469BD808.90207@oracle.com> <469C1697.2090403@sun.com> <469CFC51.3020109@oracle.com> <469D3932.4060000@sun.com> <469DB742.4000204@oracle.com> <46A01F4F.7040300@sun.com> <46A108CE.7020606@oracle.com> <46A72A2E.5090406@sun.com> Message-ID: <46A8F636.20609@oracle.com> Stanley M. Ho wrote: > Hi, > > Let's briefly recap. The arguments for the initializer I heard so far > are (correct me if I miss anything): > > 1. It's not custom import policy! Yes, as far as the class name goes. But it can still serve that role if the developer chooses to implement getImports(). > 2. It provides developers a way to perform initialization in a module > after validation succeeds but before the module instance is ready. > 3. It provides developers a way to be informed when a module is > released, so the module could potentially deny future requests, if it > chooses to. Right. > As I mentioned already, I think what has been suggested so far is not > unreasonable. What I was arguing is not whether such support is useful > at all by any means, but more specifically whether such support is > useful outside the container environment. What I am concerned with in > particular are a few issues: > > a. whether initialize() can access other imported modules > > As we briefly mentioned, this could be problematic in some situations, > e.g. cyclic dependencies. Because of that, I don't think the module > system should/could guarantee that an initializer will always be able to > access its imported modules. To be more specific, which imported modules > are accessible by the initializer will likely be determined by the > states of the module instances in the module system and will also depend > on the actual module system implementation (and things get much more > complicated if there are other module systems in the picture). In some > situations, the set of accessible imported modules from the initializer > could be none. I suggest that we should not make such guarantee in the > module system at all. Sorry, but this makes no sense to me. When a module transitions from RESOLVED to READY, it is, by definition, prepared to execute any of its code, including calls into dependent modules. And this is true even in the case of cyclic dependencies; otherwise, we could not hand out a module for use at this point either! So all we have to do is call initialize() as the last step before transitioning to READY. If it fails, we go to ERROR, else go to READY. > b. cleanup resources in release() when the module instance is released > > As we also discussed, this could be problematic if a module cleans up > itself while its classloader is still accessible by other modules. It is > possible that some developers might want to do implement something like > (3) above in their modules, but I don't think this is what most modules > outside the containers should do. I suggest we should make the potential > issues obvious to developers in the documentation, and also discourage > developers to use it as a general cleanup mechanism, unless they know > what they are doing of course. We definitely need to document this with some care. But, again, the only real potential issue here is with cycles, just as it is during initialization. In the absence of cycles, the release() calls will occur in the correct order, right? That is, if A --depends--> B --depends--> C And C is released, the importers will be released first: A, then B, and finally C. As long as we guarantee this ordering, the potential issues will be reduced to those due to cycles. And, again, just as with order-of-static-initialization, developers must be aware of the pitfalls of cycles and code around them. > > > If the EG still thinks that the initializer support is important and > also if we have consensus on the suggestions I made above, I will > incorporate this into the next revision of the specification. > > - Stanley >