Direct dependency on service provider modules? was: Service provider module dependency resolving webrev

[Paul Sandoz]

Hi,

On Jun 21, 2012, at 10:05 AM, Jaroslav Tulach wrote:

> Dne St 20. června 2012 17:42:44, Paul Sandoz napsal(a):
>> 
>>>> - service provider modules dependencies are resolved after the explicit
>>>> dependencies have been successfully resolved.
>>> 
>>> This part can be made easy with complete repository[1].
>> 
>> Would you be able to elaborate in layman's terms?
> 
> Paul, please excuse me for not expressing myself clearly:

I think it is more the case of me struggling to understand.

 
> I wanted to say that 
> "resolving explicit dependencies" can be made easy with complete 
> repository[1]. It is the easy/easier part.
> 
> Resolving later the service provider modules will then be the hard part.
> 

OK.


>> I think i might grok it: we essentially treat each service provider module
>> dependency as a direct (or top-level) dependency.
> 
> Do you say that "require service xyz" declaration would mean to enable all 
> modules which "provide service xyz with something"? E.g. like saying that a 
> module that "require service xyz" has a direct dependency on all modules that 
> "provide service xyz".

No, each service provider module dependency becomes an equivalent of a root dependency, like an application module. Such dependencies are processed *after* the application module dependencies have been resolved. Previously they were processed as direct dependencies of the service consumer module.

BUT in either either case such dependencies are *optional*. I think that might help clear up some of the discussion.

I am attaching tracing output from the resolver. Notice the "pushing service provider" and that all "resolving SERVICE PROVIDER" are at the end and they are all optional even though the "mapp" module declares "requires service ...";


> 
> That is radical approach. It can work if the set of modules is properly 
> selected at beginning (e.g. NetBeans IDE version 7.2 without external modules, 
> or Ubuntu 12.04 release) - and we know all modules that "provide service xyz" 
> can in fact be linked under (almost) all situations. 
> 
> But if you apply the same rule to a collection of modules with history, 
> provided by various parties (e.g. Maven Central), than it is very limiting - 
> there will always be at least one module that "provides service xyz" and it 
> cannot be enabled. Then the whole resolution should collapse, right?
> 
> The approach with "direct dependency on each service provider" certainly 
> eliminates the NP-complete problem. But it suffers from the fact that it is 
> not very usable. Unless the set of modules is carefully pre-selected by humans 
> (e.g. humans solve the NP-completeness first) it almost certainly leads to 
> "can't resolve" results all over.
> 
> An interesting property of "treating each service provider module dependency 
> as direct" is that one can break the system by installing new module! E.g. 
> your module which "requires service xyz" can be resolved. By installing new 
> module which "provide service xyz" and is unresolvable (for any reasons) and 
> the original module won't resolve any more either. How do you like this 
> (mis)behavior?
> 
>>>> - if service provider modules or their dependencies are service consumer
>>>> 
>>>> modules then the correspond service provide modules are resolved after
>>>> the
>>>> explicit dependences and so on.
>>> 
>>> In this step you have to select one of the modules that "provides a
>>> service" which has been requested by some module in the first step. This
>>> is the part when the algorithm turns into 3-SAT solver.
>> 
>> I would very like to do a mind meld with you on this!
>> 
>> It's interesting in that, with the current Resolver implementation, when you
>> push service provider module dependency resolution to the end then there is
>> no longer any backtracking above those dependencies, since they are
>> optional.
> 
> Are they really optional?

They are synthesized as such.


> What is optional on "requires service xyz"? But...
> 
> Heuréka! I think I know how to overcome the NP-complete problem with services. 
> We need to make sure we make the dependencies optional. E.g. there are no NP-
> complete problems with "requires service optional xyz". Thus, if we allow only 
> optional services, we simplify everything a lot. 
> 
> Of course, some "requires service" are not that optional. So the question for 
> me is: how to make the dependency system usable and still pretend all services 
> are in fact optional?
> 

I think we may be on the same page!

The Jigsaw resolver does a post processing step on successful resolution:

    private boolean run()
        throws ConfigurationException, IOException
    {
        Choice ch = null;
        for (ModuleIdQuery midq : rootQueries) {
            ViewDependence dep = new ViewDependence(EnumSet.noneOf(Modifier.class),
                                                    midq);
            ch = new Choice(null, dep,  ch);
        }
        boolean resolved = resolve(0, ch);
        if (resolved)
            ensureServicesPresent();  // <--- check mandatory requires service constraints
        return resolved;
    }

That step iterates through for all resolved modules and checks for each mandatory "requires service ..." that there is at least one "provides service ...":

    private void ensureServicesPresent() throws ConfigurationException {
        Set<String> serviceTypes = new HashSet<>();
        for (ModuleInfo mi: modules) {
            for (ModuleView view: mi.views()) {
                Map<String,Set<String>> services = view.services();
                serviceTypes.addAll(services.keySet());
            }
        }
        for (ModuleInfo mi: modules) {
            for (ServiceDependence sd: mi.requiresServices()) {
                if (!sd.modifiers().contains((Modifier.OPTIONAL))) {
                    String sn = sd.service();
                    if (!serviceTypes.contains(sn)) {
                        fail("No implementations of service %s, required by %s",
                             sn, mi.id());
                    }

                }
            }
        }
    }


>> Which i think means one can break things down into ordered sub-phases
>> resolving the dependencies of each service provider module (which IIUC is
>> still NP-Complete as you point out). That may make it easier to report
>> service provider module dependency resolution issues to the developer.
>>> No surprise, things may get unclear and complicated (which is how I
>>> understood your further discussion with Alan).
>> 
>> Yes, i am trying to run the resolver in my head. It's concerns me if
>> developers have to do the same.
> 
> ;-)
> 
> Well, people can do this on a small system (e.g. distribution). They have 
> their ideal vision and see the whole picture. They just need a language (e.g. 
> requires/provides/service/optional) to describe such vision. 
> 
> The problem arises when few such small systems are combined together (e.g. 
> Maven central). No human is able to see the whole picture then, still we want 
> computers to operate on such mess effectively.
> 

Agreed.

Paul.


    [exec] | Configuring [mapp@=1.0] using library build/mlib
    [exec] |   resolving ROOT requires mapp@=1.0
    [exec] |   - trying mapp at 1.0
    [exec] |   - pushing service provider module dependency stringer.StringTransformer -> requires optional mhasher
    [exec] |   - pushing service provider module dependency stringer.StringTransformer -> requires optional mrotter
    [exec] |   -- resolving mapp at 1.0 requires mstringer
    [exec] |   --- trying mstringer at 1.0
    [exec] |   ---- resolving mstringer at 1.0 requires synthesized java.base@>=8
    [exec] |   ----- trying java.base
    [exec] |   ----- pushing service provider module dependency java.nio.charset.spi.CharsetProvider -> requires optional sun.charsets
    [exec] |   ----- pushing service provider module dependency java.nio.file.spi.FileSystemProvider -> requires optional jdk.zipfs
    [exec] |   ----- pushing service provider module dependency sun.net.spi.nameservice.NameServiceDescriptor -> requires optional jdk.jndi
    [exec] |   ------ resolving jdk.base at 8-ea requires local optional jdk.desktop.internal@=8-ea
    [exec] |   ------- trying jdk.desktop.internal at 8-ea
    [exec] |   ------- pushing service provider module dependency javax.print.PrintServiceLookup -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.print.StreamPrintServiceFactory -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.sound.midi.spi.MidiDeviceProvider -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.sound.midi.spi.MidiFileReader -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.sound.midi.spi.MidiFileWriter -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.sound.midi.spi.SoundbankReader -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.sound.sampled.spi.AudioFileReader -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.sound.sampled.spi.AudioFileWriter -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.sound.sampled.spi.FormatConversionProvider -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency javax.sound.sampled.spi.MixerProvider -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency sun.java2d.cmm.PCMM -> requires optional jdk.desktop
    [exec] |   ------- pushing service provider module dependency sun.java2d.pipe.RenderingEngine -> requires optional jdk.desktop
    [exec] |   -------- resolving jdk.desktop at 8-ea requires local jdk.base.internal@=8-ea
    [exec] |   --------- resolving jdk.desktop at 8-ea requires jdk.jaxp@=8-ea
    [exec] |   ---------- trying jdk.jaxp at 8-ea
    [exec] |   ----------- resolving jdk.jaxp at 8-ea requires jdk.base@=8-ea
    [exec] |   ------------ resolving jdk.jaxp at 8-ea requires synthesized java.base@>=8
    [exec] |   ------------- resolving jdk.desktop at 8-ea requires jdk.logging@=8-ea
    [exec] |   -------------- trying jdk.logging at 8-ea
    [exec] |   --------------- resolving jdk.logging at 8-ea requires jdk.base.internal@=8-ea
    [exec] |   ---------------- resolving jdk.logging at 8-ea requires synthesized java.base@>=8
    [exec] |   ----------------- resolving jdk.desktop at 8-ea requires jdk.prefs@=8-ea
    [exec] |   ------------------ trying jdk.prefs at 8-ea
    [exec] |   ------------------- resolving jdk.prefs at 8-ea requires local jdk.base.internal@=8-ea
    [exec] |   -------------------- resolving jdk.prefs at 8-ea requires jdk.jaxp@=8-ea
    [exec] |   --------------------- resolving jdk.prefs at 8-ea requires synthesized java.base@>=8
    [exec] |   ---------------------- resolving jdk.desktop at 8-ea requires sun.charsets.internal@=8-ea
    [exec] |   ----------------------- trying sun.charsets.internal at 8-ea
    [exec] |   ------------------------ resolving sun.charsets at 8-ea requires local jdk.base.internal@=8-ea
    [exec] |   ------------------------- resolving sun.charsets at 8-ea requires synthesized java.base@>=8
    [exec] |   -------------------------- resolving jdk.desktop at 8-ea requires synthesized java.base@>=8
    [exec] |   --------------------------- resolving jdk.base at 8-ea requires optional jdk.jaxp.internal@=8-ea
    [exec] |   ---------------------------- resolving jdk.base at 8-ea requires local optional jdk.tls.internal@=8-ea
    [exec] |   ----------------------------- trying jdk.tls.internal at 8-ea
    [exec] |   ------------------------------ resolving jdk.tls at 8-ea requires local jdk.base.internal@=8-ea
    [exec] |   ------------------------------- resolving jdk.tls at 8-ea requires synthesized java.base@>=8
    [exec] |   -------------------------------- resolving jdk.base at 8-ea requires local optional sun.localedata@=8-ea
    [exec] |   --------------------------------- trying sun.localedata at 8-ea
    [exec] |   ---------------------------------- resolving sun.localedata at 8-ea requires local jdk.base.internal@=8-ea
    [exec] |   ----------------------------------- resolving sun.localedata at 8-ea requires synthesized java.base@>=8
    [exec] |   ------------------------------------ resolving jdk.base at 8-ea requires local optional sun.resources@=8-ea
    [exec] |   ------------------------------------- trying sun.resources at 8-ea
    [exec] |   -------------------------------------- resolving sun.resources at 8-ea requires local jdk.base.internal@=8-ea
    [exec] |   --------------------------------------- resolving sun.resources at 8-ea requires synthesized java.base@>=8
    [exec] |   ---------------------------------------- resolving mapp at 1.0 requires synthesized java.base@>=8
    [exec] |   ----------------------------------------- resolving SERVICE PROVIDER requires optional mhasher
    [exec] |   ------------------------------------------ trying mhasher at 1.0
    [exec] |   ------------------------------------------- resolving mhasher at 1.0 requires mstringer
    [exec] |   -------------------------------------------- resolving mhasher at 1.0 requires synthesized java.base@>=8
    [exec] |   --------------------------------------------- resolving SERVICE PROVIDER requires optional mrotter
    [exec] |   ---------------------------------------------- trying mrotter at 1.0
    [exec] |   ----------------------------------------------- resolving mrotter at 1.0 requires mstringer
    [exec] |   ------------------------------------------------ resolving mrotter at 1.0 requires synthesized java.base@>=8
    [exec] |   ------------------------------------------------- resolving SERVICE PROVIDER requires optional sun.charsets
    [exec] |   -------------------------------------------------- resolving SERVICE PROVIDER requires optional jdk.zipfs
    [exec] |   --------------------------------------------------- trying jdk.zipfs at 8-ea
    [exec] |   ---------------------------------------------------- resolving jdk.zipfs at 8-ea requires jdk.base@=8-ea
    [exec] |   ----------------------------------------------------- resolving jdk.zipfs at 8-ea requires synthesized java.base@>=8
    [exec] |   ------------------------------------------------------ resolving SERVICE PROVIDER requires optional jdk.jndi
    [exec] |   ------------------------------------------------------- trying jdk.jndi at 8-ea
    [exec] |   -------------------------------------------------------- resolving jdk.jndi at 8-ea requires local jdk.auth.internal@=8-ea
    [exec] |   --------------------------------------------------------- trying jdk.auth.internal at 8-ea
    [exec] |   ---------------------------------------------------------- resolving jdk.auth at 8-ea requires local jdk.base.internal@=8-ea
    [exec] |   ----------------------------------------------------------- resolving jdk.auth at 8-ea requires jdk.logging@=8-ea
    [exec] |   ------------------------------------------------------------ resolving jdk.auth at 8-ea requires synthesized java.base@>=8
    [exec] |   ------------------------------------------------------------- resolving jdk.jndi at 8-ea requires local jdk.base.internal@=8-ea
    [exec] |   -------------------------------------------------------------- resolving jdk.jndi at 8-ea requires optional jdk.desktop@=8-ea
    [exec] |   --------------------------------------------------------------- resolving jdk.jndi at 8-ea requires jdk.rmi@=8-ea
    [exec] |   ---------------------------------------------------------------- trying jdk.rmi at 8-ea
    [exec] |   ----------------------------------------------------------------- resolving jdk.rmi at 8-ea requires jdk.base.internal@=8-ea
    [exec] |   ------------------------------------------------------------------ resolving jdk.rmi at 8-ea requires jdk.logging@=8-ea
    [exec] |   ------------------------------------------------------------------- resolving jdk.rmi at 8-ea requires jdk.tls@=8-ea
    [exec] |   -------------------------------------------------------------------- resolving jdk.rmi at 8-ea requires synthesized java.base@>=8
    [exec] |   --------------------------------------------------------------------- resolving jdk.jndi at 8-ea requires jdk.tls.internal@=8-ea
    [exec] |   ---------------------------------------------------------------------- resolving jdk.jndi at 8-ea requires synthesized java.base@>=8
    [exec] |   ----------------------------------------------------------------------- resolving SERVICE PROVIDER requires optional jdk.desktop