From enatai at proton.me Thu Aug 17 04:14:58 2023 From: enatai at proton.me (Rel) Date: Thu, 17 Aug 2023 04:14:58 +0000 Subject: jextract C++ support In-Reply-To: <8797d776-b75b-e813-ad61-9b95dcf3b9dc@oracle.com> References: <837c7a24-4018-6d91-88f4-22312f19e2c3@oracle.com> <9997b556-0cd9-b390-e110-bf37cdddca39@oracle.com> <1ru-2D5hsy2KC3KU3GHuxq3ST_SzlZfAuiMjdPjex4aOmVVAD8oiJWPJ2egkRChVNONNGGgGSEfJxGQ-ZAbQaJ_8OzP9_z6LtOExpVbvU-0=@proton.me> <1997fa0d-2e51-db7c-1cd3-b6898cf31b62@oracle.com> <8797d776-b75b-e813-ad61-9b95dcf3b9dc@oracle.com> Message-ID: Hi, I start to look on rust-bindgen and did few experiments here: https://github.com/enatai/panamaexperiments/blob/main/rust-bindgen/README.md My question is around what rust-bindgen generated for std::string Mostly those are ordinary data fields for which jextract possibly can generate layout but I don't really understand purpose of: pub _phantom_0: ::std::marker::PhantomData<::std::cell::UnsafeCell<_CharT>>, and what alternative for that can be from jextract. Is it some special Rust feature, or? ------- Original Message ------- On Monday, May 29th, 2023 at 9:00 AM, Maurizio Cimadamore wrote: > On 29/05/2023 00:20, Rel wrote: > >> dynamic dispatch >> >> I tried following example [[https://github.com/enatai/panamaexperiments/blob/main/libcppexperiments/src/main/public/happy.hpp](https://urldefense.com/v3/__https://github.com/enatai/panamaexperiments/blob/main/libcppexperiments/src/main/public/happy.hpp__;!!ACWV5N9M2RV99hQ!JnMeRRKWI0A8qpGkiXzcR0f06AAFEP-dXJV8lT-ZAMcxwE3BhDeGj91EgKkeMudSckPxr_-N9AlNWSGCtyvUZL0$)] and it works fine as long as we generate proper Java bindings. >> >> See test for it [[https://github.com/enatai/panamaexperiments/blob/main/cppexperiments/src/test/java/cppexperiments/HappyTests.java#L36](https://urldefense.com/v3/__https://github.com/enatai/panamaexperiments/blob/main/cppexperiments/src/test/java/cppexperiments/HappyTests.java*L36__;Iw!!ACWV5N9M2RV99hQ!JnMeRRKWI0A8qpGkiXzcR0f06AAFEP-dXJV8lT-ZAMcxwE3BhDeGj91EgKkeMudSckPxr_-N9AlNWSGCRH2ufOA$)] >> >> Please let me know which dynamic dispatch use cases you are concerned with. Because this one seems works fine. > > Sorry, I can see that working fine because you declare a "static" function which accepts a point, so the vtable indirection is generated by the CPP compiler in that function. > > What I'm worried about is calling virtual methods on classes. E.g. calling your "distance" function directly. Jextract gives you two possibilities: Point2d::distance and Point3d::distance. If you pass a Point3d object to Point2d::distance you will "only" get Point2d::distance to be called (as if there was no dynamic dispatch). > >> std::string >> I totally forgot that such basic type like std::string in C++ is a template. >> But it seems possible to call functions which operate with string objects because symbols for them are present: >> >> 00000000000013ad T _ZN7unhappy10helloWorldB5cxx11Ev >> >> std::string helloWorld(); >> >> I guess it is possible to create/extract layout for std::string using FFM but: >> - how to initialize this layout from Java? we cannot just call std::string constructor for it, right? >> - this layout may differ between different C++ runtimes (libstdc++ etc). MS C++ may have not same std::string layout as GCC > > On the latter, e.g. layout difference, this is no different than anything else with jextract. E.g. each jextract run is platform-dependent, as it pulls in header files that are heavily influenced by the platform and OS you run on. > > If I understand correctly, "string" is the "instantiation" of a template in C++. (e.g. some basic_string). That instantiation is fully defined (e.g. not partial), and I believe it should be possible, with libclang, to obtain more information about it - such as the layout etc. (for partial template instantiation, my understanding, reading on what Rust bindgen does is that it is not possible to handle them with libclang). > > So, ideally, we should be able to construct a layout for basic_string, and then pass that to the constructor, yes. > > Maurizio > >>> Their binding generator adopts the same simple approach as the one I showed in the patch. >> >> I will take a look >> >> ------- Original Message ------- >> On Tuesday, May 23rd, 2023 at 8:58 AM, Maurizio Cimadamore [](mailto:maurizio.cimadamore at oracle.com) wrote: >> >>> On 23/05/2023 05:11, Rel wrote: >>> >>>>> What I meant for "robust analysis" was to try and establish how many _real-world_ C++ library can really be tackled in such a direct approach. >>>> >>>> Ohh I see now, I am affraid we know the answer for this :) >>>> >>>> Let's imagine if number of C++ libraries which can be covered end-to-end with "simple" approach is 0, does it mean that we should discard it and only focus on shim for binding all kinds of APIs? What about those cases which can be easily extracted using "simple" approach, like Point2d? >>> >>> Perhaps we should reach out to the Rust community? Their binding generator adopts the same simple approach as the one I showed in the patch. Given how hard it is to support C++ (because the underlying libclang C API is not very solid in that respect), I'd be surprised if they maintained all the necessary code just for stuff like Point2d? >>> >>>> Because I thought that we would like to do "analysis" of what C++ use cases can/cannot be covered with "simple" approach. For example from your previous message I see that we are not completely sure about exceptions: >>>> >>>>> * (probably way more stuff, like exceptions, etc.) >>>> >>>> Similarly for myself I would like to see what are the problems with "dynamic dispatch". I added it to "unhappy" for now, because we expect it not to work, but I plan to test it to see what are the issues there and share here. Similarly with that anyone would be able to reproduce and see same results. >>>> >>>> I guess my question now is: do we think it may be useful to know exactly how many C++ use cases can be covered with "simple" FFM approach. And if answer is yes, then we can use panamaexperiments as a playground where we can have tests for what is covered. This (possibly?) can give us more confidence in limitations of "simple" approach and how far we can go with it (and this can be easily demonstrated to everyone just by running those tests) >>> >>> I think it would be useful to know some answer to that question, yes. My intuition tells me that there are probably two kinds of C++ libraries: those who were born that way, and those that moved over from being simpler C libraries. One such example in the latter category is OpenCV [1]. While its "core" header [2] declares an exception, as well as a bunch of classes, eyeballing it, it doesn't seem "too" problematic? Perhaps that would be a good point where to start, and, if a library such as that can be used with some degree of success, perhaps we can expand the search to other similar libraries. >>> >>> [1] - [https://opencv.org/](https://urldefense.com/v3/__https://opencv.org/__;!!ACWV5N9M2RV99hQ!JnMeRRKWI0A8qpGkiXzcR0f06AAFEP-dXJV8lT-ZAMcxwE3BhDeGj91EgKkeMudSckPxr_-N9AlNWSGC3X8vPfk$) >>> [2] - [https://github.com/opencv/opencv/blob/4.x/modules/core/include/opencv2/core.hpp](https://urldefense.com/v3/__https://github.com/opencv/opencv/blob/4.x/modules/core/include/opencv2/core.hpp__;!!ACWV5N9M2RV99hQ!JnMeRRKWI0A8qpGkiXzcR0f06AAFEP-dXJV8lT-ZAMcxwE3BhDeGj91EgKkeMudSckPxr_-N9AlNWSGCszOjc6Y$) >>> >>>> Ideas? >>>> >>>> ------- Original Message ------- >>>> On Monday, May 22nd, 2023 at 9:14 AM, Maurizio Cimadamore [](mailto:maurizio.cimadamore at oracle.com) wrote: >>>> >>>>> On 22/05/2023 04:12, Rel wrote: >>>>> >>>>>>> But I believe some more robust >>>>>>> analysis should be made to understand exactly how many APIs can be >>>>>>> supported in this "simple" fashion. >>>>>> >>>>>> Yes, I started to gather such analysis here >>>>>> https://urldefense.com/v3/__https://github.com/enatai/panamaexperiments__;!!ACWV5N9M2RV99hQ!KOxdK2qmmSzlaaO3kSlSUG2-ifWAVRD6OHlz9NHYvuggmy7NnnNxWvHYcxDm0Vn4gPXlbasjfC-ehIx_KlWNYiU$ >>>>>> Currently there is only one happy case [ >>>>>> https://urldefense.com/v3/__https://github.com/enatai/panamaexperiments/blob/main/libcppexperiments/src/main/public/happy.hpp__;!!ACWV5N9M2RV99hQ!KOxdK2qmmSzlaaO3kSlSUG2-ifWAVRD6OHlz9NHYvuggmy7NnnNxWvHYcxDm0Vn4gPXlbasjfC-ehIx_K7Nl50c$ >>>>>> ] which is Point2d class from your foo.hpp file. >>>>> >>>>> This is not too surprising - after all the hacky changes I shared were built around that example. >>>>> >>>>> What I meant for "robust analysis" was to try and establish how many _real-world_ C++ library can really be tackled in such a direct approach. My feeling is "not many" - but I don't have any hard data to back up this claim. >>>>> >>>>> Maurizio -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Thu Aug 17 08:46:59 2023 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Thu, 17 Aug 2023 09:46:59 +0100 Subject: jextract C++ support In-Reply-To: References: <837c7a24-4018-6d91-88f4-22312f19e2c3@oracle.com> <9997b556-0cd9-b390-e110-bf37cdddca39@oracle.com> <1ru-2D5hsy2KC3KU3GHuxq3ST_SzlZfAuiMjdPjex4aOmVVAD8oiJWPJ2egkRChVNONNGGgGSEfJxGQ-ZAbQaJ_8OzP9_z6LtOExpVbvU-0=@proton.me> <1997fa0d-2e51-db7c-1cd3-b6898cf31b62@oracle.com> <8797d776-b75b-e813-ad61-9b95dcf3b9dc@oracle.com> Message-ID: <7c469aeb-649a-0b17-c36d-2b98ab9fe7d8@oracle.com> Hi, it seems like the binding generator is emitting bindings for private string fields? https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/include/bits/basic_string.h#L211 While PhantomData seems some magic Rust thingie: https://doc.rust-lang.org/stable/std/marker/struct.PhantomData.html Which is probably used to deal with lifetime of the string char array (but this is a guess). Maurizio On 17/08/2023 05:14, Rel wrote: > Hi, > > I start to look on rust-bindgen and did few experiments here: > > https://github.com/enatai/panamaexperiments/blob/main/rust-bindgen/README.md > > > My question is around what rust-bindgen generated for std::string > > Mostly those are ordinary data fields for which jextract possibly can > generate layout but I don't really understand purpose of: > > pub _phantom_0: > ::std::marker::PhantomData<::std::cell::UnsafeCell<_CharT>>, > > and what alternative for that can be from jextract. Is it some special > Rust feature, or? > ------- Original Message ------- > On Monday, May 29th, 2023 at 9:00 AM, Maurizio Cimadamore > wrote: > >> >> On 29/05/2023 00:20, Rel wrote: >>> dynamic dispatch >>> >>> I tried following example >>> [https://github.com/enatai/panamaexperiments/blob/main/libcppexperiments/src/main/public/happy.hpp >>> ] >>> and it works fine as long as we generate proper Java bindings. >>> >>> See test for it >>> [https://github.com/enatai/panamaexperiments/blob/main/cppexperiments/src/test/java/cppexperiments/HappyTests.java#L36 >>> ] >>> >>> Please let me know which dynamic dispatch use cases you are >>> concerned with. Because this one seems works fine. >> >> Sorry, I can see that working fine because you declare a "static" >> function which accepts a point, so the vtable indirection is >> generated by the CPP compiler in that function. >> >> What I'm worried about is calling virtual methods on classes. E.g. >> calling your "distance" function directly. Jextract gives you two >> possibilities: Point2d::distance and Point3d::distance. If you pass a >> Point3d object to Point2d::distance you will "only" get >> Point2d::distance to be called (as if there was no dynamic dispatch). >> >> >>> >>> std::string >>> I totally forgot that such basic type like std::string in C++ is a >>> template. >>> But it seems possible to call functions which operate with string >>> objects because symbols for them are present: >>> >>> 00000000000013ad T _ZN7unhappy10helloWorldB5cxx11Ev >>> >>> std::string helloWorld(); >>> >>> I guess it is possible to create/extract layout for std::string >>> using FFM but: >>> - how to initialize this layout from Java? we cannot just call >>> std::string constructor for it, right? >>> - this layout may differ between different C++ runtimes (libstdc++ >>> etc). MS C++ may have not same std::string layout as GCC >> >> On the latter, e.g. layout difference, this is no different than >> anything else with jextract. E.g. each jextract run is >> platform-dependent, as it pulls in header files that are heavily >> influenced by the platform and OS you run on. >> >> If I understand correctly, "string" is the "instantiation" of a >> template in C++. (e.g. some basic_string). That instantiation >> is fully defined (e.g. not partial), and I believe it should be >> possible, with libclang, to obtain more information about it - such >> as the layout etc. (for partial template instantiation, my >> understanding, reading on what Rust bindgen does is that it is not >> possible to handle them with libclang). >> >> So, ideally, we should be able to construct a layout for >> basic_string, and then pass that to the constructor, yes. >> >> Maurizio >> >> >>> >>> > Their binding generator adopts the same simple approach as the one I showed in the patch. >>> >>> I will take a look >>> >>> >>> ------- Original Message ------- >>> On Tuesday, May 23rd, 2023 at 8:58 AM, Maurizio Cimadamore >>> wrote: >>> >>>> >>>> On 23/05/2023 05:11, Rel wrote: >>>>> > What I meant for "robust analysis" was to try and establish how many _real-world_ >>>>> C++ library can really be tackled in such a direct approach. >>>>> >>>>> Ohh I see now, I am affraid we know the answer for this :) >>>>> >>>>> Let's imagine if number of C++ libraries which can be covered >>>>> end-to-end with "simple" approach is 0, does it mean that we >>>>> should discard it and only focus on shim for binding all kinds of >>>>> APIs? What about those cases which can be easily extracted using >>>>> "simple" approach, like Point2d? >>>> Perhaps we should reach out to the Rust community? Their binding >>>> generator adopts the same simple approach as the one I showed in >>>> the patch. Given how hard it is to support C++ (because the >>>> underlying libclang C API is not very solid in that respect), I'd >>>> be surprised if they maintained all the necessary code just for >>>> stuff like Point2d? >>>>> >>>>> Because I thought that we would like to do "analysis" of what C++ >>>>> use cases can/cannot be covered with "simple" approach. For >>>>> example from your previous message I see that we are not >>>>> completely sure about exceptions: >>>>> >>>>> > * (probably way more stuff, like exceptions, etc.) >>>>> >>>>> Similarly for myself I would like to see what are the problems >>>>> with "dynamic dispatch". I added it to "unhappy" for now, because >>>>> we expect it not to work, but I plan to test it to see what are >>>>> the issues there and share here. Similarly with that anyone would >>>>> be able to reproduce and see same results. >>>>> >>>>> I guess my question now is: do we think it may be useful to know >>>>> exactly how many C++ use cases can be covered with "simple" FFM >>>>> approach. And if answer is yes, then we can use panamaexperiments >>>>> as a playground where we can have tests for what is covered. This >>>>> (possibly?) can give us more confidence in limitations of "simple" >>>>> approach and how far we can go with it (and this can be easily >>>>> demonstrated to everyone just by running those tests) >>>> >>>> I think it would be useful to know some answer to that question, >>>> yes. My intuition tells me that there are probably two kinds of C++ >>>> libraries: those who were born that way, and those that moved over >>>> from being simpler C libraries. One such example in the latter >>>> category is OpenCV [1]. While its "core" header [2] declares an >>>> exception, as well as a bunch of classes, eyeballing it, it doesn't >>>> seem "too" problematic? Perhaps that would be a good point where to >>>> start, and, if a library such as that can be used with some degree >>>> of success, perhaps we can expand the search to other similar >>>> libraries. >>>> >>>> [1] - https://opencv.org/ >>>> [2] - >>>> https://github.com/opencv/opencv/blob/4.x/modules/core/include/opencv2/core.hpp >>>> >>>>> >>>>> Ideas? >>>>> >>>>> ------- Original Message ------- >>>>> On Monday, May 22nd, 2023 at 9:14 AM, Maurizio Cimadamore >>>>> wrote: >>>>> >>>>>> >>>>>> On 22/05/2023 04:12, Rel wrote: >>>>>>>> But I believe some more robust >>>>>>>> analysis should be made to understand exactly how many APIs can be >>>>>>>> supported in this "simple" fashion. >>>>>>> Yes, I started to gather such analysis herehttps://urldefense.com/v3/__https://github.com/enatai/panamaexperiments__;!!ACWV5N9M2RV99hQ!KOxdK2qmmSzlaaO3kSlSUG2-ifWAVRD6OHlz9NHYvuggmy7NnnNxWvHYcxDm0Vn4gPXlbasjfC-ehIx_KlWNYiU$ >>>>>>> Currently there is only one happy case [https://urldefense.com/v3/__https://github.com/enatai/panamaexperiments/blob/main/libcppexperiments/src/main/public/happy.hpp__;!!ACWV5N9M2RV99hQ!KOxdK2qmmSzlaaO3kSlSUG2-ifWAVRD6OHlz9NHYvuggmy7NnnNxWvHYcxDm0Vn4gPXlbasjfC-ehIx_K7Nl50c$ ] which is Point2d class from your foo.hpp file. >>>>>> >>>>>> This is not too surprising - after all the hacky changes I shared >>>>>> were built around that example. >>>>>> >>>>>> What I meant for "robust analysis" was to try and establish how >>>>>> many _real-world_ C++ library can really be tackled in such a >>>>>> direct approach. My feeling is "not many" - but I don't have any >>>>>> hard data to back up this claim. >>>>>> >>>>>> Maurizio >>>>>> >>>>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From enatai at proton.me Sun Aug 20 18:58:33 2023 From: enatai at proton.me (Rel) Date: Sun, 20 Aug 2023 18:58:33 +0000 Subject: debug logging Message-ID: Hi all, Developing with jextract is a bit difficult: - there is not too much debug logging, the "-Dlibclang.debug=true" enables certain logging but mostly for clang and not jextract itself. To find out what jextract is doing and trace it, is very difficult. It could be remidiated with running debug under IDE but it does not work well if you generate bindings for big "C" projects. Using "-Djextract.debug=true" as of today seems only prints more information during the exceptions but does not show what steps were taken before it - Sometime you receive "Failed to parse /tmp/jextract$1721579888733730008.h:" message, when you try to see contents of /tmp/jextract$1721579888733730008.h to understand why it is failed the file is already deleted. - no logging on what properties we pass to clang Please let me know if I all this make sense or I am doing something wrong. As part of my experiments on c++ side I integrated JUL to address these problems: https://github.com/enatai/jextractcpp/commit/531f1d6183a53b0a1da0c18d7411f15bed7a23e3 If this seems useful for jextract I can send MR for it. Regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Mon Aug 21 14:09:16 2023 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 21 Aug 2023 15:09:16 +0100 Subject: debug logging In-Reply-To: References: Message-ID: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> Hi, I think adding better debugging capabilities to jextract is definitively helpful. I agree that right now jextract tends to be on the quiet side. That said, I think it's important we identify the categories of events that we are interested in logging. Logging clang parameters might be a good idea. As for parsing, I'm not too much of a fan of logging every single parsing event - as that would lead to very very long outputs. It almost seems like what you are looking for is some way to let jextract print some more info about what it wanted to do before it crashed. And I'm sure that this use case is mostly driven by the fact that you are working to _extend_ jextract as opposed to use it. If my hunch is correct, I'd say that using a logger to allow for better debuggability when extending jextract is not the best way to do things. We should probably instead invest in making exceptions/errors more meaningful (I often wished I had a way to print the specific piece of C header we were about to parse when some exception occurred). And logging should instead be used for user-facing events (e.g. which headers are being parsed, which symbols are being excluded, which classes are being written, etc.). An example of that is this: > - Sometime you receive "Failed to parse > /tmp/jextract$1721579888733730008.h:" message, when you try to see > contents of /tmp/jextract$1721579888733730008.h to understand why it > is failed the file is already deleted. This is something a user should never see. E.g. if the user sees it - it is a bug - and you don't want better logging you (as a jextract user) just want it to be fixed. Maurizio From enatai at proton.me Wed Aug 23 01:45:43 2023 From: enatai at proton.me (Rel) Date: Wed, 23 Aug 2023 01:45:43 +0000 Subject: debug logging In-Reply-To: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> References: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> Message-ID: > That said, I think it's important we identify the categories of events that we are interested in logging. Logging clang parameters might be a good idea. As for parsing, I'm not too much of a fan of logging every single parsing event - as that would lead to very very long outputs. What do you think of: INFO - default mode, for everything where we use System.out SEVERE - default mode, for everything where we use System.err FINE - debug mode (clang parameters and some other useful logs) FINER - tracing mode (parsing events, and basically "what jextract wanted to do before it crashed" :). I understand the importance of not giving it to the users, so it will be available only to developers and would require to be manually enabled in logging.properties file first and then would require to build a new binary) Do you think JUL would be a right candidate for logging in jextract? > If my hunch is correct, I'd say that using a logger to allow for better > debuggability when extending jextract is not the best way to do things. > We should probably instead invest in making exceptions/errors more meaningful Curious to hear what others think here. The reasons why I found "tracing mode" is important are: 1. Please correct me if I am wrong: as I see, the spirit of panama branch is to use most recent Java available out there (in fact it may not be even GA yet, like right now Java 22?). Because of that, it is most likely not to be supported by IDEs (jextract requires Java 22, Eclipse last time I checked supported only Java 20) - because of no IDE support, debugging in IDE seems not possible and so "tracing mode" can help here 2. It helps to find what is wrong when use jextract for big projects (very very long outputs can be addressed with grep) How you debug jextract issues in panama branch now? And what would be the setup in this case (possibly such information is a good candidate for "jextract developer guide" later) so I can follow it ------- Original Message ------- On Monday, August 21st, 2023 at 2:09 PM, Maurizio Cimadamore wrote: > Hi, > I think adding better debugging capabilities to jextract is definitively > helpful. I agree that right now jextract tends to be on the quiet side. > > That said, I think it's important we identify the categories of events > that we are interested in logging. Logging clang parameters might be a > good idea. As for parsing, I'm not too much of a fan of logging every > single parsing event - as that would lead to very very long outputs. > > It almost seems like what you are looking for is some way to let > jextract print some more info about what it wanted to do before it > crashed. And I'm sure that this use case is mostly driven by the fact > that you are working to extend jextract as opposed to use it. > > If my hunch is correct, I'd say that using a logger to allow for better > debuggability when extending jextract is not the best way to do things. > We should probably instead invest in making exceptions/errors more > meaningful (I often wished I had a way to print the specific piece of C > header we were about to parse when some exception occurred). > > And logging should instead be used for user-facing events (e.g. which > headers are being parsed, which symbols are being excluded, which > classes are being written, etc.). > > An example of that is this: > > > - Sometime you receive "Failed to parse > > /tmp/jextract$1721579888733730008.h:" message, when you try to see > > contents of /tmp/jextract$1721579888733730008.h to understand why it > > is failed the file is already deleted. > > > This is something a user should never see. E.g. if the user sees it - it > is a bug - and you don't want better logging you (as a jextract user) > just want it to be fixed. > > Maurizio From maurizio.cimadamore at oracle.com Wed Aug 23 12:22:33 2023 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Wed, 23 Aug 2023 13:22:33 +0100 Subject: debug logging In-Reply-To: References: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> Message-ID: <8e72f1e9-d427-5181-1328-6e3411b2448a@oracle.com> On 23/08/2023 02:45, Rel wrote: >> That said, I think it's important we identify the categories of events > that we are interested in logging. Logging clang parameters might be a > good idea. As for parsing, I'm not too much of a fan of logging every > single parsing event - as that would lead to very very long outputs. > > What do you think of: > > INFO - default mode, for everything where we use System.out > SEVERE - default mode, for everything where we use System.err > FINE - debug mode (clang parameters and some other useful logs) > FINER - tracing mode (parsing events, and basically "what jextract wanted to do before it > crashed" :). I understand the importance of not giving it to the users, so it will be available only > to developers and would require to be manually enabled in logging.properties file first and then would require to build a new binary) > > Do you think JUL would be a right candidate for logging in jextract? The classification you propose, as well as using JUL as a starting point doesn't seem too unreasonable. > >> If my hunch is correct, I'd say that using a logger to allow for better >> debuggability when extending jextract is not the best way to do things. >> We should probably instead invest in making exceptions/errors more > meaningful > > Curious to hear what others think here. The reasons why I found "tracing mode" is important are: > 1. Please correct me if I am wrong: as I see, the spirit of panama branch is to use most recent Java available out there (in fact it may not be even GA yet, like right now Java 22?). Because of that, it is most likely not to be supported by IDEs (jextract requires Java 22, Eclipse last time I checked supported only Java 20) > - because of no IDE support, debugging in IDE seems not possible and so "tracing mode" can help here I can run jextract/panama fine using IntelliJ - I have not tried with Eclipse. But your comment confirms my suspicion that, rather than logging things that are "interesting", we're basically trying to debug by println statements :-) I obviously understand where you are coming from (you are just trying to get things done) - but I'm not sure whetever level of finer logging we add would ever be able to replace a debugger - I'd suggest to try and use an IDE which works for your use case :-) (separately, as we have a gradle build to build jextract, how important it is that Eclipse understands Java N ? Maybe you'll miss some highlighting for the new features, but the rest should work? Or will Eclipse categorically refuse to debug a class with a version number it doesn't understand?) > 2. It helps to find what is wrong when use jextract for big projects (very very long outputs can be addressed with grep) > > How you debug jextract issues in panama branch now? And what would be the setup in this case (possibly such information is a good candidate for "jextract developer guide" later) so I can follow it See above - I use IntelliJ and its Gradle integration. I'm able to build fine. I have also installed the jtreg plugin: https://github.com/openjdk/jtreg/tree/master/plugins/idea Which allows me to run tests and debug them. Finally, I have defined some execution targets to run jextract against some header files I've defined locally, to allow me to debug what happens. I'm pretty happy with this setup, and I feel quite productive with it. Hope this helps Maurizio > > ------- Original Message ------- > On Monday, August 21st, 2023 at 2:09 PM, Maurizio Cimadamore wrote: > > >> Hi, >> I think adding better debugging capabilities to jextract is definitively >> helpful. I agree that right now jextract tends to be on the quiet side. >> >> That said, I think it's important we identify the categories of events >> that we are interested in logging. Logging clang parameters might be a >> good idea. As for parsing, I'm not too much of a fan of logging every >> single parsing event - as that would lead to very very long outputs. >> >> It almost seems like what you are looking for is some way to let >> jextract print some more info about what it wanted to do before it >> crashed. And I'm sure that this use case is mostly driven by the fact >> that you are working to extend jextract as opposed to use it. >> >> If my hunch is correct, I'd say that using a logger to allow for better >> debuggability when extending jextract is not the best way to do things. >> We should probably instead invest in making exceptions/errors more >> meaningful (I often wished I had a way to print the specific piece of C >> header we were about to parse when some exception occurred). >> >> And logging should instead be used for user-facing events (e.g. which >> headers are being parsed, which symbols are being excluded, which >> classes are being written, etc.). >> >> An example of that is this: >> >>> - Sometime you receive "Failed to parse >>> /tmp/jextract$1721579888733730008.h:" message, when you try to see >>> contents of /tmp/jextract$1721579888733730008.h to understand why it >>> is failed the file is already deleted. >> >> This is something a user should never see. E.g. if the user sees it - it >> is a bug - and you don't want better logging you (as a jextract user) >> just want it to be fixed. >> >> Maurizio From enatai at proton.me Sun Aug 27 01:23:16 2023 From: enatai at proton.me (Rel) Date: Sun, 27 Aug 2023 01:23:16 +0000 Subject: debug logging In-Reply-To: <8e72f1e9-d427-5181-1328-6e3411b2448a@oracle.com> References: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> <8e72f1e9-d427-5181-1328-6e3411b2448a@oracle.com> Message-ID: Thanks for reply and sharing the setup. I will keep tracing separately in my branch and will not include it as part of logging Merge Request. About MR: which branch should I use? I tried "panama" branch with latest changes but since we switched to Java 22 in commit "ca8942c Update jextract/panama branch to track JDK 22" the build start failing for me: jextract/src/main/java/org/openjdk/jextract/clang/Index.java:74: error: cannot find symbol MemorySegment src = arena.allocateFrom(file); ^ symbol: method allocateFrom(String) location: variable arena of type Arena Java is: java -version openjdk version "22-ea" 2024-03-19 OpenJDK Runtime Environment (build 22-ea+12-877) OpenJDK 64-Bit Server VM (build 22-ea+12-877, mixed mode, sharing) This is the latest from https://jdk.java.net/22/ ------- Original Message ------- On Wednesday, August 23rd, 2023 at 12:22 PM, Maurizio Cimadamore wrote: > On 23/08/2023 02:45, Rel wrote: > > > > That said, I think it's important we identify the categories of events > > > that we are interested in logging. Logging clang parameters might be a > > > good idea. As for parsing, I'm not too much of a fan of logging every > > > single parsing event - as that would lead to very very long outputs. > > > > What do you think of: > > > > INFO - default mode, for everything where we use System.out > > SEVERE - default mode, for everything where we use System.err > > FINE - debug mode (clang parameters and some other useful logs) > > FINER - tracing mode (parsing events, and basically "what jextract wanted to do before it > > crashed" :). I understand the importance of not giving it to the users, so it will be available only > > to developers and would require to be manually enabled in logging.properties file first and then would require to build a new binary) > > > > Do you think JUL would be a right candidate for logging in jextract? > > The classification you propose, as well as using JUL as a starting point > doesn't seem too unreasonable. > > > > If my hunch is correct, I'd say that using a logger to allow for better > > > debuggability when extending jextract is not the best way to do things. > > > We should probably instead invest in making exceptions/errors more > > > meaningful > > > > Curious to hear what others think here. The reasons why I found "tracing mode" is important are: > > 1. Please correct me if I am wrong: as I see, the spirit of panama branch is to use most recent Java available out there (in fact it may not be even GA yet, like right now Java 22?). Because of that, it is most likely not to be supported by IDEs (jextract requires Java 22, Eclipse last time I checked supported only Java 20) > > - because of no IDE support, debugging in IDE seems not possible and so "tracing mode" can help here > > > I can run jextract/panama fine using IntelliJ - I have not tried with > Eclipse. > > But your comment confirms my suspicion that, rather than logging things > that are "interesting", we're basically trying to debug by println > statements :-) > > I obviously understand where you are coming from (you are just trying to > get things done) - but I'm not sure whetever level of finer logging we > add would ever be able to replace a debugger - I'd suggest to try and > use an IDE which works for your use case :-) > > (separately, as we have a gradle build to build jextract, how important > it is that Eclipse understands Java N ? Maybe you'll miss some > highlighting for the new features, but the rest should work? Or will > Eclipse categorically refuse to debug a class with a version number it > doesn't understand?) > > > 2. It helps to find what is wrong when use jextract for big projects (very very long outputs can be addressed with grep) > > > > How you debug jextract issues in panama branch now? And what would be the setup in this case (possibly such information is a good candidate for "jextract developer guide" later) so I can follow it > > > See above - I use IntelliJ and its Gradle integration. I'm able to build > fine. I have also installed the jtreg plugin: > > https://github.com/openjdk/jtreg/tree/master/plugins/idea > > Which allows me to run tests and debug them. > > Finally, I have defined some execution targets to run jextract against > some header files I've defined locally, to allow me to debug what happens. > > I'm pretty happy with this setup, and I feel quite productive with it. > > Hope this helps > Maurizio > > > ------- Original Message ------- > > On Monday, August 21st, 2023 at 2:09 PM, Maurizio Cimadamore maurizio.cimadamore at oracle.com wrote: > > > > > Hi, > > > I think adding better debugging capabilities to jextract is definitively > > > helpful. I agree that right now jextract tends to be on the quiet side. > > > > > > That said, I think it's important we identify the categories of events > > > that we are interested in logging. Logging clang parameters might be a > > > good idea. As for parsing, I'm not too much of a fan of logging every > > > single parsing event - as that would lead to very very long outputs. > > > > > > It almost seems like what you are looking for is some way to let > > > jextract print some more info about what it wanted to do before it > > > crashed. And I'm sure that this use case is mostly driven by the fact > > > that you are working to extend jextract as opposed to use it. > > > > > > If my hunch is correct, I'd say that using a logger to allow for better > > > debuggability when extending jextract is not the best way to do things. > > > We should probably instead invest in making exceptions/errors more > > > meaningful (I often wished I had a way to print the specific piece of C > > > header we were about to parse when some exception occurred). > > > > > > And logging should instead be used for user-facing events (e.g. which > > > headers are being parsed, which symbols are being excluded, which > > > classes are being written, etc.). > > > > > > An example of that is this: > > > > > > > - Sometime you receive "Failed to parse > > > > /tmp/jextract$1721579888733730008.h:" message, when you try to see > > > > contents of /tmp/jextract$1721579888733730008.h to understand why it > > > > is failed the file is already deleted. > > > > > > This is something a user should never see. E.g. if the user sees it - it > > > is a bug - and you don't want better logging you (as a jextract user) > > > just want it to be fixed. > > > > > > Maurizio From sundararajan.athijegannathan at oracle.com Mon Aug 28 04:22:19 2023 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Mon, 28 Aug 2023 04:22:19 +0000 Subject: debug logging In-Reply-To: References: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> <8e72f1e9-d427-5181-1328-6e3411b2448a@oracle.com> Message-ID: Hi, The "panama" branch of "jextract" repo is the bleeding edge code. You need to build "foreign-memaccess+abi" branch of https://github.com/openjdk/panama-foreign and pass the built JDK for "jdk22_home" parameter for jextract Gradle build script. Hope this helps, -Sundar [https://opengraph.githubassets.com/2ade114bd2e33dddd5b0abf2e0b7789cdf5e118da91091f53a82150011d94dcb/openjdk/panama-foreign] GitHub - openjdk/panama-foreign: https://openjdk.org/projects/panama https://openjdk.org/projects/panama. Contribute to openjdk/panama-foreign development by creating an account on GitHub. github.com ________________________________ From: jextract-dev on behalf of Rel Sent: 27 August 2023 06:53 To: Maurizio Cimadamore Cc: jextract-dev at openjdk.org Subject: Re: debug logging Thanks for reply and sharing the setup. I will keep tracing separately in my branch and will not include it as part of logging Merge Request. About MR: which branch should I use? I tried "panama" branch with latest changes but since we switched to Java 22 in commit "ca8942c Update jextract/panama branch to track JDK 22" the build start failing for me: jextract/src/main/java/org/openjdk/jextract/clang/Index.java:74: error: cannot find symbol MemorySegment src = arena.allocateFrom(file); ^ symbol: method allocateFrom(String) location: variable arena of type Arena Java is: java -version openjdk version "22-ea" 2024-03-19 OpenJDK Runtime Environment (build 22-ea+12-877) OpenJDK 64-Bit Server VM (build 22-ea+12-877, mixed mode, sharing) This is the latest from https://jdk.java.net/22/ ------- Original Message ------- On Wednesday, August 23rd, 2023 at 12:22 PM, Maurizio Cimadamore wrote: > On 23/08/2023 02:45, Rel wrote: > > > > That said, I think it's important we identify the categories of events > > > that we are interested in logging. Logging clang parameters might be a > > > good idea. As for parsing, I'm not too much of a fan of logging every > > > single parsing event - as that would lead to very very long outputs. > > > > What do you think of: > > > > INFO - default mode, for everything where we use System.out > > SEVERE - default mode, for everything where we use System.err > > FINE - debug mode (clang parameters and some other useful logs) > > FINER - tracing mode (parsing events, and basically "what jextract wanted to do before it > > crashed" :). I understand the importance of not giving it to the users, so it will be available only > > to developers and would require to be manually enabled in logging.properties file first and then would require to build a new binary) > > > > Do you think JUL would be a right candidate for logging in jextract? > > The classification you propose, as well as using JUL as a starting point > doesn't seem too unreasonable. > > > > If my hunch is correct, I'd say that using a logger to allow for better > > > debuggability when extending jextract is not the best way to do things. > > > We should probably instead invest in making exceptions/errors more > > > meaningful > > > > Curious to hear what others think here. The reasons why I found "tracing mode" is important are: > > 1. Please correct me if I am wrong: as I see, the spirit of panama branch is to use most recent Java available out there (in fact it may not be even GA yet, like right now Java 22?). Because of that, it is most likely not to be supported by IDEs (jextract requires Java 22, Eclipse last time I checked supported only Java 20) > > - because of no IDE support, debugging in IDE seems not possible and so "tracing mode" can help here > > > I can run jextract/panama fine using IntelliJ - I have not tried with > Eclipse. > > But your comment confirms my suspicion that, rather than logging things > that are "interesting", we're basically trying to debug by println > statements :-) > > I obviously understand where you are coming from (you are just trying to > get things done) - but I'm not sure whetever level of finer logging we > add would ever be able to replace a debugger - I'd suggest to try and > use an IDE which works for your use case :-) > > (separately, as we have a gradle build to build jextract, how important > it is that Eclipse understands Java N ? Maybe you'll miss some > highlighting for the new features, but the rest should work? Or will > Eclipse categorically refuse to debug a class with a version number it > doesn't understand?) > > > 2. It helps to find what is wrong when use jextract for big projects (very very long outputs can be addressed with grep) > > > > How you debug jextract issues in panama branch now? And what would be the setup in this case (possibly such information is a good candidate for "jextract developer guide" later) so I can follow it > > > See above - I use IntelliJ and its Gradle integration. I'm able to build > fine. I have also installed the jtreg plugin: > > https://github.com/openjdk/jtreg/tree/master/plugins/idea > > Which allows me to run tests and debug them. > > Finally, I have defined some execution targets to run jextract against > some header files I've defined locally, to allow me to debug what happens. > > I'm pretty happy with this setup, and I feel quite productive with it. > > Hope this helps > Maurizio > > > ------- Original Message ------- > > On Monday, August 21st, 2023 at 2:09 PM, Maurizio Cimadamore maurizio.cimadamore at oracle.com wrote: > > > > > Hi, > > > I think adding better debugging capabilities to jextract is definitively > > > helpful. I agree that right now jextract tends to be on the quiet side. > > > > > > That said, I think it's important we identify the categories of events > > > that we are interested in logging. Logging clang parameters might be a > > > good idea. As for parsing, I'm not too much of a fan of logging every > > > single parsing event - as that would lead to very very long outputs. > > > > > > It almost seems like what you are looking for is some way to let > > > jextract print some more info about what it wanted to do before it > > > crashed. And I'm sure that this use case is mostly driven by the fact > > > that you are working to extend jextract as opposed to use it. > > > > > > If my hunch is correct, I'd say that using a logger to allow for better > > > debuggability when extending jextract is not the best way to do things. > > > We should probably instead invest in making exceptions/errors more > > > meaningful (I often wished I had a way to print the specific piece of C > > > header we were about to parse when some exception occurred). > > > > > > And logging should instead be used for user-facing events (e.g. which > > > headers are being parsed, which symbols are being excluded, which > > > classes are being written, etc.). > > > > > > An example of that is this: > > > > > > > - Sometime you receive "Failed to parse > > > > /tmp/jextract$1721579888733730008.h:" message, when you try to see > > > > contents of /tmp/jextract$1721579888733730008.h to understand why it > > > > is failed the file is already deleted. > > > > > > This is something a user should never see. E.g. if the user sees it - it > > > is a bug - and you don't want better logging you (as a jextract user) > > > just want it to be fixed. > > > > > > Maurizio -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Mon Aug 28 11:03:56 2023 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 28 Aug 2023 12:03:56 +0100 Subject: debug logging In-Reply-To: References: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> <8e72f1e9-d427-5181-1328-6e3411b2448a@oracle.com> Message-ID: <41ef0fc0-c64c-07d0-67b9-50422f5cddbe@oracle.com> On 27/08/2023 02:23, Rel wrote: > jextract/src/main/java/org/openjdk/jextract/clang/Index.java:74: error: cannot find symbol > MemorySegment src = arena.allocateFrom(file); > ^ > symbol: method allocateFrom(String) > location: variable arena of type Arena > > Java is: > > java -version > openjdk version "22-ea" 2024-03-19 I believe the issue is that 22 still doesn't contain the bits in the Panama repository. A PR to integrate those bits is available here: https://git.openjdk.org/jdk/pull/15103 So the following combinations work: 1. JDK 22 + the above PR, jextract panama branch 2. Panama JDK, jextract panama branch (this is the intended use of the panama branch) 3. JDK 21 + jextract jdk21 branch Maurizio From enatai at proton.me Tue Aug 29 06:38:27 2023 From: enatai at proton.me (Rel) Date: Tue, 29 Aug 2023 06:38:27 +0000 Subject: debug logging In-Reply-To: References: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> <8e72f1e9-d427-5181-1328-6e3411b2448a@oracle.com> Message-ID: Hi Sundar, Thanks for reply. I suppose this means that I need to use "panama" branch for the MR? If so, do we have any nightly builds of "foreign-memaccess+abi"? The reason why I ask is that, it may really help me to focus on contributing to jextract rather than building OpenJDK from scratch :) Some Dockerfile with all build setup could help as well (so I then can just build OpenJDK and avoid preparing environment for that) Otherwise I possibly wait until next Java 22 early access build? Thank you, ------- Original Message ------- On Monday, August 28th, 2023 at 4:22 AM, Sundararajan Athijegannathan wrote: > Hi, > > The "panama" branch of "jextract" repo is the bleeding edge code. You need to build "foreign-memaccess+abi" branch of https://github.com/openjdk/panama-foreign and pass the built JDK for "jdk22_home" parameter for jextract Gradle build script. > > Hope this helps, > -Sundar > > https://github.com/openjdk/panama-foreign [GitHub - openjdk/panama-foreign: https://openjdk.org/projects/panama](https://github.com/openjdk/panama-foreign) > https://openjdk.org/projects/panama. Contribute to openjdk/panama-foreign development by creating an account on GitHub. > github.com > > --------------------------------------------------------------- > > From: jextract-dev on behalf of Rel > Sent: 27 August 2023 06:53 > To: Maurizio Cimadamore > Cc: jextract-dev at openjdk.org > Subject: Re: debug logging > > Thanks for reply and sharing the setup. > > I will keep tracing separately in my branch and will not include it as part of logging Merge Request. > > About MR: which branch should I use? > > I tried "panama" branch with latest changes but since we switched to Java 22 in commit "ca8942c Update jextract/panama branch to track JDK 22" the build start failing for me: > > jextract/src/main/java/org/openjdk/jextract/clang/Index.java:74: error: cannot find symbol > MemorySegment src = arena.allocateFrom(file); > ^ > symbol: method allocateFrom(String) > location: variable arena of type Arena > > Java is: > > java -version > openjdk version "22-ea" 2024-03-19 > OpenJDK Runtime Environment (build 22-ea+12-877) > OpenJDK 64-Bit Server VM (build 22-ea+12-877, mixed mode, sharing) > > This is the latest from https://jdk.java.net/22/ > > ------- Original Message ------- > On Wednesday, August 23rd, 2023 at 12:22 PM, Maurizio Cimadamore wrote: > >> On 23/08/2023 02:45, Rel wrote: >> >> > > That said, I think it's important we identify the categories of events >> > > that we are interested in logging. Logging clang parameters might be a >> > > good idea. As for parsing, I'm not too much of a fan of logging every >> > > single parsing event - as that would lead to very very long outputs. >> > >> > What do you think of: >> > >> > INFO - default mode, for everything where we use System.out >> > SEVERE - default mode, for everything where we use System.err >> > FINE - debug mode (clang parameters and some other useful logs) >> > FINER - tracing mode (parsing events, and basically "what jextract wanted to do before it >> > crashed" :). I understand the importance of not giving it to the users, so it will be available only >> > to developers and would require to be manually enabled in logging.properties file first and then would require to build a new binary) >> > >> > Do you think JUL would be a right candidate for logging in jextract? >> >> The classification you propose, as well as using JUL as a starting point >> doesn't seem too unreasonable. >> >> > > If my hunch is correct, I'd say that using a logger to allow for better >> > > debuggability when extending jextract is not the best way to do things. >> > > We should probably instead invest in making exceptions/errors more >> > > meaningful >> > >> > Curious to hear what others think here. The reasons why I found "tracing mode" is important are: >> > 1. Please correct me if I am wrong: as I see, the spirit of panama branch is to use most recent Java available out there (in fact it may not be even GA yet, like right now Java 22?). Because of that, it is most likely not to be supported by IDEs (jextract requires Java 22, Eclipse last time I checked supported only Java 20) >> > - because of no IDE support, debugging in IDE seems not possible and so "tracing mode" can help here >> >> >> I can run jextract/panama fine using IntelliJ - I have not tried with >> Eclipse. >> >> But your comment confirms my suspicion that, rather than logging things >> that are "interesting", we're basically trying to debug by println >> statements :-) >> >> I obviously understand where you are coming from (you are just trying to >> get things done) - but I'm not sure whetever level of finer logging we >> add would ever be able to replace a debugger - I'd suggest to try and >> use an IDE which works for your use case :-) >> >> (separately, as we have a gradle build to build jextract, how important >> it is that Eclipse understands Java N ? Maybe you'll miss some >> highlighting for the new features, but the rest should work? Or will >> Eclipse categorically refuse to debug a class with a version number it >> doesn't understand?) >> >> > 2. It helps to find what is wrong when use jextract for big projects (very very long outputs can be addressed with grep) >> > >> > How you debug jextract issues in panama branch now? And what would be the setup in this case (possibly such information is a good candidate for "jextract developer guide" later) so I can follow it >> >> >> See above - I use IntelliJ and its Gradle integration. I'm able to build >> fine. I have also installed the jtreg plugin: >> >> https://github.com/openjdk/jtreg/tree/master/plugins/idea >> >> Which allows me to run tests and debug them. >> >> Finally, I have defined some execution targets to run jextract against >> some header files I've defined locally, to allow me to debug what happens. >> >> I'm pretty happy with this setup, and I feel quite productive with it. >> >> Hope this helps >> Maurizio >> >> > ------- Original Message ------- >> > On Monday, August 21st, 2023 at 2:09 PM, Maurizio Cimadamore maurizio.cimadamore at oracle.com wrote: >> > >> > > Hi, >> > > I think adding better debugging capabilities to jextract is definitively >> > > helpful. I agree that right now jextract tends to be on the quiet side. >> > > >> > > That said, I think it's important we identify the categories of events >> > > that we are interested in logging. Logging clang parameters might be a >> > > good idea. As for parsing, I'm not too much of a fan of logging every >> > > single parsing event - as that would lead to very very long outputs. >> > > >> > > It almost seems like what you are looking for is some way to let >> > > jextract print some more info about what it wanted to do before it >> > > crashed. And I'm sure that this use case is mostly driven by the fact >> > > that you are working to extend jextract as opposed to use it. >> > > >> > > If my hunch is correct, I'd say that using a logger to allow for better >> > > debuggability when extending jextract is not the best way to do things. >> > > We should probably instead invest in making exceptions/errors more >> > > meaningful (I often wished I had a way to print the specific piece of C >> > > header we were about to parse when some exception occurred). >> > > >> > > And logging should instead be used for user-facing events (e.g. which >> > > headers are being parsed, which symbols are being excluded, which >> > > classes are being written, etc.). >> > > >> > > An example of that is this: >> > > >> > > > - Sometime you receive "Failed to parse >> > > > /tmp/jextract$1721579888733730008.h:" message, when you try to see >> > > > contents of /tmp/jextract$1721579888733730008.h to understand why it >> > > > is failed the file is already deleted. >> > > >> > > This is something a user should never see. E.g. if the user sees it - it >> > > is a bug - and you don't want better logging you (as a jextract user) >> > > just want it to be fixed. >> > > >> > > Maurizio -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Tue Aug 29 10:27:18 2023 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Tue, 29 Aug 2023 11:27:18 +0100 Subject: debug logging In-Reply-To: References: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> <8e72f1e9-d427-5181-1328-6e3411b2448a@oracle.com> Message-ID: On 29/08/2023 07:38, Rel wrote: > Hi Sundar, > > Thanks for reply. > I suppose this means that I need to use "panama" branch for the MR? I believe that would be best - all new development typically happens on that branch. Other branches are meant to be more stable. > > If so, do we have any nightly builds of "foreign-memaccess+abi"? The > reason why I ask is that, it may really help me to focus on > contributing to jextract rather than building OpenJDK from scratch :) > Some Dockerfile with all build setup could help as well (so I then can > just build OpenJDK and avoid preparing environment for that) We do not have any nightly builds for that branch. That branch is "bleeding-edge" and so if you want to work on that, it is assumed you are ok with building the panama repo. This assumption might not be 100% correct, but it's not 100% wrong either :-) > > Otherwise I possibly wait until next Java 22 early access build? I'd say waiting for the first Java 22 EA which has support for the new FFM API would probably be the easiest option. Cheers Maurizio > > Thank you, > > ------- Original Message ------- > On Monday, August 28th, 2023 at 4:22 AM, Sundararajan Athijegannathan > wrote: > >> Hi, >> >> The "panama" branch of "jextract" repo is the bleeding edge code. You >> need to build "foreign-memaccess+abi" branch of >> https://github.com/openjdk/panama-foreign >> ?and >> pass the built JDK for "jdk22_home" parameter for jextract Gradle >> build script. >> >> Hope this helps, >> -Sundar >> >> >> GitHub - openjdk/panama-foreign: https://openjdk.org/projects/panama >> >> https://openjdk.org/projects/panama. Contribute to >> openjdk/panama-foreign development by creating an account on GitHub. >> github.com >> >> >> ------------------------------------------------------------------------ >> *From:* jextract-dev on behalf of Rel >> >> *Sent:* 27 August 2023 06:53 >> *To:* Maurizio Cimadamore >> *Cc:* jextract-dev at openjdk.org >> *Subject:* Re: debug logging >> Thanks for reply and sharing the setup. >> >> I will keep tracing separately in my branch and will not include it >> as part of logging Merge Request. >> >> About MR: which branch should I use? >> >> I tried "panama" branch with latest changes but since we switched to >> Java 22 in commit "ca8942c Update jextract/panama branch to track JDK >> 22" the build start failing for me: >> >> jextract/src/main/java/org/openjdk/jextract/clang/Index.java:74: >> error: cannot find symbol >> ??????????? MemorySegment src = arena.allocateFrom(file); >> ???????????????????????????????????? ^ >> ? symbol:?? method allocateFrom(String) >> ? location: variable arena of type Arena >> >> Java is: >> >> java -version >> openjdk version "22-ea" 2024-03-19 >> OpenJDK Runtime Environment (build 22-ea+12-877) >> OpenJDK 64-Bit Server VM (build 22-ea+12-877, mixed mode, sharing) >> >> This is the latest from https://jdk.java.net/22/ >> >> >> ------- Original Message ------- >> On Wednesday, August 23rd, 2023 at 12:22 PM, Maurizio Cimadamore >> wrote: >> >> >> > On 23/08/2023 02:45, Rel wrote: >> > >> > > > That said, I think it's important we identify the categories of >> events >> > > > that we are interested in logging. Logging clang parameters >> might be a >> > > > good idea. As for parsing, I'm not too much of a fan of logging >> every >> > > > single parsing event - as that would lead to very very long >> outputs. >> > > >> > > What do you think of: >> > > >> > > INFO - default mode, for everything where we use System.out >> > > SEVERE - default mode, for everything where we use System.err >> > > FINE - debug mode (clang parameters and some other useful logs) >> > > FINER - tracing mode (parsing events, and basically "what >> jextract wanted to do before it >> > > crashed" :). I understand the importance of not giving it to the >> users, so it will be available only >> > > to developers and would require to be manually enabled in >> logging.properties file first and then would require to build a new >> binary) >> > > >> > > Do you think JUL would be a right candidate for logging in jextract? >> > >> > The classification you propose, as well as using JUL as a starting >> point >> > doesn't seem too unreasonable. >> > >> > > > If my hunch is correct, I'd say that using a logger to allow >> for better >> > > > debuggability when extending jextract is not the best way to do >> things. >> > > > We should probably instead invest in making exceptions/errors more >> > > > meaningful >> > > >> > > Curious to hear what others think here. The reasons why I found >> "tracing mode" is important are: >> > > 1. Please correct me if I am wrong: as I see, the spirit of >> panama branch is to use most recent Java available out there (in fact >> it may not be even GA yet, like right now Java 22?). Because of that, >> it is most likely not to be supported by IDEs (jextract requires Java >> 22, Eclipse last time I checked supported only Java 20) >> > > - because of no IDE support, debugging in IDE seems not possible >> and so "tracing mode" can help here >> > >> > >> > I can run jextract/panama fine using IntelliJ - I have not tried with >> > Eclipse. >> > >> > But your comment confirms my suspicion that, rather than logging things >> > that are "interesting", we're basically trying to debug by println >> > statements :-) >> > >> > I obviously understand where you are coming from (you are just >> trying to >> > get things done) - but I'm not sure whetever level of finer logging we >> > add would ever be able to replace a debugger - I'd suggest to try and >> > use an IDE which works for your use case :-) >> > >> > (separately, as we have a gradle build to build jextract, how important >> > it is that Eclipse understands Java N ? Maybe you'll miss some >> > highlighting for the new features, but the rest should work? Or will >> > Eclipse categorically refuse to debug a class with a version number it >> > doesn't understand?) >> > >> > > 2. It helps to find what is wrong when use jextract for big >> projects (very very long outputs can be addressed with grep) >> > > >> > > How you debug jextract issues in panama branch now? And what >> would be the setup in this case (possibly such information is a good >> candidate for "jextract developer guide" later) so I can follow it >> > >> > >> > See above - I use IntelliJ and its Gradle integration. I'm able to >> build >> > fine. I have also installed the jtreg plugin: >> > >> > https://github.com/openjdk/jtreg/tree/master/plugins/idea >> >> > >> > Which allows me to run tests and debug them. >> > >> > Finally, I have defined some execution targets to run jextract against >> > some header files I've defined locally, to allow me to debug what >> happens. >> > >> > I'm pretty happy with this setup, and I feel quite productive with it. >> > >> > Hope this helps >> > Maurizio >> > >> > > ------- Original Message ------- >> > > On Monday, August 21st, 2023 at 2:09 PM, Maurizio Cimadamore >> maurizio.cimadamore at oracle.com wrote: >> > > >> > > > Hi, >> > > > I think adding better debugging capabilities to jextract is >> definitively >> > > > helpful. I agree that right now jextract tends to be on the >> quiet side. >> > > > >> > > > That said, I think it's important we identify the categories of >> events >> > > > that we are interested in logging. Logging clang parameters >> might be a >> > > > good idea. As for parsing, I'm not too much of a fan of logging >> every >> > > > single parsing event - as that would lead to very very long >> outputs. >> > > > >> > > > It almost seems like what you are looking for is some way to let >> > > > jextract print some more info about what it wanted to do before it >> > > > crashed. And I'm sure that this use case is mostly driven by >> the fact >> > > > that you are working to extend jextract as opposed to use it. >> > > > >> > > > If my hunch is correct, I'd say that using a logger to allow >> for better >> > > > debuggability when extending jextract is not the best way to do >> things. >> > > > We should probably instead invest in making exceptions/errors more >> > > > meaningful (I often wished I had a way to print the specific >> piece of C >> > > > header we were about to parse when some exception occurred). >> > > > >> > > > And logging should instead be used for user-facing events (e.g. >> which >> > > > headers are being parsed, which symbols are being excluded, which >> > > > classes are being written, etc.). >> > > > >> > > > An example of that is this: >> > > > >> > > > > - Sometime you receive "Failed to parse >> > > > > /tmp/jextract$1721579888733730008.h:" message, when you try >> to see >> > > > > contents of /tmp/jextract$1721579888733730008.h to understand >> why it >> > > > > is failed the file is already deleted. >> > > > >> > > > This is something a user should never see. E.g. if the user >> sees it - it >> > > > is a bug - and you don't want better logging you (as a jextract >> user) >> > > > just want it to be fixed. >> > > > >> > > > Maurizio > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sundararajan.athijegannathan at oracle.com Tue Aug 29 10:42:39 2023 From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan) Date: Tue, 29 Aug 2023 10:42:39 +0000 Subject: [External] : Re: debug logging In-Reply-To: References: <7a305085-c11c-6f4d-015d-8bf72f3a2de1@oracle.com> <8e72f1e9-d427-5181-1328-6e3411b2448a@oracle.com> Message-ID: Hi, There is no nightly build of "foreign-memaccess+abi". If you don't want to build JDK, I think you may want to wait for a future JDK 22 early access build ? Thanks, -Sundar ________________________________ From: Rel Sent: 29 August 2023 12:08 To: Sundararajan Athijegannathan Cc: Maurizio Cimadamore ; jextract-dev at openjdk.org Subject: [External] : Re: debug logging Hi Sundar, Thanks for reply. I suppose this means that I need to use "panama" branch for the MR? If so, do we have any nightly builds of "foreign-memaccess+abi"? The reason why I ask is that, it may really help me to focus on contributing to jextract rather than building OpenJDK from scratch :) Some Dockerfile with all build setup could help as well (so I then can just build OpenJDK and avoid preparing environment for that) Otherwise I possibly wait until next Java 22 early access build? Thank you, ------- Original Message ------- On Monday, August 28th, 2023 at 4:22 AM, Sundararajan Athijegannathan wrote: Hi, The "panama" branch of "jextract" repo is the bleeding edge code. You need to build "foreign-memaccess+abi" branch of https://github.com/openjdk/panama-foreign and pass the built JDK for "jdk22_home" parameter for jextract Gradle build script. Hope this helps, -Sundar [https://opengraph.githubassets.com/2ade114bd2e33dddd5b0abf2e0b7789cdf5e118da91091f53a82150011d94dcb/openjdk/panama-foreign] GitHub - openjdk/panama-foreign: https://openjdk.org/projects/panama https://openjdk.org/projects/panama. Contribute to openjdk/panama-foreign development by creating an account on GitHub. github.com ________________________________ From: jextract-dev on behalf of Rel Sent: 27 August 2023 06:53 To: Maurizio Cimadamore Cc: jextract-dev at openjdk.org Subject: Re: debug logging Thanks for reply and sharing the setup. I will keep tracing separately in my branch and will not include it as part of logging Merge Request. About MR: which branch should I use? I tried "panama" branch with latest changes but since we switched to Java 22 in commit "ca8942c Update jextract/panama branch to track JDK 22" the build start failing for me: jextract/src/main/java/org/openjdk/jextract/clang/Index.java:74: error: cannot find symbol MemorySegment src = arena.allocateFrom(file); ^ symbol: method allocateFrom(String) location: variable arena of type Arena Java is: java -version openjdk version "22-ea" 2024-03-19 OpenJDK Runtime Environment (build 22-ea+12-877) OpenJDK 64-Bit Server VM (build 22-ea+12-877, mixed mode, sharing) This is the latest from https://jdk.java.net/22/ ------- Original Message ------- On Wednesday, August 23rd, 2023 at 12:22 PM, Maurizio Cimadamore wrote: > On 23/08/2023 02:45, Rel wrote: > > > > That said, I think it's important we identify the categories of events > > > that we are interested in logging. Logging clang parameters might be a > > > good idea. As for parsing, I'm not too much of a fan of logging every > > > single parsing event - as that would lead to very very long outputs. > > > > What do you think of: > > > > INFO - default mode, for everything where we use System.out > > SEVERE - default mode, for everything where we use System.err > > FINE - debug mode (clang parameters and some other useful logs) > > FINER - tracing mode (parsing events, and basically "what jextract wanted to do before it > > crashed" :). I understand the importance of not giving it to the users, so it will be available only > > to developers and would require to be manually enabled in logging.properties file first and then would require to build a new binary) > > > > Do you think JUL would be a right candidate for logging in jextract? > > The classification you propose, as well as using JUL as a starting point > doesn't seem too unreasonable. > > > > If my hunch is correct, I'd say that using a logger to allow for better > > > debuggability when extending jextract is not the best way to do things. > > > We should probably instead invest in making exceptions/errors more > > > meaningful > > > > Curious to hear what others think here. The reasons why I found "tracing mode" is important are: > > 1. Please correct me if I am wrong: as I see, the spirit of panama branch is to use most recent Java available out there (in fact it may not be even GA yet, like right now Java 22?). Because of that, it is most likely not to be supported by IDEs (jextract requires Java 22, Eclipse last time I checked supported only Java 20) > > - because of no IDE support, debugging in IDE seems not possible and so "tracing mode" can help here > > > I can run jextract/panama fine using IntelliJ - I have not tried with > Eclipse. > > But your comment confirms my suspicion that, rather than logging things > that are "interesting", we're basically trying to debug by println > statements :-) > > I obviously understand where you are coming from (you are just trying to > get things done) - but I'm not sure whetever level of finer logging we > add would ever be able to replace a debugger - I'd suggest to try and > use an IDE which works for your use case :-) > > (separately, as we have a gradle build to build jextract, how important > it is that Eclipse understands Java N ? Maybe you'll miss some > highlighting for the new features, but the rest should work? Or will > Eclipse categorically refuse to debug a class with a version number it > doesn't understand?) > > > 2. It helps to find what is wrong when use jextract for big projects (very very long outputs can be addressed with grep) > > > > How you debug jextract issues in panama branch now? And what would be the setup in this case (possibly such information is a good candidate for "jextract developer guide" later) so I can follow it > > > See above - I use IntelliJ and its Gradle integration. I'm able to build > fine. I have also installed the jtreg plugin: > > https://github.com/openjdk/jtreg/tree/master/plugins/idea > > Which allows me to run tests and debug them. > > Finally, I have defined some execution targets to run jextract against > some header files I've defined locally, to allow me to debug what happens. > > I'm pretty happy with this setup, and I feel quite productive with it. > > Hope this helps > Maurizio > > > ------- Original Message ------- > > On Monday, August 21st, 2023 at 2:09 PM, Maurizio Cimadamore maurizio.cimadamore at oracle.com wrote: > > > > > Hi, > > > I think adding better debugging capabilities to jextract is definitively > > > helpful. I agree that right now jextract tends to be on the quiet side. > > > > > > That said, I think it's important we identify the categories of events > > > that we are interested in logging. Logging clang parameters might be a > > > good idea. As for parsing, I'm not too much of a fan of logging every > > > single parsing event - as that would lead to very very long outputs. > > > > > > It almost seems like what you are looking for is some way to let > > > jextract print some more info about what it wanted to do before it > > > crashed. And I'm sure that this use case is mostly driven by the fact > > > that you are working to extend jextract as opposed to use it. > > > > > > If my hunch is correct, I'd say that using a logger to allow for better > > > debuggability when extending jextract is not the best way to do things. > > > We should probably instead invest in making exceptions/errors more > > > meaningful (I often wished I had a way to print the specific piece of C > > > header we were about to parse when some exception occurred). > > > > > > And logging should instead be used for user-facing events (e.g. which > > > headers are being parsed, which symbols are being excluded, which > > > classes are being written, etc.). > > > > > > An example of that is this: > > > > > > > - Sometime you receive "Failed to parse > > > > /tmp/jextract$1721579888733730008.h:" message, when you try to see > > > > contents of /tmp/jextract$1721579888733730008.h to understand why it > > > > is failed the file is already deleted. > > > > > > This is something a user should never see. E.g. if the user sees it - it > > > is a bug - and you don't want better logging you (as a jextract user) > > > just want it to be fixed. > > > > > > Maurizio -------------- next part -------------- An HTML attachment was scrubbed... URL: From sundar at openjdk.org Tue Aug 29 11:31:47 2023 From: sundar at openjdk.org (Athijegannathan Sundararajan) Date: Tue, 29 Aug 2023 11:31:47 GMT Subject: RFR: 7903542: jextract sample compilation scripts can avoid specifying standard include directory in Mac OS Message-ID: jextract infers standard include directory automatically. ------------- Commit messages: - 7903542: jextract sample compilation scripts can avoid specifying standard include directory in Mac OS Changes: https://git.openjdk.org/jextract/pull/127/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=127&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903542 Stats: 31 lines in 29 files changed: 0 ins; 31 del; 0 mod Patch: https://git.openjdk.org/jextract/pull/127.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/127/head:pull/127 PR: https://git.openjdk.org/jextract/pull/127 From mcimadamore at openjdk.org Tue Aug 29 11:53:37 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Tue, 29 Aug 2023 11:53:37 GMT Subject: RFR: 7903542: jextract sample compilation scripts can avoid specifying standard include directory in Mac OS In-Reply-To: References: Message-ID: On Tue, 29 Aug 2023 11:18:33 GMT, Athijegannathan Sundararajan wrote: > jextract infers standard include directory automatically. Marked as reviewed by mcimadamore (Committer). ------------- PR Review: https://git.openjdk.org/jextract/pull/127#pullrequestreview-1600223652 From sundar at openjdk.org Tue Aug 29 13:14:36 2023 From: sundar at openjdk.org (Athijegannathan Sundararajan) Date: Tue, 29 Aug 2023 13:14:36 GMT Subject: Integrated: 7903542: jextract sample compilation scripts can avoid specifying standard include directory in Mac OS In-Reply-To: References: Message-ID: <48WFoBbTA7ZpXWgP09RleJRBv9cZ2Hwc70ipTYoBfq4=.3064ad71-fa61-4d20-9a3e-5041421d7c3b@github.com> On Tue, 29 Aug 2023 11:18:33 GMT, Athijegannathan Sundararajan wrote: > jextract infers standard include directory automatically. This pull request has now been integrated. Changeset: 80b53e87 Author: Athijegannathan Sundararajan URL: https://git.openjdk.org/jextract/commit/80b53e87ff42e350eeb71a2076c952408f91a8e6 Stats: 31 lines in 29 files changed: 0 ins; 31 del; 0 mod 7903542: jextract sample compilation scripts can avoid specifying standard include directory in Mac OS Reviewed-by: mcimadamore ------------- PR: https://git.openjdk.org/jextract/pull/127 From sundar at openjdk.org Thu Aug 31 09:39:31 2023 From: sundar at openjdk.org (Athijegannathan Sundararajan) Date: Thu, 31 Aug 2023 09:39:31 GMT Subject: RFR: 7903547: libffmpeg sample crashes JVM Message-ID: try..finally is moved inside another enclosing try-with-resources for the confined Arena object. Repeated runs of sample many times do not crash anymore. ------------- Commit messages: - 7903547: libffmpeg sample crashes JVM Changes: https://git.openjdk.org/jextract/pull/128/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=128&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903547 Stats: 241 lines in 1 file changed: 63 ins; 79 del; 99 mod Patch: https://git.openjdk.org/jextract/pull/128.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/128/head:pull/128 PR: https://git.openjdk.org/jextract/pull/128 From mcimadamore at openjdk.org Thu Aug 31 10:00:28 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Thu, 31 Aug 2023 10:00:28 GMT Subject: RFR: 7903547: libffmpeg sample crashes JVM In-Reply-To: References: Message-ID: On Thu, 31 Aug 2023 09:33:39 GMT, Athijegannathan Sundararajan wrote: > try..finally is moved inside another enclosing try-with-resources for the confined Arena object. Repeated runs of sample many times do not crash anymore. Looks good. I believe there's probably a better way to write the code that doesn't use a double try, but let's go with this for the time being. ------------- Marked as reviewed by mcimadamore (Committer). PR Review: https://git.openjdk.org/jextract/pull/128#pullrequestreview-1604390873 From sundar at openjdk.org Thu Aug 31 10:07:26 2023 From: sundar at openjdk.org (Athijegannathan Sundararajan) Date: Thu, 31 Aug 2023 10:07:26 GMT Subject: Integrated: 7903547: libffmpeg sample crashes JVM In-Reply-To: References: Message-ID: <5hxn_71kyCewLg91vmPtdLxXT6q2RtIu9d3Zmphb6YQ=.451416a1-6aef-4c4b-9bdc-a6ab9537afb3@github.com> On Thu, 31 Aug 2023 09:33:39 GMT, Athijegannathan Sundararajan wrote: > try..finally is moved inside another enclosing try-with-resources for the confined Arena object. Repeated runs of sample many times do not crash anymore. This pull request has now been integrated. Changeset: 06be24da Author: Athijegannathan Sundararajan URL: https://git.openjdk.org/jextract/commit/06be24da0fcc3cff459270a91428ec7658b91526 Stats: 241 lines in 1 file changed: 63 ins; 79 del; 99 mod 7903547: libffmpeg sample crashes JVM Reviewed-by: mcimadamore ------------- PR: https://git.openjdk.org/jextract/pull/128