Message ID | 1536186133-9933-1-git-send-email-angelo.compagnucci@gmail.com |
---|---|
Headers | show |
Series | Add tainting support to buildroot | expand |
Hello, On Thu, 6 Sep 2018 00:22:10 +0200, Angelo Compagnucci wrote: > Packages that need to resolve dependencies internally > and use a package manager would harm the reproducibility > of a build, moreover they escape the legal infrastructure > not giving enough informations on licensing. > > This patch adds a tainting mechanism in the form of a > variable FOO_TAINTS that can be used to signal that > a package harms the reproducibility or licensing under > certain conditions. > > This opens the door to include per language dependency > managers in buildroot. > > Angelo Compagnucci (3): > Makefile: add tainting support > docs/manual: adding infos about tainting > package/nodejs: taint the build on external modules It would be good to include a changelog of what has been changed between the different iterations of your series. Thomas
Angelo, Thomas, All, On 2018-09-06 00:22 +0200, Angelo Compagnucci spake thusly: > Packages that need to resolve dependencies internally > and use a package manager would harm the reproducibility > of a build, moreover they escape the legal infrastructure > not giving enough informations on licensing. So, I am not in favour of having support for such package managers in Buildroot, to begin with. As you say, the reproducibility of a build, as offered by Buildroot, is really a very important point. In my opinion, reproducibility is even paramount. Whatever gets in the way should be banned. > This patch adds a tainting mechanism in the form of a > variable FOO_TAINTS that can be used to signal that > a package harms the reproducibility or licensing under > certain conditions. When I read that, it comes that the tainting mechanism serves two purposes: first, mark (non-)reproducibility, and second, mark incorrect and/or incomplete licensing information. This does not sound nice to me. For the licensing information, I would just rely on the existing licensing infra, and be done with it, i.e. add : ifneq ($(FOO_NON_REPRODUCIBLE_EXTERNAL_DATA),) FOO_LICENSE += Unknown (non reproducible external data) endif I.e., no need for the tainting mechanism to represent the licensing problem, as we already have what is needed to cover it. Now, what would be the purpose for this "tainted" flag? I understand clearly what it provides, indeed, technically, but what would it serve to the user? If I were to use a non-reproduible set of data ffor;, say, nodejs, then I know that this is not handled by Buildroot, and thus it is not reproducible. I don't need to be told it, escept maybe as a note in the manual: "If you use external data from npm/pypi, cpan, whatnot, then your build is not reproducible; that will be hurting kittens." Instead, I am more in favour of packaging such external stuff as Buildroot packages, like we've been doing for erlang, lua, perl, and python, even if we have to add new infrastructures for thos (npm, I'm looking at you!) Besides, you're missing a big piece of potential non-reproducibility source: br2-external trees. If we ever were to have such a tainting mechanism, we should mark the build as tainted as soon as there is a br2-external tree. Ditto as soon as we have a FOO_OVERRIDE_SRCDIR set, which should mark the build as tainted. Except we already use FOO_OVERRIDE_SRCDIR when FOO_SITE = local... > This opens the door to include per language dependency > managers in buildroot. Instead, why not provide a helper script (or enhance the existing ones if need be), than generated buildroot packages for those external packge managers? Sorry, this is not the technical review you may have expected. Instead, it is a position statement on my part. Regards, Yann E. MORIN. > Angelo Compagnucci (3): > Makefile: add tainting support > docs/manual: adding infos about tainting > package/nodejs: taint the build on external modules > > Makefile | 10 ++++++++++ > docs/manual/adding-packages-generic.txt | 6 ++++++ > docs/manual/legal-notice.txt | 12 ++++++++++++ > package/nodejs/nodejs.mk | 1 + > package/pkg-generic.mk | 15 +++++++++++++++ > 5 files changed, 44 insertions(+) > > -- > 2.7.4 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Hello Yann, On Sun, 9 Sep 2018 09:36:16 +0200, Yann E. MORIN wrote: > So, I am not in favour of having support for such package managers in > Buildroot, to begin with. We already had this discussion in the first iteration of this patch series, and even before. I think everybody agrees that such package managers (npm and al.) are really not nice in the context of Linux distributions or build systems. However, from a practical point of view, some people want those features, and are ready to trade the reproducibility against "ease of use". In addition, specifically for nodejs/npm, the number of packages is so enormous that I am not sure it is really realistic to create one Buildroot package for each nodejs module, and keep things sane. Especially if only a few people are interested in nodejs modules. > As you say, the reproducibility of a build, as offered by Buildroot, is > really a very important point. In my opinion, reproducibility is even > paramount. Whatever gets in the way should be banned. I disagree on the "should be banned", and that's the whole point of this series: allow it, while making sure that the user is well aware of the fact that he has given up on the "reproducibility" by using one of those package managers. > > This patch adds a tainting mechanism in the form of a > > variable FOO_TAINTS that can be used to signal that > > a package harms the reproducibility or licensing under > > certain conditions. > > When I read that, it comes that the tainting mechanism serves two > purposes: first, mark (non-)reproducibility, and second, mark incorrect > and/or incomplete licensing information. > > This does not sound nice to me. > > For the licensing information, I would just rely on the existing > licensing infra, and be done with it, i.e. add : > > ifneq ($(FOO_NON_REPRODUCIBLE_EXTERNAL_DATA),) > FOO_LICENSE += Unknown (non reproducible external data) > endif That's true. > Now, what would be the purpose for this "tainted" flag? I understand > clearly what it provides, indeed, technically, but what would it serve > to the user? See above: it makes it absolutely clear to the user that he is using some feature that kills one key advantage of build system: reproducibility. Hiding this information in the manual or in a Config.in help text is IMO not enough: we really want the user to have a prominent warning at the end of the build. > If I were to use a non-reproduible set of data ffor;, say, nodejs, then > I know that this is not handled by Buildroot, and thus it is not > reproducible. I don't need to be told it, escept maybe as a note in the > manual: "If you use external data from npm/pypi, cpan, whatnot, then > your build is not reproducible; that will be hurting kittens." I disagree, newcomers are unlikely to realize this and read our manual end to end. > Instead, I am more in favour of packaging such external stuff as > Buildroot packages, like we've been doing for erlang, lua, perl, and > python, even if we have to add new infrastructures for thos (npm, I'm > looking at you!) Yes, that is the ideal situation. But that is not necessarily realistic for npm modules (due to their number), and having this option of using the provided package manager is reasonable, as long as the user is aware of the drawbacks. > Besides, you're missing a big piece of potential non-reproducibility > source: br2-external trees. If we ever were to have such a tainting > mechanism, we should mark the build as tainted as soon as there is a > br2-external tree. I disagree, a BR2_EXTERNAL does not make the build non-reproducible. As long as you have the same BR2_EXTERNAL + Buildroot, you can rebuild the same system. > Ditto as soon as we have a FOO_OVERRIDE_SRCDIR set, which should mark > the build as tainted. Except we already use FOO_OVERRIDE_SRCDIR when > FOO_SITE = local... FOO_OVERRIDE_SRCDIR is indeed a better example of thing that makes the build non-reproducible, we could potentially include this in the tainted mechanism. Though to me, this is somewhat less important: only advanced users are likely to use <pkg>_OVERRIDE_SRCDIR, and if they do, they will definitely realize the impact on reproducibility. While the newcomer toying around with nodejs/npm may not realize the impact of using a third-party package manager on reproducibility. So, yes, perhaps <pkg>_OVERRIDE_SRCDIR should ultimately taint the build, but I don't see handling that as a requirement to start introducing this "taint" flag. > Instead, why not provide a helper script (or enhance the existing ones > if need be), than generated buildroot packages for those external packge > managers? We really don't want thousands of npm packages I believe :-/ Best regards, Thomas
Thomas, Yann, On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > Hello Yann, > > On Sun, 9 Sep 2018 09:36:16 +0200, Yann E. MORIN wrote: > > > So, I am not in favour of having support for such package managers in > > Buildroot, to begin with. > > We already had this discussion in the first iteration of this patch > series, and even before. I think everybody agrees that such package > managers (npm and al.) are really not nice in the context of Linux > distributions or build systems. > > However, from a practical point of view, some people want those > features, and are ready to trade the reproducibility against "ease of > use". In addition, specifically for nodejs/npm, the number of packages > is so enormous that I am not sure it is really realistic to create one > Buildroot package for each nodejs module, and keep things sane. > Especially if only a few people are interested in nodejs modules. > > > As you say, the reproducibility of a build, as offered by Buildroot, is > > really a very important point. In my opinion, reproducibility is even > > paramount. Whatever gets in the way should be banned. > > I disagree on the "should be banned", and that's the whole point of > this series: allow it, while making sure that the user is well aware of > the fact that he has given up on the "reproducibility" by using one of > those package managers. > > > > This patch adds a tainting mechanism in the form of a > > > variable FOO_TAINTS that can be used to signal that > > > a package harms the reproducibility or licensing under > > > certain conditions. > > > > When I read that, it comes that the tainting mechanism serves two > > purposes: first, mark (non-)reproducibility, and second, mark incorrect > > and/or incomplete licensing information. > > > > This does not sound nice to me. > > > > For the licensing information, I would just rely on the existing > > licensing infra, and be done with it, i.e. add : > > > > ifneq ($(FOO_NON_REPRODUCIBLE_EXTERNAL_DATA),) > > FOO_LICENSE += Unknown (non reproducible external data) > > endif > > That's true. > > > Now, what would be the purpose for this "tainted" flag? I understand > > clearly what it provides, indeed, technically, but what would it serve > > to the user? > > See above: it makes it absolutely clear to the user that he is using > some feature that kills one key advantage of build system: > reproducibility. Hiding this information in the manual or in a > Config.in help text is IMO not enough: we really want the user to have > a prominent warning at the end of the build. > > > If I were to use a non-reproduible set of data ffor;, say, nodejs, then > > I know that this is not handled by Buildroot, and thus it is not > > reproducible. I don't need to be told it, escept maybe as a note in the > > manual: "If you use external data from npm/pypi, cpan, whatnot, then > > your build is not reproducible; that will be hurting kittens." > > I disagree, newcomers are unlikely to realize this and read our manual > end to end. > > > Instead, I am more in favour of packaging such external stuff as > > Buildroot packages, like we've been doing for erlang, lua, perl, and > > python, even if we have to add new infrastructures for thos (npm, I'm > > looking at you!) > > Yes, that is the ideal situation. But that is not necessarily realistic > for npm modules (due to their number), and having this option of using > the provided package manager is reasonable, as long as the user is > aware of the drawbacks. > > > Besides, you're missing a big piece of potential non-reproducibility > > source: br2-external trees. If we ever were to have such a tainting > > mechanism, we should mark the build as tainted as soon as there is a > > br2-external tree. > > I disagree, a BR2_EXTERNAL does not make the build non-reproducible. As > long as you have the same BR2_EXTERNAL + Buildroot, you can rebuild the > same system. > > > Ditto as soon as we have a FOO_OVERRIDE_SRCDIR set, which should mark > > the build as tainted. Except we already use FOO_OVERRIDE_SRCDIR when > > FOO_SITE = local... > > FOO_OVERRIDE_SRCDIR is indeed a better example of thing that makes the > build non-reproducible, we could potentially include this in the > tainted mechanism. > > Though to me, this is somewhat less important: only advanced users are > likely to use <pkg>_OVERRIDE_SRCDIR, and if they do, they will > definitely realize the impact on reproducibility. While the newcomer > toying around with nodejs/npm may not realize the impact of using a > third-party package manager on reproducibility. > > So, yes, perhaps <pkg>_OVERRIDE_SRCDIR should ultimately taint the > build, but I don't see handling that as a requirement to start > introducing this "taint" flag. > > > Instead, why not provide a helper script (or enhance the existing ones > > if need be), than generated buildroot packages for those external packge > > managers? > > We really don't want thousands of npm packages I believe :-/ Thomas said evrithing exceptionally well, but I would like to underling another thing: automated building infrastructures like continuos integration. If a project hasa nedd of reproducibility, the countinous integration could check if a random developer introduced something not reproducible and mark the build as invalid. I think this is really a big plus of this solution. Sincerely, Angelo > > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Thomas, Angelo, All, On 2018-09-09 14:10 +0200, Thomas Petazzoni spake thusly: > On Sun, 9 Sep 2018 09:36:16 +0200, Yann E. MORIN wrote: > > So, I am not in favour of having support for such package managers in > > Buildroot, to begin with. > > We already had this discussion in the first iteration of this patch > series, and even before. I think everybody agrees that such package > managers (npm and al.) are really not nice in the context of Linux > distributions or build systems. > > However, from a practical point of view, some people want those > features, and are ready to trade the reproducibility against "ease of > use". In addition, specifically for nodejs/npm, the number of packages > is so enormous that I am not sure it is really realistic to create one > Buildroot package for each nodejs module, and keep things sane. Well, we already have ~269 python packages, and that is already a lot. There are 151,504 (as displayed today on pypi.org) python packages. Yet, we have an infrastructure for them. > Especially if only a few people are interested in nodejs modules. > > > As you say, the reproducibility of a build, as offered by Buildroot, is > > really a very important point. In my opinion, reproducibility is even > > paramount. Whatever gets in the way should be banned. > > I disagree on the "should be banned", and that's the whole point of > this series: allow it, while making sure that the user is well aware of > the fact that he has given up on the "reproducibility" by using one of > those package managers. The taint status is not something that we can correctly come up with: we have no way to know whether a specific set of external data, packages, dependencies, stuff, whatever, will or won't taint the build, as outlined by the examples I provided in the nodejs/npm case. So, because we can not come up with a conclusive status of whether a build is or is not tainted, we should not have infra for reporting it. > > Now, what would be the purpose for this "tainted" flag? I understand > > clearly what it provides, indeed, technically, but what would it serve > > to the user? > > See above: it makes it absolutely clear to the user that he is using > some feature that kills one key advantage of build system: > reproducibility. And what I am saying is that we can't provide that status, because we don't know whether a build is tainted or is not. And if we report a build is tainted when it is not, then it becomes useless, as a real taint that is later introduced will go unnoticed. [--SNIP--] > > Instead, I am more in favour of packaging such external stuff as > > Buildroot packages, like we've been doing for erlang, lua, perl, and > > python, even if we have to add new infrastructures for thos (npm, I'm > > looking at you!) > > Yes, that is the ideal situation. But that is not necessarily realistic > for npm modules (due to their number), and having this option of using > the provided package manager is reasonable, as long as the user is > aware of the drawbacks. > > > Besides, you're missing a big piece of potential non-reproducibility > > source: br2-external trees. If we ever were to have such a tainting > > mechanism, we should mark the build as tainted as soon as there is a > > br2-external tree. > > I disagree, a BR2_EXTERNAL does not make the build non-reproducible. I said "potential non-reproducibility source". Because it is in a br2-external, the user is free to do so many hacks that we can't assess whether the build is reproducible or not. And again, the important piece of information is that: we _can_ _not_ assess the tainted status, either way. So, I am arguing that we should not have a way to report the tainted status to the userm because we don't know how to decide whther a build is tainted or not. > As > long as you have the same BR2_EXTERNAL + Buildroot, you can rebuild the > same system. Yeah, I know it very well, thank you! ;-) But how do you know that the combination is reproducible or is not? > > Ditto as soon as we have a FOO_OVERRIDE_SRCDIR set, which should mark > > the build as tainted. Except we already use FOO_OVERRIDE_SRCDIR when > > FOO_SITE = local... > > FOO_OVERRIDE_SRCDIR is indeed a better example of thing that makes the > build non-reproducible, we could potentially include this in the > tainted mechanism. But remember that the OVERRIDE_SRCDIR variable is also used when we use _SITE = local too, so if a package has its sources bundled besides the .mk (for example, al ot of packagers do that for very trivial packages), you would set that the build is tainted when it is not. > Though to me, this is somewhat less important: only advanced users are > likely to use <pkg>_OVERRIDE_SRCDIR, and if they do, they will > definitely realize the impact on reproducibility. I've seen people shipping into production, builds that were made locally with locally modified sources, and/or in uncomitted and/or un-pushed VCS trees. So, even "advanced users" do not always "realize" what they are doing. ;-) > While the newcomer > toying around with nodejs/npm may not realize the impact of using a > third-party package manager on reproducibility. > > So, yes, perhaps <pkg>_OVERRIDE_SRCDIR should ultimately taint the > build, but I don't see handling that as a requirement to start > introducing this "taint" flag. > > > Instead, why not provide a helper script (or enhance the existing ones > > if need be), than generated buildroot packages for those external packge > > managers? > > We really don't want thousands of npm packages I believe :-/ Yet we're ready to have thousands of python packages... Regards, Yann E. MORIN.
Angelo, Thomas, All, On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni > <thomas.petazzoni@bootlin.com> wrote: [--SNIP--] > Thomas said evrithing exceptionally well, but I would like to > underling another thing: automated building infrastructures like > continuos integration. > > If a project hasa nedd of reproducibility, the countinous integration > could check if a random developer introduced something not > reproducible and mark the build as invalid. I think this is really a > big plus of this solution. I do understand the concern, trust me, I do. What I am saying is that the solution you propose will not allow that, because there is no way to decide whether a specific .config is or is not reproducible, as per the examples I provided in the nodejs case. If a build is imprperly marked as tainted, then users will just disregard that information and never consult it, and just not use it in their automated buildsystemsd (jenkins, gitlab-ci, whatever). And even if they do have a job doing the check, that job can detect a change from "not tainted" to "tainted" because the job will always report "tainted". So, your solution does not even cover your automated builds use-case either. Regards, Yann E. MORIN.
Yann, All, On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, Thomas, All, > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > > On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni > > <thomas.petazzoni@bootlin.com> wrote: > [--SNIP--] > > Thomas said evrithing exceptionally well, but I would like to > > underling another thing: automated building infrastructures like > > continuos integration. > > > > If a project hasa nedd of reproducibility, the countinous integration > > could check if a random developer introduced something not > > reproducible and mark the build as invalid. I think this is really a > > big plus of this solution. > > I do understand the concern, trust me, I do. > > What I am saying is that the solution you propose will not allow that, > because there is no way to decide whether a specific .config is or is > not reproducible, as per the examples I provided in the nodejs case. > > If a build is imprperly marked as tainted, then users will just > disregard that information and never consult it, and just not use it in > their automated buildsystemsd (jenkins, gitlab-ci, whatever). And even > if they do have a job doing the check, that job can detect a change from > "not tainted" to "tainted" because the job will always report "tainted". My concern here is that you start from a reproducible build, add your packages right and so maintain your build reproducible, buildroot will work as before. As soon you use a package manager tainting will be signaled. Taint is mean to signal that there is a potential problem, and if you don't want to slip into it, you can always do the right thing and package your software and packaging also it's dependencies. As soon as you do this, the taint disappear. I thin it could even be a deterrent to package the software randomly! Sincerely, Angelo > > So, your solution does not even cover your automated builds use-case > either. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Angelo, All, On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > > > On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni > > > <thomas.petazzoni@bootlin.com> wrote: > > [--SNIP--] > > > Thomas said evrithing exceptionally well, but I would like to > > > underling another thing: automated building infrastructures like > > > continuos integration. > > > > > > If a project hasa nedd of reproducibility, the countinous integration > > > could check if a random developer introduced something not > > > reproducible and mark the build as invalid. I think this is really a > > > big plus of this solution. > > > > I do understand the concern, trust me, I do. > > > > What I am saying is that the solution you propose will not allow that, > > because there is no way to decide whether a specific .config is or is > > not reproducible, as per the examples I provided in the nodejs case. > > > > If a build is imprperly marked as tainted, then users will just > > disregard that information and never consult it, and just not use it in > > their automated buildsystemsd (jenkins, gitlab-ci, whatever). And even > > if they do have a job doing the check, that job can detect a change from > > "not tainted" to "tainted" because the job will always report "tainted". > > My concern here is that you start from a reproducible build, add your > packages right and so maintain your build reproducible, buildroot will > work as before. So you are, like I am, in fact arguing that we should have actual packages for such external modules? ;-) > As soon you use a package manager tainting will be > signaled. This is where I disagree. Using such package managers does not imply that the build is tainted. This is a false dichotomy. > Taint is mean to signal that there is a potential problem, and if you > don't want to slip into it, you can always do the right thing and > package your software and packaging also it's dependencies. And what I am saying is that the heuristic you suggest to decide whether a build should be considered tainted or not is incorrect. > As soon as you do this, the taint disappear. I thin it could even be a > deterrent to package the software randomly! Regards, Yann E. MORIN.
Yann, All On Sun, Sep 9, 2018 at 3:20 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > > > > On Sun, Sep 9, 2018 at 1:10 PM Thomas Petazzoni > > > > <thomas.petazzoni@bootlin.com> wrote: > > > [--SNIP--] > > > > Thomas said evrithing exceptionally well, but I would like to > > > > underling another thing: automated building infrastructures like > > > > continuos integration. > > > > > > > > If a project hasa nedd of reproducibility, the countinous integration > > > > could check if a random developer introduced something not > > > > reproducible and mark the build as invalid. I think this is really a > > > > big plus of this solution. > > > > > > I do understand the concern, trust me, I do. > > > > > > What I am saying is that the solution you propose will not allow that, > > > because there is no way to decide whether a specific .config is or is > > > not reproducible, as per the examples I provided in the nodejs case. > > > > > > If a build is imprperly marked as tainted, then users will just > > > disregard that information and never consult it, and just not use it in > > > their automated buildsystemsd (jenkins, gitlab-ci, whatever). And even > > > if they do have a job doing the check, that job can detect a change from > > > "not tainted" to "tainted" because the job will always report "tainted". > > > > My concern here is that you start from a reproducible build, add your > > packages right and so maintain your build reproducible, buildroot will > > work as before. > > So you are, like I am, in fact arguing that we should have actual packages > for such external modules? ;-) I can't see any value on having hundreds of npm packages probably outdated at wich you should add hundreds of php composer packages, go modules, cargo packages and so on. >> > As soon you use a package manager tainting will be > > signaled. > > This is where I disagree. > > Using such package managers does not imply that the build is tainted. > This is a false dichotomy. Probably it is but my experience says not, anyway, if you are dealing with a complex php software for example you can: * call php composer manually on the modules you are sure being reproducible in your .mk and live happy * use the package host-composer distributed with buildroot asking it to do a "composer install" living with the fact that it could do anything and you build will be tainted. > > Taint is mean to signal that there is a potential problem, and if you > > don't want to slip into it, you can always do the right thing and > > package your software and packaging also it's dependencies. > > And what I am saying is that the heuristic you suggest to decide whether > a build should be considered tainted or not is incorrect. It's not an heuristic, it's a rule: you ask to a package manager to resolve your dependencies, your build _could_ be tainted. You want to be sure: you write your own rule. If you write a clean mk that does everything right you shouldn't add FOO_TAINTS, that's it. I can agree with you that some packages could be considered first class citizens, the others based on package managers could not be as good as first ones. But at least, you give the tools to developers who wants to add a non trivial web package or similar to buildroot. What I'm saying here is that we don't have such a rule we simply cannot add any time soon any software based on package managers. Probably there is some other smarter way to do it that I cannot see ... I'm open to suggestions! > > > As soon as you do this, the taint disappear. I thin it could even be a > > deterrent to package the software randomly! > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Angelo, All, This is the last email I'll be sending in this thread, because it looks like the both of us are re-hashing the same arguments again and again... Yet, let me reply to parts of your email... On 2018-09-09 17:58 +0100, Angelo Compagnucci spake thusly: > On Sun, Sep 9, 2018 at 3:20 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > > > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: [--SNIP--] > > > My concern here is that you start from a reproducible build, add your > > > packages right and so maintain your build reproducible, buildroot will > > > work as before. > > > > So you are, like I am, in fact arguing that we should have actual packages > > for such external modules? ;-) > > I can't see any value on having hundreds of npm packages probably > outdated at wich you should add hundreds of php composer packages, go > modules, cargo packages and so on. Why not? We do have hundreds (~270, which literaly makes that number 'hundreds') of python packages, some (a lot?) of them probably outdated. Worse, yet: we *do* have thousands of packages, a lot of them outdated. I don't see why we would single out npm packages at all. (Besides the fact that I don't like them; I don't like perl either, but I'm still quite happy to see perl packages.) > >> > As soon you use a package manager tainting will be > > > signaled. > > > > This is where I disagree. > > > > Using such package managers does not imply that the build is tainted. > > This is a false dichotomy. > > Probably it is but my experience says not, And in my experience, it can be. If you are using an internal repository of which you control the content, then the list of nodejs modules *is* reproducible. This is not a theoretical situation, by the way; this is a real situation. Your heursitic is indeed not a heuristic, but a policy. Your policy states "any external module taints the build". My policy states "use our internal repository of trusted and known modules, anything else taints the build". So, the code currently expreses your policy, and precludes using mine, which I believe is still superior because I do not even rely on external data which may disapear at the whim of an external entity; everything I need is already internal and stable. > anyway, if you are dealing > with a complex php software for example you can: > * call php composer manually on the modules you are sure being > reproducible in your .mk and live happy I have no idea what php-composer is. I guess it would download and install php files, not unlike npm does, and probably looks as ugly as npm. > * use the package host-composer distributed with buildroot asking it > to do a "composer install" living with the fact that it could do > anything and you build will be tainted. > > > > Taint is mean to signal that there is a potential problem, and if you > > > don't want to slip into it, you can always do the right thing and > > > package your software and packaging also it's dependencies. > > > > And what I am saying is that the heuristic you suggest to decide whether > > a build should be considered tainted or not is incorrect. > > It's not an heuristic, it's a rule: you ask to a package manager to > resolve your dependencies, your build _could_ be tainted. You want to > be sure: you write your own rule. So, because the build _could_ be tainted, you want to make all builds as tainted? That's insane... > If you write a clean mk that does everything right you shouldn't add > FOO_TAINTS, that's it. > > I can agree with you that some packages could be considered first > class citizens, the others based on package managers could not be as > good as first ones. But at least, you give the tools to developers who > wants to add a non trivial web package or similar to buildroot. Well, the tainted stuff is basically orthogonal to the package manager stuff, I think. Even if I despise the package managers, I can see that people are so keen on using them... What I am arguing in this thread, is that the tainted status can *not* be deduced from the configuratioe tainted stuff is basically orthogonal to the package manager stuff, I think. Even if I despise the package managers, I can see that people are so keen on using them... What I am arguing in this thread, is that the tainted status can *not* be deduced from the configuration. The way you are currently coding the tainted flag would break for the people that are already doing it right, and for whom this tainted flag would ultimately prove to be the most valuable. Regards, Yann E. MORIN. > What I'm saying here is that we don't have such a rule we simply > cannot add any time soon any software based on package managers. > > Probably there is some other smarter way to do it that I cannot see ... > > I'm open to suggestions! > > > > > > As soon as you do this, the taint disappear. I thin it could even be a > > > deterrent to package the software randomly! > > > > Regards, > > Yann E. MORIN. > > > > -- > > .-----------------.--------------------.------------------.--------------------. > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > '------------------------------^-------^------------------^--------------------'
Yann, All, On Sun, Sep 9, 2018 at 7:55 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > This is the last email I'll be sending in this thread, because it looks > like the both of us are re-hashing the same arguments again and again... > Yet, let me reply to parts of your email... Last one, I promise! > On 2018-09-09 17:58 +0100, Angelo Compagnucci spake thusly: > > On Sun, Sep 9, 2018 at 3:20 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > > > On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > > > > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > [--SNIP--] > > > > My concern here is that you start from a reproducible build, add your > > > > packages right and so maintain your build reproducible, buildroot will > > > > work as before. > > > > > > So you are, like I am, in fact arguing that we should have actual packages > > > for such external modules? ;-) > > > > I can't see any value on having hundreds of npm packages probably > > outdated at wich you should add hundreds of php composer packages, go > > modules, cargo packages and so on. > > Why not? We do have hundreds (~270, which literaly makes that number > 'hundreds') of python packages, some (a lot?) of them probably outdated. > > Worse, yet: we *do* have thousands of packages, a lot of them outdated. > > I don't see why we would single out npm packages at all. IMHO I think we should start do it differently. Even the major distributions started to not packaging everything anymore. Debian/Ubuntu have a really low number of npm packages or php ones. Owncloud homepage clearly states to not use the distributions packaged version cause it's outdated and insecure. Offering ton of outdated and insecure packages by default it's not on the best of interest of anyone. > (Besides the fact that I don't like them; I don't like perl either, but > I'm still quite happy to see perl packages.) > > > >> > As soon you use a package manager tainting will be > > > > signaled. > > > > > > This is where I disagree. > > > > > > Using such package managers does not imply that the build is tainted. > > > This is a false dichotomy. > > > > Probably it is but my experience says not, > > And in my experience, it can be. > > If you are using an internal repository of which you control the > content, then the list of nodejs modules *is* reproducible. This is > not a theoretical situation, by the way; this is a real situation. > > Your heursitic is indeed not a heuristic, but a policy. Your policy > states "any external module taints the build". My policy states "use > our internal repository of trusted and known modules, anything else > taints the build". > > So, the code currently expreses your policy, and precludes using mine, > which I believe is still superior because I do not even rely on external > data which may disapear at the whim of an external entity; everything I > need is already internal and stable. That's the point I can't understand. What precludes you to use the workflow you used until now to add a package in buildroot? Probably you don't use a package manager and after this series nothing changes for you or for all the people that uses an internal repository. If you have an internal repository with all of your dependencies mirrored you don't need to use a package manager, else you can call npm install or composer install or whatever else manually. > > anyway, if you are dealing > > with a complex php software for example you can: > > * call php composer manually on the modules you are sure being > > reproducible in your .mk and live happy > > I have no idea what php-composer is. I guess it would download and > install php files, not unlike npm does, and probably looks as ugly as > npm. That's the point. For each programming language there are at least 3/4 major package managers that do the same thing in a more or less different way. Or we cope with this, or we should state clearly in the manual that buildroot is not usable on that scenarios. > > > * use the package host-composer distributed with buildroot asking it > > to do a "composer install" living with the fact that it could do > > anything and you build will be tainted. > > > > > > Taint is mean to signal that there is a potential problem, and if you > > > > don't want to slip into it, you can always do the right thing and > > > > package your software and packaging also it's dependencies. > > > > > > And what I am saying is that the heuristic you suggest to decide whether > > > a build should be considered tainted or not is incorrect. > > > > It's not an heuristic, it's a rule: you ask to a package manager to > > resolve your dependencies, your build _could_ be tainted. You want to > > be sure: you write your own rule. > > So, because the build _could_ be tainted, you want to make all builds as > tainted? That's insane... All builds that uses a package manager that not ensure your dependency chain is clean. If it's clean, non need to taint. Taiting is an explicit property of a package after all. This doesn't look so insane to me after all ... > > > If you write a clean mk that does everything right you shouldn't add > > FOO_TAINTS, that's it. > > > > I can agree with you that some packages could be considered first > > class citizens, the others based on package managers could not be as > > good as first ones. But at least, you give the tools to developers who > > wants to add a non trivial web package or similar to buildroot. > > Well, the tainted stuff is basically orthogonal to the package manager > stuff, I think. Even if I despise the package managers, I can see that > people are so keen on using them... > What I am arguing in this thread, is that the tainted status can *not* > be deduced from the configuration. The way you are currently coding the > tainted flag would break for the people that are already doing it right, > and for whom this tainted flag would ultimately prove to be the most > valuable. Honestly I can't follow you here. If people are doing right why the heck the should be worried by the tainting? Npm package in buildroot is plain wrong right now. It should be decoupled from nodejs and made a host package. After that in your package you can do: * Call the host-npm on your reproducible modules and live happy (you current behavior is maintained) * Use the host-npm package on your non-reproducible modules and add a FOO_TAINTS to your package I have a host-glide package manager for golang that works this way and it is pure awesome! If not, what you propose? Denying package managers completely? Sincerely, Angelo > Regards, > Yann E. MORIN. > > > What I'm saying here is that we don't have such a rule we simply > > cannot add any time soon any software based on package managers. > > > > Probably there is some other smarter way to do it that I cannot see ... > > > > I'm open to suggestions! > > > > > > > > > As soon as you do this, the taint disappear. I thin it could even be a > > > > deterrent to package the software randomly! > > > > > > Regards, > > > Yann E. MORIN. > > > > > > -- > > > .-----------------.--------------------.------------------.--------------------. > > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > > '------------------------------^-------^------------------^--------------------' > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Yann, All, On Sun, Sep 9, 2018 at 10:18 PM Angelo Compagnucci <angelo@amarulasolutions.com> wrote: > > Yann, All, > On Sun, Sep 9, 2018 at 7:55 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > Angelo, All, > > > > This is the last email I'll be sending in this thread, because it looks > > like the both of us are re-hashing the same arguments again and again... > > Yet, let me reply to parts of your email... > > Last one, I promise! > > > On 2018-09-09 17:58 +0100, Angelo Compagnucci spake thusly: > > > On Sun, Sep 9, 2018 at 3:20 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > > > > > On 2018-09-09 14:44 +0100, Angelo Compagnucci spake thusly: > > > > > On Sun, Sep 9, 2018 at 2:33 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > > > On 2018-09-09 13:25 +0100, Angelo Compagnucci spake thusly: > > [--SNIP--] > > > > > My concern here is that you start from a reproducible build, add your > > > > > packages right and so maintain your build reproducible, buildroot will > > > > > work as before. > > > > > > > > So you are, like I am, in fact arguing that we should have actual packages > > > > for such external modules? ;-) > > > > > > I can't see any value on having hundreds of npm packages probably > > > outdated at wich you should add hundreds of php composer packages, go > > > modules, cargo packages and so on. > > > > Why not? We do have hundreds (~270, which literaly makes that number > > 'hundreds') of python packages, some (a lot?) of them probably outdated. > > > > Worse, yet: we *do* have thousands of packages, a lot of them outdated. > > > > I don't see why we would single out npm packages at all. > > IMHO I think we should start do it differently. Even the major > distributions started to not packaging everything anymore. > Debian/Ubuntu have a really low number of npm packages or php ones. > Owncloud homepage clearly states to not use the distributions packaged > version cause it's outdated and insecure. > > Offering ton of outdated and insecure packages by default it's not on > the best of interest of anyone. > > > (Besides the fact that I don't like them; I don't like perl either, but > > I'm still quite happy to see perl packages.) > > > > > >> > As soon you use a package manager tainting will be > > > > > signaled. > > > > > > > > This is where I disagree. > > > > > > > > Using such package managers does not imply that the build is tainted. > > > > This is a false dichotomy. > > > > > > Probably it is but my experience says not, > > > > And in my experience, it can be. > > > > If you are using an internal repository of which you control the > > content, then the list of nodejs modules *is* reproducible. This is > > not a theoretical situation, by the way; this is a real situation. > > > > Your heursitic is indeed not a heuristic, but a policy. Your policy > > states "any external module taints the build". My policy states "use > > our internal repository of trusted and known modules, anything else > > taints the build". > > > > So, the code currently expreses your policy, and precludes using mine, > > which I believe is still superior because I do not even rely on external > > data which may disapear at the whim of an external entity; everything I > > need is already internal and stable. > > That's the point I can't understand. What precludes you to use the > workflow you used until now to add a package in buildroot? > Probably you don't use a package manager and after this series nothing > changes for you or for all the people that uses an internal > repository. > If you have an internal repository with all of your dependencies > mirrored you don't need to use a package manager, else you can call > npm install or composer install or whatever else manually. > > > > anyway, if you are dealing > > > with a complex php software for example you can: > > > * call php composer manually on the modules you are sure being > > > reproducible in your .mk and live happy > > > > I have no idea what php-composer is. I guess it would download and > > install php files, not unlike npm does, and probably looks as ugly as > > npm. > > That's the point. For each programming language there are at least 3/4 > major package managers that do the same thing in a more or less > different way. Or we cope with this, or we should state clearly in the > manual that buildroot is not usable on that scenarios. > > > > > > * use the package host-composer distributed with buildroot asking it > > > to do a "composer install" living with the fact that it could do > > > anything and you build will be tainted. > > > > > > > > Taint is mean to signal that there is a potential problem, and if you > > > > > don't want to slip into it, you can always do the right thing and > > > > > package your software and packaging also it's dependencies. > > > > > > > > And what I am saying is that the heuristic you suggest to decide whether > > > > a build should be considered tainted or not is incorrect. > > > > > > It's not an heuristic, it's a rule: you ask to a package manager to > > > resolve your dependencies, your build _could_ be tainted. You want to > > > be sure: you write your own rule. > > > > So, because the build _could_ be tainted, you want to make all builds as > > tainted? That's insane... > > All builds that uses a package manager that not ensure your dependency > chain is clean. If it's clean, non need to taint. Taiting is an > explicit property of a package after all. > This doesn't look so insane to me after all ... > > > > > > If you write a clean mk that does everything right you shouldn't add > > > FOO_TAINTS, that's it. > > > > > > I can agree with you that some packages could be considered first > > > class citizens, the others based on package managers could not be as > > > good as first ones. But at least, you give the tools to developers who > > > wants to add a non trivial web package or similar to buildroot. > > > > Well, the tainted stuff is basically orthogonal to the package manager > > stuff, I think. Even if I despise the package managers, I can see that > > people are so keen on using them... > > > What I am arguing in this thread, is that the tainted status can *not* > > be deduced from the configuration. The way you are currently coding the > > tainted flag would break for the people that are already doing it right, > > and for whom this tainted flag would ultimately prove to be the most > > valuable. > > Honestly I can't follow you here. If people are doing right why the > heck the should be worried by the tainting? > > Npm package in buildroot is plain wrong right now. It should be > decoupled from nodejs and made a host package. > After that in your package you can do: > * Call the host-npm on your reproducible modules and live happy (you > current behavior is maintained) > * Use the host-npm package on your non-reproducible modules and add a > FOO_TAINTS to your package > > I have a host-glide package manager for golang that works this way and > it is pure awesome! I just pushed o POC on my local branch if you want to have a look. It's what I mean for tainting applied to a package with a more robust ad correct approach that the npm case: https://github.com/angeloc/buildroot/commits/watchtower Sincerely, Angelo > > If not, what you propose? Denying package managers completely? > > Sincerely, Angelo > > > Regards, > > Yann E. MORIN. > > > > > What I'm saying here is that we don't have such a rule we simply > > > cannot add any time soon any software based on package managers. > > > > > > Probably there is some other smarter way to do it that I cannot see ... > > > > > > I'm open to suggestions! > > > > > > > > > > > > As soon as you do this, the taint disappear. I thin it could even be a > > > > > deterrent to package the software randomly! > > > > > > > > Regards, > > > > Yann E. MORIN. > > > > > > > > -- > > > > .-----------------.--------------------.------------------.--------------------. > > > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > > > '------------------------------^-------^------------------^--------------------' > > > > -- > > .-----------------.--------------------.------------------.--------------------. > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > '------------------------------^-------^------------------^--------------------'
Angelo, All, On 2018-09-10 09:50 +0200, Angelo Compagnucci spake thusly: > > Last one, I promise! [--SNIP--] > I just pushed o POC on my local branch if you want to have a look. > It's what I mean for tainting applied to a package with a more robust > ad correct approach that the npm case: > https://github.com/angeloc/buildroot/commits/watchtower So, here is my last stance on the subject, in which I tried to summarise my position. Why would the "tainted" flag be set? - unknown licensing information: it is better to use the existing licensing infrastructure, which is made for this very purpose: FOO_LICENSE := $(FOO_LICENSE), Unknown (external data used) - non-reproducible packages in Buildroot: I don't think we should accept packages in Buildroot that would taint the build; and if we were, we could hide them behind !BR2_REPRODUCIBLE (with or without a comment stating "foo is not reproducible"); - packages that are in a br2-external, or in a private fork: I believe that people who do non-reproducible packages either don't care, have no choice, or both. If they did care, they would not create tainting packages; if they did care but still had no choice, they could also decide to hide them behind !BR2_REPRODUCIBLE; - packages with a list of external resources, like we have for nodejs/npm: we can't know if that list references reproducible resources or not. That last point is very important: there *are* people that do care about the reproduciblity of a build, and thus have already taken care that BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL *does* point to stable and reproducible set of nodejs modules [0]. Patch 3 in the series would mark for them that the build is tainted when it is not; since those people do care, the tainted flag would be most important to them, but by always marking their build as tainted, the flag becomes useless to the very people that do care... Yes, a lot of people will just use a non-stable list, and they even probably do not care either. I do not want to have to impose that limitation unto those who know what they are doing. So, I still conclude that we do not need to have a tainted flag. [0] For example, consider an explicit and complete list such as (spitted for readability): BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL=" http://internal-server/node/mod-1 http://internal-server/node/mod-2 http://internal-server/node/mod-3 http://internal-server/node/mod-4 " and that all the dependencies of those modules *are* present in that list, leaving npm no chance to download anything. Regards, Yann E. MORIN.
Angelo, All,
On 2018-09-10 17:00 +0200, Yann E. MORIN spake thusly:
> [0] For example, consider an explicit and complete list such as (spitted
Woops... s/spitted/splitted/
Regards,
Yann E. MORIN.
Yann, All, Il giorno lun 10 set 2018 alle ore 17:00 Yann E. MORIN <yann.morin.1998@free.fr> ha scritto: > > Angelo, All, > > On 2018-09-10 09:50 +0200, Angelo Compagnucci spake thusly: > > > Last one, I promise! > [--SNIP--] > > I just pushed o POC on my local branch if you want to have a look. > > It's what I mean for tainting applied to a package with a more robust > > ad correct approach that the npm case: > > https://github.com/angeloc/buildroot/commits/watchtower > > So, here is my last stance on the subject, in which I tried to summarise > my position. > > Why would the "tainted" flag be set? > > - unknown licensing information: it is better to use the existing > licensing infrastructure, which is made for this very purpose: > FOO_LICENSE := $(FOO_LICENSE), Unknown (external data used) > > - non-reproducible packages in Buildroot: I don't think we should > accept packages in Buildroot that would taint the build; and if we > were, we could hide them behind !BR2_REPRODUCIBLE (with or without > a comment stating "foo is not reproducible"); > > - packages that are in a br2-external, or in a private fork: I believe > that people who do non-reproducible packages either don't care, > have no choice, or both. If they did care, they would not create > tainting packages; if they did care but still had no choice, they > could also decide to hide them behind !BR2_REPRODUCIBLE; > > - packages with a list of external resources, like we have for > nodejs/npm: we can't know if that list references reproducible > resources or not. > > That last point is very important: there *are* people that do care about > the reproduciblity of a build, and thus have already taken care that > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL *does* point to stable and > reproducible set of nodejs modules [0]. Patch 3 in the series would mark > for them that the build is tainted when it is not; since those people do > care, the tainted flag would be most important to them, but by always > marking their build as tainted, the flag becomes useless to the very > people that do care... > > Yes, a lot of people will just use a non-stable list, and they even > probably do not care either. I do not want to have to impose that > limitation unto those who know what they are doing. > > So, I still conclude that we do not need to have a tainted flag. I can agree with you, but what do you propose for handling package managers: Simply put a !BR2_REPRODUCIBLE for a package that needs dependencies resolved and cannot handle dependencies cleanly? > > [0] For example, consider an explicit and complete list such as (spitted > for readability): > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL=" > http://internal-server/node/mod-1 > http://internal-server/node/mod-2 > http://internal-server/node/mod-3 > http://internal-server/node/mod-4 > " > and that all the dependencies of those modules *are* present in that > list, leaving npm no chance to download anything. As I said previously, hiding npm inside nodejs is wrong. What if you have a couple nodejs based software and for one you have a clear picture of dependencies but not for the other? Npm should decoupled and used each time a nodejs software needs it. This is what I did with watchtower: +################################################################################ +# +# watchtower +# +################################################################################ + +WATCHTOWER_VERSION = v0.3.0 +WATCHTOWER_SITE = $(call github,v2tec,watchtower,$(WATCHTOWER_VERSION)) +WATCHTOWER_LICENSE = Apache-2.0 +WATCHTOWER_LICENSE_FILES = LICENSE.md +WATCHTOWER_DEPENDENCIES = host-glide +WATCHTOWER_TAINTS = YES + +define WATCHTOWER_RESOLVE_DEPS +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install +endef + +WATCHTOWER_POST_CONFIGURE_HOOKS = WATCHTOWER_RESOLVE_DEPS + +$(eval $(golang-package)) This clearly mark watchtower as a tainting package, but it doesn't harm any other package that could use host-glide. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Angelo, All, On 2018-09-10 19:10 +0200, Angelo Compagnucci spake thusly: > Il giorno lun 10 set 2018 alle ore 17:00 Yann E. MORIN > <yann.morin.1998@free.fr> ha scritto: > I can agree with you, but what do you propose for handling package > managers: Simply put a !BR2_REPRODUCIBLE for a package that needs > dependencies resolved and cannot handle dependencies cleanly? Yes, see below... > > [0] For example, consider an explicit and complete list such as (spitted > > for readability): > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL=" > > http://internal-server/node/mod-1 > > http://internal-server/node/mod-2 > > http://internal-server/node/mod-3 > > http://internal-server/node/mod-4 > > " > > and that all the dependencies of those modules *are* present in that > > list, leaving npm no chance to download anything. > > As I said previously, hiding npm inside nodejs is wrong. What if you > have a couple nodejs based software and for one you have a clear > picture of dependencies but not for the other? > Npm should decoupled and used each time a nodejs software needs it. npm (the utility) comes from nodejs (the source tree). So, the npm we use in Buildroot comes from host-nodejs (the Buildroot package). Thus, the package that calls npm (i.e. nodejs today) is already decorelated from the package that installs npm. And you can already use npm from a generic-package in Buildroot as it is today... There are two ways to install stuff with npm in Buildroot: - add a pacakge name (URL, name and version, or just name) in the existing BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL config option; - add a new generic-package that calls npm as part of its _INSTALL_CMDS. In either case you may or may not have reproducibility. In the former, Buildroot can not decide either way, because only a human can tell, so your patch 3 can't be applied, because it is wring. As for the latter case, the human was in charge of writing the .mk and Config.in. Whether they write TAINT=YES in the .mk, or depends on !BR2_REPRODUCIBLE in the Config.in has the same effect. As we already have BR2_REPRODUCIBLE, we don't need TAINT. Or I don't understand the problem... > This is what I did with watchtower: [--SNIP--] > +WATCHTOWER_TAINTS = YES If you are going so far as to explicitly write this line, just hide the package behing BR2_REPRODUCIBLE instead, yes. And even better yet, for packages we would include in Buildroot, provide a clean and reproducible list of packages for the dependencies for those packages. Although this would be the ideal situation, if that is not possible, then hide behind !BR2_REPRODUCIBLE. For packages that stay in a br2-external or a random fork elsewhere, I don't think we should care, as I explained previously. For a parallel: if I do a build without threads, then packages than need threads are hidden (with a comment). Similarly, if I want a reproducible build, the packages that can't be reproducible are hidden (with a comment). By the way, this has nothing to do with whether they are packages coming from a "package manager". Even "traditional" packages may want to do unclean downloads at build time; for example, some care had to be taken for asterisk, or it would download some sound stuff during the install step (although that was reproducible). > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install What would prevent having a script like we have in utils/scancpan or utils/scanpypi to do the same processing for Go packages? Or npm modules? Yes, some people are afraid we get thousands of them. But we don't know how far we'll go. Today, we're pretty happy adding random python modules. Why would it be different for the other "package managers"? If we find that this is going "way too far", we can revisit that later. Regards, Yann E. MORIN.
Yann, All, On Mon, Sep 10, 2018 at 7:07 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-10 19:10 +0200, Angelo Compagnucci spake thusly: > > Il giorno lun 10 set 2018 alle ore 17:00 Yann E. MORIN > > <yann.morin.1998@free.fr> ha scritto: > > I can agree with you, but what do you propose for handling package > > managers: Simply put a !BR2_REPRODUCIBLE for a package that needs > > dependencies resolved and cannot handle dependencies cleanly? > > Yes, see below... > > > > [0] For example, consider an explicit and complete list such as (spitted > > > for readability): > > > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL=" > > > http://internal-server/node/mod-1 > > > http://internal-server/node/mod-2 > > > http://internal-server/node/mod-3 > > > http://internal-server/node/mod-4 > > > " > > > and that all the dependencies of those modules *are* present in that > > > list, leaving npm no chance to download anything. > > > > As I said previously, hiding npm inside nodejs is wrong. What if you > > have a couple nodejs based software and for one you have a clear > > picture of dependencies but not for the other? > > Npm should decoupled and used each time a nodejs software needs it. > > npm (the utility) comes from nodejs (the source tree). So, the npm we > use in Buildroot comes from host-nodejs (the Buildroot package). Thus, > the package that calls npm (i.e. nodejs today) is already decorelated > from the package that installs npm. > > And you can already use npm from a generic-package in Buildroot as it is > today... > > There are two ways to install stuff with npm in Buildroot: > > - add a pacakge name (URL, name and version, or just name) in the > existing BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL config option; > > - add a new generic-package that calls npm as part of its > _INSTALL_CMDS. > > In either case you may or may not have reproducibility. In the former, > Buildroot can not decide either way, because only a human can tell, so > your patch 3 can't be applied, because it is wring. > > As for the latter case, the human was in charge of writing the .mk and > Config.in. Whether they write TAINT=YES in the .mk, or depends on > !BR2_REPRODUCIBLE in the Config.in has the same effect. As we already > have BR2_REPRODUCIBLE, we don't need TAINT. > > Or I don't understand the problem... > > > This is what I did with watchtower: > [--SNIP--] > > +WATCHTOWER_TAINTS = YES > > If you are going so far as to explicitly write this line, just hide the > package behing BR2_REPRODUCIBLE instead, yes. Ok, I'm sold. I'm not interested on how we do it, it's way more important that we define a way to add packages based on package managers. > And even better yet, for packages we would include in Buildroot, provide > a clean and reproducible list of packages for the dependencies for those > packages. Although this would be the ideal situation, if that is not > possible, then hide behind !BR2_REPRODUCIBLE. Yes it could, only if the dependency chain is not nested and it easy to walk. That's not always true. > For packages that stay in a br2-external or a random fork elsewhere, I > don't think we should care, as I explained previously. > > For a parallel: if I do a build without threads, then packages than need > threads are hidden (with a comment). Similarly, if I want a reproducible > build, the packages that can't be reproducible are hidden (with a > comment). > > By the way, this has nothing to do with whether they are packages coming > from a "package manager". Even "traditional" packages may want to do > unclean downloads at build time; for example, some care had to be taken > for asterisk, or it would download some sound stuff during the install > step (although that was reproducible). > > > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install > > What would prevent having a script like we have in utils/scancpan or > utils/scanpypi to do the same processing for Go packages? Or npm > modules? This is not possible imho, at least fo golang packages. Golang packages is basically git repositories, we could end up packaging a good part of github! > Yes, some people are afraid we get thousands of them. But we don't know > how far we'll go. Today, we're pretty happy adding random python modules. > Why would it be different for the other "package managers"? Well, the main difference is that python packages is self contained, npm packages for example not, they can depend on other modules in a very intricate way. You should write something that recursively walks all the dependency tree and outputs a package for everything. Probably we should only embrace the shift paradigm of having packages that downloads their own dependencies. There is also another reason why python packages are added frequently: we have no way to call a host-pip to resolve dependencies. I'm sure that there would be way less python packages if we had a host-pip package. Sincerely, Angelo > If we find that this is going "way too far", we can revisit that later. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Hello, On Mon, 10 Sep 2018 20:07:05 +0200, Yann E. MORIN wrote: > If you are going so far as to explicitly write this line, just hide the > package behing BR2_REPRODUCIBLE instead, yes. I think BR2_REPRODUCIBLE is not the same thing. BR2_REPRODUCIBLE means "assuming we have the same source code and configuration, we guarantee that the binary results will be bit-to-bit identical". While the reproducibility issue we are talking about in this thread is about having the same source code each time the build is done. It's obviously a requirement for BR2_REPRODUCIBLE to work, but the fact that we build with the same source code each time the build is done is a property of Buildroot regardless of whether BR2_REPRODUCIBLE is enabled or not. So I'm sorry, but this suggestion to hide a package behind BR2_REPRODUCIBLE because the *source* is not guaranteed to be the same each time is not a good suggestion IMO, as it's confusing two different "reproducible" properties. Best regards, Thomas
Angelo, All, On 2018-09-10 20:17 +0100, Angelo Compagnucci spake thusly: > On Mon, Sep 10, 2018 at 7:07 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: [--SNIP--] > > > +WATCHTOWER_TAINTS = YES > > If you are going so far as to explicitly write this line, just hide the > > package behing BR2_REPRODUCIBLE instead, yes. > Ok, I'm sold. OK, thanks. Should someone mark this series as rejected in patchwork, then? > > > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install > > What would prevent having a script like we have in utils/scancpan or > > utils/scanpypi to do the same processing for Go packages? Or npm > > modules? > > This is not possible imho, at least fo golang packages. Golang > packages is basically git repositories, we could end up packaging a > good part of github! > > > Yes, some people are afraid we get thousands of them. But we don't know > > how far we'll go. Today, we're pretty happy adding random python modules. > > Why would it be different for the other "package managers"? > > Well, the main difference is that python packages is self contained, Hmm. Not so much: python packages do have dependencies onto other python packages. > npm packages for example not, they can depend on other modules in a > very intricate way. > You should write something that recursively walks all the dependency > tree and outputs a package for everything. Probably we should only > embrace the shift paradigm of having packages that downloads their own > dependencies. Now we're going somewhere, I believe. Would you expect that we could write something like (roughly): $ cat package/foo/Config.in config BR2_PACKAGE_FOO bool "foo" depends on !BR2_REPRODUCIBLE select SOME_PACKAGE $ cat package/foo/foo.mk FOO_VERSION = 1.2.3 FOO_DEPENDENCIES = some dependencies on other BR packages $(eval $(npm-package)) and that the npm-package infra would do the call and install whatever foo requires? Optionally, if the dependencies are already installed (e.g. because of a FOO_DEPENDENCIES or another previous npm-package already installed), then some dependencies may or may not be downloaded by npm. That would be fine by me. There is one gotcha though: this won't work for purely off-line builds. This can be a big limitation in some setups, where access to an external network is not always guaranteed, so we should carefully say so in the manual. > There is also another reason why python packages are added frequently: > we have no way to call a host-pip to resolve dependencies. I'm sure > that there would be way less python packages if we had a host-pip > package. Well, nothing prevents one from proposing a pip-package infra, like the npm-package I hinted at above. ;-) I may even offer some review of those infras. Not that I would be interested in using those infras, mind you... ;-] Regards, Yann E. MORIN.
Thomas, All, On Mon, Sep 10, 2018 at 8:38 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > Hello, > > On Mon, 10 Sep 2018 20:07:05 +0200, Yann E. MORIN wrote: > > > If you are going so far as to explicitly write this line, just hide the > > package behing BR2_REPRODUCIBLE instead, yes. > > I think BR2_REPRODUCIBLE is not the same thing. BR2_REPRODUCIBLE means > "assuming we have the same source code and configuration, we guarantee > that the binary results will be bit-to-bit identical". > > While the reproducibility issue we are talking about in this thread is > about having the same source code each time the build is done. It's > obviously a requirement for BR2_REPRODUCIBLE to work, but the fact that > we build with the same source code each time the build is done is a > property of Buildroot regardless of whether BR2_REPRODUCIBLE is enabled > or not. > > So I'm sorry, but this suggestion to hide a package behind > BR2_REPRODUCIBLE because the *source* is not guaranteed to be the same > each time is not a good suggestion IMO, as it's confusing two > different "reproducible" properties. I agree with you and indeed this patch series born after our discussion at the Buildroot developers meeting. I think we should find a common ground on this ... > > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
Yann, All, On Mon, Sep 10, 2018 at 8:44 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-10 20:17 +0100, Angelo Compagnucci spake thusly: > > On Mon, Sep 10, 2018 at 7:07 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > [--SNIP--] > > > > +WATCHTOWER_TAINTS = YES > > > If you are going so far as to explicitly write this line, just hide the > > > package behing BR2_REPRODUCIBLE instead, yes. > > Ok, I'm sold. > > OK, thanks. > > Should someone mark this series as rejected in patchwork, then? > > > > > +cd $(WATCHTOWER_SRC_PATH) && GOPATH="$(@D)/$(WATCHTOWER_WORKSPACE)" $(HOST_DIR)/usr/bin/glide install > > > What would prevent having a script like we have in utils/scancpan or > > > utils/scanpypi to do the same processing for Go packages? Or npm > > > modules? > > > > This is not possible imho, at least fo golang packages. Golang > > packages is basically git repositories, we could end up packaging a > > good part of github! > > > > > Yes, some people are afraid we get thousands of them. But we don't know > > > how far we'll go. Today, we're pretty happy adding random python modules. > > > Why would it be different for the other "package managers"? > > > > Well, the main difference is that python packages is self contained, > > Hmm. Not so much: python packages do have dependencies onto other python > packages. > > > npm packages for example not, they can depend on other modules in a > > very intricate way. > > You should write something that recursively walks all the dependency > > tree and outputs a package for everything. Probably we should only > > embrace the shift paradigm of having packages that downloads their own > > dependencies. > > Now we're going somewhere, I believe. > > Would you expect that we could write something like (roughly): > > $ cat package/foo/Config.in > config BR2_PACKAGE_FOO > bool "foo" > depends on !BR2_REPRODUCIBLE > select SOME_PACKAGE > > $ cat package/foo/foo.mk > FOO_VERSION = 1.2.3 > FOO_DEPENDENCIES = some dependencies on other BR packages > $(eval $(npm-package)) > > and that the npm-package infra would do the call and install whatever > foo requires? Optionally, if the dependencies are already installed > (e.g. because of a FOO_DEPENDENCIES or another previous npm-package > already installed), then some dependencies may or may not be downloaded > by npm. I'm not into this honestly. What I want to achieve is that a package can call a npm install or a glide install on whatever dependencies file the source code distributes. Without the need to open that file, understanding the dependency chain and reflect that in buildroot. I think that the original developer knows better the dependencies of his software and we should trust the building infrastructure he choose, even more if what he choose is the best practice for that programming language. Exactly As I did for the example I posted before. > That would be fine by me. > > There is one gotcha though: this won't work for purely off-line builds. > This can be a big limitation in some setups, where access to an external > network is not always guaranteed, so we should carefully say so in the > manual. Not true. A package can call the package manager in it's post download hook, so a make source should do an offline build without troubles. > > There is also another reason why python packages are added frequently: > > we have no way to call a host-pip to resolve dependencies. I'm sure > > that there would be way less python packages if we had a host-pip > > package. > > Well, nothing prevents one from proposing a pip-package infra, like the > npm-package I hinted at above. ;-) I may even offer some review of those > infras. Yes, if we can agree on a way to add package managers to buildroot! > Not that I would be interested in using those infras, mind you... ;-] Great! > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Angelo, All, On 2018-09-10 21:03 +0100, Angelo Compagnucci spake thusly: > On Mon, Sep 10, 2018 at 8:44 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > $ cat package/foo/Config.in > > config BR2_PACKAGE_FOO > > bool "foo" > > depends on !BR2_REPRODUCIBLE > > select SOME_PACKAGE > > > > $ cat package/foo/foo.mk > > FOO_VERSION = 1.2.3 > > FOO_DEPENDENCIES = some dependencies on other BR packages > > $(eval $(npm-package)) > > > > and that the npm-package infra would do the call and install whatever > > foo requires? Optionally, if the dependencies are already installed > > (e.g. because of a FOO_DEPENDENCIES or another previous npm-package > > already installed), then some dependencies may or may not be downloaded > > by npm. > > I'm not into this honestly. What I want to achieve is that a package > can call a npm install or a glide install on whatever dependencies > file the source code distributes. Without the need to open that file, > understanding the dependency chain and reflect that in buildroot. > > I think that the original developer knows better the dependencies of > his software and we should trust the building infrastructure he > choose, even more if what he choose is the best practice for that > programming language. > > Exactly As I did for the example I posted before. And this is exactly what the npm-package (or glide-package, or whatever) would end up doing: expand to FOO_BUILD_CMDS, FOO_INSTALL_CMDS et al. that do the job, instead of repeating it in every package, like the autotools-package infra allows to not repeat the .configure; make; make install dance in all the autotools-based package. For example, your watchtower would be writen as: WATCHTOWER_VERSION = v0.3.0 WATCHTOWER_LICENSE = Apache-2.0 WATCHTOWER_LICENSE_FILES = LICENSE.md $(eval $(glide-package)) And then the glide-pacakge infra would be defined sdmething like: define glide-pacakge-inner $(2)_DEPENDENCIES += host-glide define $(2)_GLIDE cd $$($(2)_SRC_PATH); GOPATH=$$($(@)_GOPATH) $$(HOST_DIR)/usr/bin/glide install endef $(2)_POST_CONFIIGURE_HOOKS += $(2)_GLIDE $(call golang-package,$(1),$(2),$(3),$(4)) endef Or something similar, like we do for the rebar-package infra, for example, which piggy-backs onto either the generic-package or the autotools-package infra as needed. > > That would be fine by me. > > > > There is one gotcha though: this won't work for purely off-line builds. > > This can be a big limitation in some setups, where access to an external > > network is not always guaranteed, so we should carefully say so in the > > manual. > > Not true. A package can call the package manager in it's post download > hook, so a make source should do an offline build without troubles. But then said package manager would do only the download in the download step? I.e. I want to be able to do on 'somemachine' which has acces to both the internal and external networks: $ export BR2_DL_DIR=/some/location $ make meh_defconfig $ make source $ scp -r /some/location someone@someserver:/somewhere And on 'someserver' which does not have access to the network at all (by a firewall that blocks out-going connections): $ export BR2_DL_DIR=/somewhere $ make meh_defconfig $ make This is what I call an off-line build. > Yes, if we can agree on a way to add package managers to buildroot! I don't see any technical limitation to that, given the example you provided. Of course, I guess it is just a trivial example, and that real life is little bit more tricky as usual, but there is nothing (besides my own stubbornness, that is) standing in the road, is there? Regards, Yann E. MORIN.
Thomas, All, On 2018-09-10 21:37 +0200, Thomas Petazzoni spake thusly: > On Mon, 10 Sep 2018 20:07:05 +0200, Yann E. MORIN wrote: > > If you are going so far as to explicitly write this line, just hide the > > package behing BR2_REPRODUCIBLE instead, yes. > I think BR2_REPRODUCIBLE is not the same thing. BR2_REPRODUCIBLE means > "assuming we have the same source code and configuration, we guarantee > that the binary results will be bit-to-bit identical". > > While the reproducibility issue we are talking about in this thread is > about having the same source code each time the build is done. It's > obviously a requirement for BR2_REPRODUCIBLE to work, but the fact that > we build with the same source code each time the build is done is a > property of Buildroot regardless of whether BR2_REPRODUCIBLE is enabled > or not. > > So I'm sorry, but this suggestion to hide a package behind > BR2_REPRODUCIBLE because the *source* is not guaranteed to be the same > each time is not a good suggestion IMO, as it's confusing two > different "reproducible" properties. But if you select BR2_REPRODUCIBLE, do you accept that a package makes it so that the build is actually not reproducible? That is completely contradictory... But I'm really done this time on this topic, now. I'm pretty sure I clearly explained why a tainted flag is a bad idea... Regards, Yann E. MORIN.
Yann, All, On Mon, Sep 10, 2018 at 10:26 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Angelo, All, > > On 2018-09-10 21:03 +0100, Angelo Compagnucci spake thusly: > > On Mon, Sep 10, 2018 at 8:44 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > $ cat package/foo/Config.in > > > config BR2_PACKAGE_FOO > > > bool "foo" > > > depends on !BR2_REPRODUCIBLE > > > select SOME_PACKAGE > > > > > > $ cat package/foo/foo.mk > > > FOO_VERSION = 1.2.3 > > > FOO_DEPENDENCIES = some dependencies on other BR packages > > > $(eval $(npm-package)) > > > > > > and that the npm-package infra would do the call and install whatever > > > foo requires? Optionally, if the dependencies are already installed > > > (e.g. because of a FOO_DEPENDENCIES or another previous npm-package > > > already installed), then some dependencies may or may not be downloaded > > > by npm. > > > > I'm not into this honestly. What I want to achieve is that a package > > can call a npm install or a glide install on whatever dependencies > > file the source code distributes. Without the need to open that file, > > understanding the dependency chain and reflect that in buildroot. > > > > I think that the original developer knows better the dependencies of > > his software and we should trust the building infrastructure he > > choose, even more if what he choose is the best practice for that > > programming language. > > > > Exactly As I did for the example I posted before. > > And this is exactly what the npm-package (or glide-package, or whatever) > would end up doing: expand to FOO_BUILD_CMDS, FOO_INSTALL_CMDS et al. > that do the job, instead of repeating it in every package, like the > autotools-package infra allows to not repeat the .configure; make; make > install dance in all the autotools-based package. > > For example, your watchtower would be writen as: > > WATCHTOWER_VERSION = v0.3.0 > WATCHTOWER_LICENSE = Apache-2.0 > WATCHTOWER_LICENSE_FILES = LICENSE.md > $(eval $(glide-package)) > > And then the glide-pacakge infra would be defined sdmething like: > > define glide-pacakge-inner > > $(2)_DEPENDENCIES += host-glide > > define $(2)_GLIDE > cd $$($(2)_SRC_PATH); GOPATH=$$($(@)_GOPATH) $$(HOST_DIR)/usr/bin/glide install > endef > $(2)_POST_CONFIIGURE_HOOKS += $(2)_GLIDE > > $(call golang-package,$(1),$(2),$(3),$(4)) > > endef > > Or something similar, like we do for the rebar-package infra, for > example, which piggy-backs onto either the generic-package or the > autotools-package infra as needed. I don't say it could not be done, and indeed it's a pretty clever solution. But for golang there are at least a couples of different packages managers I'm aware of, so it could end up having too many infras. > > > > That would be fine by me. > > > > > > There is one gotcha though: this won't work for purely off-line builds. > > > This can be a big limitation in some setups, where access to an external > > > network is not always guaranteed, so we should carefully say so in the > > > manual. > > > > Not true. A package can call the package manager in it's post download > > hook, so a make source should do an offline build without troubles. > > But then said package manager would do only the download in the download > step? > > I.e. I want to be able to do on 'somemachine' which has acces to both > the internal and external networks: > > $ export BR2_DL_DIR=/some/location > $ make meh_defconfig > $ make source > $ scp -r /some/location someone@someserver:/somewhere > > And on 'someserver' which does not have access to the network at all (by > a firewall that blocks out-going connections): > > $ export BR2_DL_DIR=/somewhere > $ make meh_defconfig > $ make > > This is what I call an off-line build. It could be tricky but yes, every package manager has a caching mechanism, so with a bit of hackery we can end up having the download step downloading also the dependencies and then you can cut your connection. > > > Yes, if we can agree on a way to add package managers to buildroot! > > I don't see any technical limitation to that, given the example you > provided. The real thing here is, we have a policy right now stating that package managers are not admitted. I asked this explicitly to Peter and Thomas at the Buidroot developers meeting. We have to find the bast way to have them included imho. > > Of course, I guess it is just a trivial example, and that real life is > little bit more tricky as usual, but there is nothing (besides my own > stubbornness, that is) standing in the road, is there? Not true, most of the web software I worked with needs only an npm install or glide install and nothing more. > > Regards, > Yann E. MORIN. > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------'
Hi Angelo, On 06/09/18 00:22, Angelo Compagnucci wrote: > Packages that need to resolve dependencies internally > and use a package manager would harm the reproducibility > of a build, moreover they escape the legal infrastructure > not giving enough informations on licensing. > > This patch adds a tainting mechanism in the form of a > variable FOO_TAINTS that can be used to signal that > a package harms the reproducibility or licensing under > certain conditions. We had a discussion about this at the BR developer meeting, and we decided that the taints mechanism is not worth it. As noted by Yann, it can only ever be used for a warning, not to block anything, because there are ways to do the right thing for e.g. BR2_REPRODUCIBLE. And if it is just a warning, then it can just be included in the help text of the Config.in option. Or, if we want to make it stronger, a conditional comment. > This opens the door to include per language dependency > managers in buildroot. To be honest (not discussed, so purely my personal opinion), maybe we should just relax our opposition against language package managers. Yes, there are those 7 points (which I still haven't added to the documentation, shame on me) but that doesn't mean it's entirely blocking. I hope I haven't opened a box of Pandora by saying this :-) Regards, Arnout
Arnout, Angelo, On 2018-11-01 13:14 +0100, Arnout Vandecappelle spake thusly: > On 06/09/18 00:22, Angelo Compagnucci wrote: > > Packages that need to resolve dependencies internally > > and use a package manager would harm the reproducibility > > of a build, moreover they escape the legal infrastructure > > not giving enough informations on licensing. > > > > This patch adds a tainting mechanism in the form of a > > variable FOO_TAINTS that can be used to signal that > > a package harms the reproducibility or licensing under > > certain conditions. > > We had a discussion about this at the BR developer meeting, and we decided that > the taints mechanism is not worth it. As noted by Yann, it can only ever be used > for a warning, not to block anything, because there are ways to do the right > thing for e.g. BR2_REPRODUCIBLE. > > And if it is just a warning, then it can just be included in the help text of > the Config.in option. Or, if we want to make it stronger, a conditional comment. > > > This opens the door to include per language dependency > > managers in buildroot. > > To be honest (not discussed, so purely my personal opinion), maybe we should > just relax our opposition against language package managers. Yes, there are > those 7 points (which I still haven't added to the documentation, shame on me) > but that doesn't mean it's entirely blocking. I thought we did discuss it, at least informally, and came to the conclusion that we should not block them, indeed. > I hope I haven't opened a box of Pandora by saying this :-) It was already opened! ;-) Regards, Yann E. MORIN. > Regards, > Arnout > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot