| Message ID | 20220805143625.16542-2-guillaume.bressaix@gmail.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [RFC,v3,1/2] package/pkg-meson.mk: support fortran cross-compilation | expand |
Hello Thomas, Thanks to Ralf Gommers from the Scipy dev team, we understand what was wrong in the previous submissions, we have a fix, see my v3 submission. I wanted to mark this one as ready for review, but decided not to.. Build passes for standard platforms, but the runtime test fails. $scipy/scipy/meson.build l70: the pybind11 include path is resolved to some $(STAGING_DIR)/include location in the runtime test env, which "does not exist". Scipy uses a "python import pybind" command to deduce the pybind11 lib location, which is totally fine, but somehow leads to faulty behavior in the qemu environment Guillaume W. Bres Software engineer <guillaume.bressaix@gmail.com> Le ven. 5 août 2022 à 16:37, Guillaume W. Bres <guillaume.bressaix@gmail.com> a écrit : > SciPy is moving to a meson build infrastructure. > The current meson build scripts deduce all internal > locations based on the host-python location, which is incorrect > when dealing with "npyrandom" and "npymath", that are included > in code that gets cross-compiled. > > https://github.com/scipy/scipy/issues/14812 > > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> > --- > As expected, moving to meson already makes things much easier for us, > as meson is capable of deducing a cross-compilation context and can take > advantage of pkg-config too. Also, the distutils approach will get removed > before 1.12.0. > > The npy core lib "problem" is acknowledged by the scipy dev team and we > can expect improved > behavior in the next releases. Ralf Gommers from the team provided > valuable input for this patch serie, thank you! > > Run the runtime test with > ./support/testing/run-tests -d dl -o test -k > test.package.test_python_scipy.TestPythonPy3SciPy > --- > ...uild-fix-numpy-core-include-location.patch | 33 +++++++++++++++ > package/python-scipy/python-scipy.hash | 2 +- > package/python-scipy/python-scipy.mk | 42 +++++++++---------- > 3 files changed, 53 insertions(+), 24 deletions(-) > create mode 100644 > package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch > > diff --git > a/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch > b/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch > new file mode 100644 > index 0000000000..3b5ee78f3e > --- /dev/null > +++ > b/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch > @@ -0,0 +1,33 @@ > +From 3045d4ebc83dda5922175758adc74018aebccdd1 Mon Sep 17 00:00:00 2001 > +From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > +Date: Fri, 5 Aug 2022 16:02:21 +0200 > +Subject: [PATCH] scipy/meson.build: fix numpy core include location > + > +The numpy core location is deduced from host-python, > +while we will include lib npyrandom and npymath when cross compiling. > +These two must be previously cross-compiled. > + > +Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> > +--- > + scipy/meson.build | 4 ++-- > + 1 file changed, 2 insertions(+), 2 deletions(-) > + > +diff --git a/scipy/meson.build b/scipy/meson.build > +index 76bb3606e..f65601aee 100644 > +--- a/scipy/meson.build > ++++ b/scipy/meson.build > +@@ -52,9 +52,9 @@ inc_f2py = include_directories(incdir_f2py) > + fortranobject_c = incdir_f2py / 'fortranobject.c' > + > + cc = meson.get_compiler('c') > +-npymath_path = incdir_numpy / '..' / 'lib' > ++npymath_path = @NUMPY_SYSROOT_INCDIR@ / '..' / 'lib' > + npymath_lib = cc.find_library('npymath', dirs: npymath_path) > +-npyrandom_path = incdir_numpy / '..' / '..' / 'random' / 'lib' > ++npyrandom_path = @NUMPY_SYSROOT_INCDIR@ / '..' / '..' / 'random' / 'lib' > + # Note: `required: false` can be removed once numpy 1.19 is the minimum > version > + npyrandom_lib = cc.find_library('npyrandom', dirs: npyrandom_path, > required: false) > + > +-- > +2.30.2 > + > diff --git a/package/python-scipy/python-scipy.hash > b/package/python-scipy/python-scipy.hash > index 0274679c43..6637faf5b4 100644 > --- a/package/python-scipy/python-scipy.hash > +++ b/package/python-scipy/python-scipy.hash > @@ -1,5 +1,5 @@ > # Locally generated > -sha256 0140dd24d14ce194fdb3df496c2190f0896ca84ac98bf5d69c3b3fb7bde1a1e9 > scipy-1.8.1.tar.xz > +sha256 c0dfd7d2429452e7e94904c6a3af63cbaa3cf51b348bd9d35b42db7e9ad42791 > scipy-1.9.0.tar.gz > sha256 96599f8ec69a2cc609265eed13fc75090aacb28742260e55ce623788a33da19c > LICENSE.txt > sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35 > scipy/linalg/src/lapack_deprecations/LICENSE > sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609 > scipy/ndimage/LICENSE.txt > diff --git a/package/python-scipy/python-scipy.mk b/package/python-scipy/ > python-scipy.mk > index b5006a2e15..cb0b078e0b 100644 > --- a/package/python-scipy/python-scipy.mk > +++ b/package/python-scipy/python-scipy.mk > @@ -4,8 +4,8 @@ > # > > ################################################################################ > > -PYTHON_SCIPY_VERSION = 1.8.1 > -PYTHON_SCIPY_SOURCE = scipy-$(PYTHON_SCIPY_VERSION).tar.xz > +PYTHON_SCIPY_VERSION = 1.9.0 > +PYTHON_SCIPY_SOURCE = scipy-$(PYTHON_SCIPY_VERSION).tar.gz > PYTHON_SCIPY_SITE = > https://github.com/scipy/scipy/releases/download/v$(PYTHON_SCIPY_VERSION) > PYTHON_SCIPY_LICENSE = \ > BSD-3-Clause, \ > @@ -23,6 +23,7 @@ PYTHON_SCIPY_LICENSE_FILES = \ > scipy/sparse/linalg/_eigen/arpack/ARPACK/COPYING \ > scipy/spatial/qhull_src/COPYING.txt > PYTHON_SCIPY_DEPENDENCIES += \ > + host-pkgconf \ > host-python-numpy \ > host-python-pythran \ > zlib \ > @@ -31,26 +32,21 @@ PYTHON_SCIPY_DEPENDENCIES += \ > python-pybind > PYTHON_SCIPY_INSTALL_STAGING = YES > > -PYTHON_SCIPY_SETUP_TYPE = setuptools > -PYTHON_SCIPY_BUILD_OPTS = config_fc --fcompiler=gnu95 > - > -PYTHON_SCIPY_LDFLAGS = $(TARGET_LDFLAGS) -shared \ > - -L$(PYTHON3_PATH)/site-packages/numpy/core/lib > -# -lnpyrandom localization > -PYTHON_SCIPY_LDFLAGS += \ > - > -L$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/numpy/random/lib > - > -PYTHON_SCIPY_ENV = \ > - F90=$(TARGET_FC) \ > - LDFLAGS="$(PYTHON_SCIPY_LDFLAGS)" > - > -# Provide system configuration options to numpy distutils extensions, > telling > -# to find all include files and libraries in staging directory. > -define PYTHON_SCIPY_CONFIGURE_CMDS > - -rm -f $(@D)/site.cfg > - echo "[DEFAULT]" >> $(@D)/site.cfg > - echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg > - echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg > +# the meson.build infrastructure, currently uses host-python > +# for code generation, f2py invokation, but also to determine > +# all internal dependencies location. It is incorrect regarging > +# libnpyrandom and libnpymath because we need the cross-compiled version > +# for these two > +# sed -i 's/@NUMPY_SYSROOT_INCDIR@/$(STAGING_DIR)\/usr\/lib\/python$(PYTHON3_VERSION_MAJOR)\/site-packages\/numpy/' > $(@D)/scipy/meson.build > +define PYTHON_SCIPY_FIX_NUMPY_CORE_SYSROOT_LOCATION > + sed -i \ > + "s:@NUMPY_SYSROOT_INCDIR@:'$(STAGING_DIR)\/usr\/lib\/python$(PYTHON3_VERSION_MAJOR)\/site-packages\/numpy\/core\/include\/':" > \ > + $(@D)/scipy/meson.build > endef > +PYTHON_SCIPY_PRE_CONFIGURE_HOOKS += > PYTHON_SCIPY_FIX_NUMPY_CORE_SYSROOT_LOCATION > + > +PYTHON_SCIPY_CONF_OPTS = \ > + -Dblas=lapack \ > + -Dlapack=lapack > > -$(eval $(python-package)) > +$(eval $(meson-package)) > -- > 2.30.2 > >
On 05/08/2022 16:46, Guillaume Bres wrote: > Hello Thomas, > > Thanks to Ralf Gommers from the Scipy dev team, we understand what was wrong in > the previous submissions, we have a fix, see my v3 submission. > I wanted to mark this one as ready for review, but decided not to.. Build passes > for standard platforms, but the runtime test fails. > > $scipy/scipy/meson.build l70: the pybind11 include path is resolved to some > $(STAGING_DIR)/include location in the runtime test env, > which "does not exist". Scipy uses a "python import pybind" command to deduce > the pybind11 lib location, which is totally fine, but somehow leads to faulty > behavior in the qemu environment pybind is consistently annoying, it uses ldd and stuff to discover paths... Regards, Arnout > > Guillaume W. Bres > Software engineer > <guillaume.bressaix@gmail.com <mailto:guillaume.bressaix@gmail.com>> > > > Le ven. 5 août 2022 à 16:37, Guillaume W. Bres <guillaume.bressaix@gmail.com > <mailto:guillaume.bressaix@gmail.com>> a écrit : > > SciPy is moving to a meson build infrastructure. > The current meson build scripts deduce all internal > locations based on the host-python location, which is incorrect > when dealing with "npyrandom" and "npymath", that are included > in code that gets cross-compiled. > > https://github.com/scipy/scipy/issues/14812 > <https://github.com/scipy/scipy/issues/14812> > > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com > <mailto:guillaume.bressaix@gmail.com>> > --- > As expected, moving to meson already makes things much easier for us, > as meson is capable of deducing a cross-compilation context and can take > advantage of pkg-config too. Also, the distutils approach will get removed > before 1.12.0. > > The npy core lib "problem" is acknowledged by the scipy dev team and we can > expect improved > behavior in the next releases. Ralf Gommers from the team provided > valuable input for this patch serie, thank you! > > Run the runtime test with > ./support/testing/run-tests -d dl -o test -k > test.package.test_python_scipy.TestPythonPy3SciPy > --- > ...uild-fix-numpy-core-include-location.patch | 33 +++++++++++++++ > package/python-scipy/python-scipy.hash | 2 +- > package/python-scipy/python-scipy.mk <http://python-scipy.mk> | > 42 +++++++++---------- > 3 files changed, 53 insertions(+), 24 deletions(-) > create mode 100644 > package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch > > diff --git > a/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch > b/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch > new file mode 100644 > index 0000000000..3b5ee78f3e > --- /dev/null > +++ > b/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch > @@ -0,0 +1,33 @@ > +From 3045d4ebc83dda5922175758adc74018aebccdd1 Mon Sep 17 00:00:00 2001 > +From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com > <mailto:guillaume.bressaix@gmail.com>> > +Date: Fri, 5 Aug 2022 16:02:21 +0200 > +Subject: [PATCH] scipy/meson.build: fix numpy core include location > + > +The numpy core location is deduced from host-python, > +while we will include lib npyrandom and npymath when cross compiling. > +These two must be previously cross-compiled. > + > +Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com > <mailto:guillaume.bressaix@gmail.com>> > +--- > + scipy/meson.build | 4 ++-- > + 1 file changed, 2 insertions(+), 2 deletions(-) > + > +diff --git a/scipy/meson.build b/scipy/meson.build > +index 76bb3606e..f65601aee 100644 > +--- a/scipy/meson.build > ++++ b/scipy/meson.build > +@@ -52,9 +52,9 @@ inc_f2py = include_directories(incdir_f2py) > + fortranobject_c = incdir_f2py / 'fortranobject.c' > + > + cc = meson.get_compiler('c') > +-npymath_path = incdir_numpy / '..' / 'lib' > ++npymath_path = @NUMPY_SYSROOT_INCDIR@ / '..' / 'lib' > + npymath_lib = cc.find_library('npymath', dirs: npymath_path) > +-npyrandom_path = incdir_numpy / '..' / '..' / 'random' / 'lib' > ++npyrandom_path = @NUMPY_SYSROOT_INCDIR@ / '..' / '..' / 'random' / 'lib' > + # Note: `required: false` can be removed once numpy 1.19 is the minimum > version > + npyrandom_lib = cc.find_library('npyrandom', dirs: npyrandom_path, > required: false) > + > +-- > +2.30.2 > + > diff --git a/package/python-scipy/python-scipy.hash > b/package/python-scipy/python-scipy.hash > index 0274679c43..6637faf5b4 100644 > --- a/package/python-scipy/python-scipy.hash > +++ b/package/python-scipy/python-scipy.hash > @@ -1,5 +1,5 @@ > # Locally generated > -sha256 0140dd24d14ce194fdb3df496c2190f0896ca84ac98bf5d69c3b3fb7bde1a1e9 > scipy-1.8.1.tar.xz > +sha256 c0dfd7d2429452e7e94904c6a3af63cbaa3cf51b348bd9d35b42db7e9ad42791 > scipy-1.9.0.tar.gz > sha256 96599f8ec69a2cc609265eed13fc75090aacb28742260e55ce623788a33da19c > LICENSE.txt > sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35 > scipy/linalg/src/lapack_deprecations/LICENSE > sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609 > scipy/ndimage/LICENSE.txt > diff --git a/package/python-scipy/python-scipy.mk <http://python-scipy.mk> > b/package/python-scipy/python-scipy.mk <http://python-scipy.mk> > index b5006a2e15..cb0b078e0b 100644 > --- a/package/python-scipy/python-scipy.mk <http://python-scipy.mk> > +++ b/package/python-scipy/python-scipy.mk <http://python-scipy.mk> > @@ -4,8 +4,8 @@ > # > ################################################################################ > > -PYTHON_SCIPY_VERSION = 1.8.1 > -PYTHON_SCIPY_SOURCE = scipy-$(PYTHON_SCIPY_VERSION).tar.xz > +PYTHON_SCIPY_VERSION = 1.9.0 > +PYTHON_SCIPY_SOURCE = scipy-$(PYTHON_SCIPY_VERSION).tar.gz > PYTHON_SCIPY_SITE = > https://github.com/scipy/scipy/releases/download/v$(PYTHON_SCIPY_VERSION) > <https://github.com/scipy/scipy/releases/download/v$(PYTHON_SCIPY_VERSION)> > PYTHON_SCIPY_LICENSE = \ > BSD-3-Clause, \ > @@ -23,6 +23,7 @@ PYTHON_SCIPY_LICENSE_FILES = \ > scipy/sparse/linalg/_eigen/arpack/ARPACK/COPYING \ > scipy/spatial/qhull_src/COPYING.txt > PYTHON_SCIPY_DEPENDENCIES += \ > + host-pkgconf \ > host-python-numpy \ > host-python-pythran \ > zlib \ > @@ -31,26 +32,21 @@ PYTHON_SCIPY_DEPENDENCIES += \ > python-pybind > PYTHON_SCIPY_INSTALL_STAGING = YES > > -PYTHON_SCIPY_SETUP_TYPE = setuptools > -PYTHON_SCIPY_BUILD_OPTS = config_fc --fcompiler=gnu95 > - > -PYTHON_SCIPY_LDFLAGS = $(TARGET_LDFLAGS) -shared \ > - -L$(PYTHON3_PATH)/site-packages/numpy/core/lib > -# -lnpyrandom localization > -PYTHON_SCIPY_LDFLAGS += \ > - > -L$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/numpy/random/lib > - > -PYTHON_SCIPY_ENV = \ > - F90=$(TARGET_FC) \ > - LDFLAGS="$(PYTHON_SCIPY_LDFLAGS)" > - > -# Provide system configuration options to numpy distutils extensions, telling > -# to find all include files and libraries in staging directory. > -define PYTHON_SCIPY_CONFIGURE_CMDS > - -rm -f $(@D)/site.cfg > - echo "[DEFAULT]" >> $(@D)/site.cfg > - echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg > - echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg > +# the meson.build infrastructure, currently uses host-python > +# for code generation, f2py invokation, but also to determine > +# all internal dependencies location. It is incorrect regarging > +# libnpyrandom and libnpymath because we need the cross-compiled version > +# for these two > +# sed -i > 's/@NUMPY_SYSROOT_INCDIR@/$(STAGING_DIR)\/usr\/lib\/python$(PYTHON3_VERSION_MAJOR)\/site-packages\/numpy/' > $(@D)/scipy/meson.build > +define PYTHON_SCIPY_FIX_NUMPY_CORE_SYSROOT_LOCATION > + sed -i \ > + > "s:@NUMPY_SYSROOT_INCDIR@:'$(STAGING_DIR)\/usr\/lib\/python$(PYTHON3_VERSION_MAJOR)\/site-packages\/numpy\/core\/include\/':" \ > + $(@D)/scipy/meson.build > endef > +PYTHON_SCIPY_PRE_CONFIGURE_HOOKS += > PYTHON_SCIPY_FIX_NUMPY_CORE_SYSROOT_LOCATION > + > +PYTHON_SCIPY_CONF_OPTS = \ > + -Dblas=lapack \ > + -Dlapack=lapack > > -$(eval $(python-package)) > +$(eval $(meson-package)) > -- > 2.30.2 > > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot
On Mon, 8 Aug 2022 22:58:46 +0200
Arnout Vandecappelle <arnout@mind.be> wrote:
> pybind is consistently annoying, it uses ldd and stuff to discover paths...
Well, it's not just pybind. Python itself, in its ctype.find_library()
function, uses ldconfig, tries to use gcc on the target, and more funky
stuff that borks terribly in a Buildroot situation.
Thomas
for information, pybind will soon become pkg-config knowledgeable, as this issue is about to get merged https://github.com/pybind/pybind11/pull/4077 Guillaume W. Bres Software engineer <guillaume.bressaix@gmail.com> Le lun. 8 août 2022 à 23:13, Thomas Petazzoni <thomas.petazzoni@bootlin.com> a écrit : > On Mon, 8 Aug 2022 22:58:46 +0200 > Arnout Vandecappelle <arnout@mind.be> wrote: > > > pybind is consistently annoying, it uses ldd and stuff to discover > paths... > > Well, it's not just pybind. Python itself, in its ctype.find_library() > function, uses ldconfig, tries to use gcc on the target, and more funky > stuff that borks terribly in a Buildroot situation. > > Thomas > -- > Thomas Petazzoni, co-owner and CEO, Bootlin > Embedded Linux and Kernel engineering and training > https://bootlin.com >
Hello all, I suggest we wait for new updates of scipy before converting the package to meson. In the meantime, I expect pybind and numpy to be sanitized too (probably the impulse of scipy), so we should keep track of these. For information, scipy-v1.9.1 has just been released but it does not come with anything new regarding meson scripts. I currently have a v3 patch that is able to upgrade scipy to v1.9.0 and meson, but it involves patching the build script (it's fairly minor). But this patch is not 100% ready and needs to be finalized. What do you guys think is best? Should I conclude this, or we remain as is (which works fine) and wait for new releases? I am worried such tweaks appear fine at first, but might raise lots of issues once injected into BR Guillaume W. Bres Software engineer <guillaume.bressaix@gmail.com> Le mar. 9 août 2022 à 10:37, Guillaume Bres <guillaume.bressaix@gmail.com> a écrit : > for information, > pybind will soon become pkg-config knowledgeable, as this issue is about > to get merged https://github.com/pybind/pybind11/pull/4077 > > Guillaume W. Bres > Software engineer > <guillaume.bressaix@gmail.com> > > > Le lun. 8 août 2022 à 23:13, Thomas Petazzoni < > thomas.petazzoni@bootlin.com> a écrit : > >> On Mon, 8 Aug 2022 22:58:46 +0200 >> Arnout Vandecappelle <arnout@mind.be> wrote: >> >> > pybind is consistently annoying, it uses ldd and stuff to discover >> paths... >> >> Well, it's not just pybind. Python itself, in its ctype.find_library() >> function, uses ldconfig, tries to use gcc on the target, and more funky >> stuff that borks terribly in a Buildroot situation. >> >> Thomas >> -- >> Thomas Petazzoni, co-owner and CEO, Bootlin >> Embedded Linux and Kernel engineering and training >> https://bootlin.com >> >
>1.9.2 should be the first version built with Meson now that we've got the wheel build machinery switched over [..] >I will note that SciPy CI now contains one cross-compilation job (macOS x-86-64 to arm64), so that should be more robust now Thank you this is valuable information Ralf >The main issue for cross-compilation is running the Python interpreter on the build machine to retrieve numpy and pybind11 Indeed I understand the problem >Note that you can either wait a bit longer, or if you do want to include SciPy 1.9.1, just build it with distutils - if that worked for 1.8.x it will still work for 1.9.1 Ok let's do that then Guillaume W. Bres Software engineer <guillaume.bressaix@gmail.com> Le sam. 27 août 2022 à 11:54, Ralf Gommers <ralf.gommers@gmail.com> a écrit : > Hi Guillaume, all, > > > On Sat, Aug 27, 2022 at 12:11 PM Guillaume Bres < > guillaume.bressaix@gmail.com> wrote: > >> Hello all, >> >> I suggest we wait for new updates of scipy before converting the package >> to meson. >> > > That does sound like the right call. The SciPy wheels on PyPI for 1.9.1 > are still built with distutils; 1.9.2 should be the first version built > with Meson now that we've got the wheel build machinery switched over. > Making that move to Meson before SciPy itself has finalized it is likely to > uncover some more issues. I will note that SciPy CI now contains one > cross-compilation job (macOS x-86-64 to arm64), so that should be more > robust now. > > >> In the meantime, I expect pybind and numpy to be sanitized too (probably >> the impulse of scipy), so we should keep track of these. >> > > Indeed. The main issue for cross-compilation is running the Python > interpreter on the build machine to retrieve numpy and pybind11. I'm trying > to get rid of that need. I will note that the distutils-based build does > the same and ends up with incorrect static libraries included in the Python > extension modules, but it happens to work for odd reasons (basically two > bugs cancelling each other out there). The Meson builds correctly detects > that situation as problematic and errors out rather than including a static > library with the wrong architecture. > > For information, scipy-v1.9.1 has just been released but it does not come >> with anything new regarding meson scripts. >> >> I currently have a v3 patch that is able to upgrade scipy to v1.9.0 and >> meson, but it involves patching the build script (it's fairly minor). >> But this patch is not 100% ready and needs to be finalized. >> What do you guys think is best? Should I conclude this, or we remain as >> is (which works fine) and wait for new releases? >> > > Note that you can either wait a bit longer, or if you do want to include > SciPy 1.9.1, just build it with distutils - if that worked for 1.8.x it > will still work for 1.9.1 > > Cheers, > Ralf > > > I am worried such tweaks appear fine at first, but might raise lots of >> issues once injected into BR >> >> Guillaume W. Bres >> Software engineer >> <guillaume.bressaix@gmail.com> >> >> >> Le mar. 9 août 2022 à 10:37, Guillaume Bres <guillaume.bressaix@gmail.com> >> a écrit : >> >>> for information, >>> pybind will soon become pkg-config knowledgeable, as this issue is about >>> to get merged https://github.com/pybind/pybind11/pull/4077 >>> >>> Guillaume W. Bres >>> Software engineer >>> <guillaume.bressaix@gmail.com> >>> >>> >>> Le lun. 8 août 2022 à 23:13, Thomas Petazzoni < >>> thomas.petazzoni@bootlin.com> a écrit : >>> >>>> On Mon, 8 Aug 2022 22:58:46 +0200 >>>> Arnout Vandecappelle <arnout@mind.be> wrote: >>>> >>>> > pybind is consistently annoying, it uses ldd and stuff to discover >>>> paths... >>>> >>>> Well, it's not just pybind. Python itself, in its ctype.find_library() >>>> function, uses ldconfig, tries to use gcc on the target, and more funky >>>> stuff that borks terribly in a Buildroot situation. >>>> >>>> Thomas >>>> -- >>>> Thomas Petazzoni, co-owner and CEO, Bootlin >>>> Embedded Linux and Kernel engineering and training >>>> https://bootlin.com >>>> >>>
diff --git a/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch b/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch new file mode 100644 index 0000000000..3b5ee78f3e --- /dev/null +++ b/package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch @@ -0,0 +1,33 @@ +From 3045d4ebc83dda5922175758adc74018aebccdd1 Mon Sep 17 00:00:00 2001 +From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> +Date: Fri, 5 Aug 2022 16:02:21 +0200 +Subject: [PATCH] scipy/meson.build: fix numpy core include location + +The numpy core location is deduced from host-python, +while we will include lib npyrandom and npymath when cross compiling. +These two must be previously cross-compiled. + +Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> +--- + scipy/meson.build | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +diff --git a/scipy/meson.build b/scipy/meson.build +index 76bb3606e..f65601aee 100644 +--- a/scipy/meson.build ++++ b/scipy/meson.build +@@ -52,9 +52,9 @@ inc_f2py = include_directories(incdir_f2py) + fortranobject_c = incdir_f2py / 'fortranobject.c' + + cc = meson.get_compiler('c') +-npymath_path = incdir_numpy / '..' / 'lib' ++npymath_path = @NUMPY_SYSROOT_INCDIR@ / '..' / 'lib' + npymath_lib = cc.find_library('npymath', dirs: npymath_path) +-npyrandom_path = incdir_numpy / '..' / '..' / 'random' / 'lib' ++npyrandom_path = @NUMPY_SYSROOT_INCDIR@ / '..' / '..' / 'random' / 'lib' + # Note: `required: false` can be removed once numpy 1.19 is the minimum version + npyrandom_lib = cc.find_library('npyrandom', dirs: npyrandom_path, required: false) + +-- +2.30.2 + diff --git a/package/python-scipy/python-scipy.hash b/package/python-scipy/python-scipy.hash index 0274679c43..6637faf5b4 100644 --- a/package/python-scipy/python-scipy.hash +++ b/package/python-scipy/python-scipy.hash @@ -1,5 +1,5 @@ # Locally generated -sha256 0140dd24d14ce194fdb3df496c2190f0896ca84ac98bf5d69c3b3fb7bde1a1e9 scipy-1.8.1.tar.xz +sha256 c0dfd7d2429452e7e94904c6a3af63cbaa3cf51b348bd9d35b42db7e9ad42791 scipy-1.9.0.tar.gz sha256 96599f8ec69a2cc609265eed13fc75090aacb28742260e55ce623788a33da19c LICENSE.txt sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35 scipy/linalg/src/lapack_deprecations/LICENSE sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609 scipy/ndimage/LICENSE.txt diff --git a/package/python-scipy/python-scipy.mk b/package/python-scipy/python-scipy.mk index b5006a2e15..cb0b078e0b 100644 --- a/package/python-scipy/python-scipy.mk +++ b/package/python-scipy/python-scipy.mk @@ -4,8 +4,8 @@ # ################################################################################ -PYTHON_SCIPY_VERSION = 1.8.1 -PYTHON_SCIPY_SOURCE = scipy-$(PYTHON_SCIPY_VERSION).tar.xz +PYTHON_SCIPY_VERSION = 1.9.0 +PYTHON_SCIPY_SOURCE = scipy-$(PYTHON_SCIPY_VERSION).tar.gz PYTHON_SCIPY_SITE = https://github.com/scipy/scipy/releases/download/v$(PYTHON_SCIPY_VERSION) PYTHON_SCIPY_LICENSE = \ BSD-3-Clause, \ @@ -23,6 +23,7 @@ PYTHON_SCIPY_LICENSE_FILES = \ scipy/sparse/linalg/_eigen/arpack/ARPACK/COPYING \ scipy/spatial/qhull_src/COPYING.txt PYTHON_SCIPY_DEPENDENCIES += \ + host-pkgconf \ host-python-numpy \ host-python-pythran \ zlib \ @@ -31,26 +32,21 @@ PYTHON_SCIPY_DEPENDENCIES += \ python-pybind PYTHON_SCIPY_INSTALL_STAGING = YES -PYTHON_SCIPY_SETUP_TYPE = setuptools -PYTHON_SCIPY_BUILD_OPTS = config_fc --fcompiler=gnu95 - -PYTHON_SCIPY_LDFLAGS = $(TARGET_LDFLAGS) -shared \ - -L$(PYTHON3_PATH)/site-packages/numpy/core/lib -# -lnpyrandom localization -PYTHON_SCIPY_LDFLAGS += \ - -L$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/numpy/random/lib - -PYTHON_SCIPY_ENV = \ - F90=$(TARGET_FC) \ - LDFLAGS="$(PYTHON_SCIPY_LDFLAGS)" - -# Provide system configuration options to numpy distutils extensions, telling -# to find all include files and libraries in staging directory. -define PYTHON_SCIPY_CONFIGURE_CMDS - -rm -f $(@D)/site.cfg - echo "[DEFAULT]" >> $(@D)/site.cfg - echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg - echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg +# the meson.build infrastructure, currently uses host-python +# for code generation, f2py invokation, but also to determine +# all internal dependencies location. It is incorrect regarging +# libnpyrandom and libnpymath because we need the cross-compiled version +# for these two +# sed -i 's/@NUMPY_SYSROOT_INCDIR@/$(STAGING_DIR)\/usr\/lib\/python$(PYTHON3_VERSION_MAJOR)\/site-packages\/numpy/' $(@D)/scipy/meson.build +define PYTHON_SCIPY_FIX_NUMPY_CORE_SYSROOT_LOCATION + sed -i \ + "s:@NUMPY_SYSROOT_INCDIR@:'$(STAGING_DIR)\/usr\/lib\/python$(PYTHON3_VERSION_MAJOR)\/site-packages\/numpy\/core\/include\/':" \ + $(@D)/scipy/meson.build endef +PYTHON_SCIPY_PRE_CONFIGURE_HOOKS += PYTHON_SCIPY_FIX_NUMPY_CORE_SYSROOT_LOCATION + +PYTHON_SCIPY_CONF_OPTS = \ + -Dblas=lapack \ + -Dlapack=lapack -$(eval $(python-package)) +$(eval $(meson-package))
SciPy is moving to a meson build infrastructure. The current meson build scripts deduce all internal locations based on the host-python location, which is incorrect when dealing with "npyrandom" and "npymath", that are included in code that gets cross-compiled. https://github.com/scipy/scipy/issues/14812 Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> --- As expected, moving to meson already makes things much easier for us, as meson is capable of deducing a cross-compilation context and can take advantage of pkg-config too. Also, the distutils approach will get removed before 1.12.0. The npy core lib "problem" is acknowledged by the scipy dev team and we can expect improved behavior in the next releases. Ralf Gommers from the team provided valuable input for this patch serie, thank you! Run the runtime test with ./support/testing/run-tests -d dl -o test -k test.package.test_python_scipy.TestPythonPy3SciPy --- ...uild-fix-numpy-core-include-location.patch | 33 +++++++++++++++ package/python-scipy/python-scipy.hash | 2 +- package/python-scipy/python-scipy.mk | 42 +++++++++---------- 3 files changed, 53 insertions(+), 24 deletions(-) create mode 100644 package/python-scipy/0001-scipy-meson.build-fix-numpy-core-include-location.patch