Message ID | 20230707185106.98208-1-ckhardin@gmail.com |
---|---|
State | New |
Headers | show |
Series | [1/1] package/go-native: add a new download only package for go compilers | expand |
Hi Charles, This is not necessary with this patch series: https://patchwork.ozlabs.org/project/buildroot/list/?series=358451 We can correctly bootstrap on arm64. On Fri, Jul 7, 2023 at 11:51 AM Charles Hardin <ckhardin@gmail.com> wrote: > > The go-native is mainly for the aarch64 support since that was > added after the go-bootstrap written in C was dropped. So, to > get a cross compiler for go a host/native compiler needs to be > available to generate the setup for the needed target. This is > using the same go version as the expected cross-compiler to > keep the packages in sync. A specific version could be used. Thanks, Christian Stewart
Hi Charles, On Fri, Jul 7, 2023 at 12:04 PM Christian Stewart <christian@aperture.us> wrote: > This is not necessary with this patch series: > > https://patchwork.ozlabs.org/project/buildroot/list/?series=358451 > > We can correctly bootstrap on arm64. I should clarify: actually go bootstrap stage1 cannot work on arm64 (you're correct on this). I had submitted this patch as well to use the host "go" in this case: https://patchwork.ozlabs.org/project/buildroot/patch/20230215073256.186476-3-christian@paral.in/ Similar to how we use host gcc. However, I understand that your series intends to add downloading the binaries for the Go compiler to Buildroot as well. This is potentially useful but I still submit that we need to merge the above series updating to Go 1.20 and splitting the bootstrap into two phases before merging the go-native series. I'll start testing your patch as well. Thanks, Christian > On Fri, Jul 7, 2023 at 11:51 AM Charles Hardin <ckhardin@gmail.com> wrote: > > > > The go-native is mainly for the aarch64 support since that was > > added after the go-bootstrap written in C was dropped. So, to > > get a cross compiler for go a host/native compiler needs to be > > available to generate the setup for the needed target. This is > > using the same go version as the expected cross-compiler to > > keep the packages in sync. A specific version could be used.
Christian On Fri, Jul 7, 2023 at 12:08 PM Christian Stewart <christian@aperture.us> wrote: > Hi Charles, > > > On Fri, Jul 7, 2023 at 12:04 PM Christian Stewart <christian@aperture.us> > wrote: > > This is not necessary with this patch series: > > > > https://patchwork.ozlabs.org/project/buildroot/list/?series=358451 > > > > We can correctly bootstrap on arm64. > > I should clarify: actually go bootstrap stage1 cannot work on arm64 > (you're correct on this). > > I had submitted this patch as well to use the host "go" in this case: > > > https://patchwork.ozlabs.org/project/buildroot/patch/20230215073256.186476-3-christian@paral.in/ > > Similar to how we use host gcc. > > Weird - i was looking for a fix and never saw - perhaps i didn't have the magic google search keywords. So, this was "our fix" to just get something working specifically on a aarch64 builds. This is our annual timeline of let's take our patches we have locally and send them upstream to see what happens and see if we can drop our local patches in favor of upstream fixes. > However, I understand that your series intends to add downloading the > binaries for the Go compiler to Buildroot as well. > > This is potentially useful but I still submit that we need to merge > the above series updating to Go 1.20 and splitting the bootstrap into > two phases before merging the go-native series. > > I just went for a quick and dirty to just use a native instead of doing the bootstrap - we have a major issue in that we don't have many x86_64 build systems anymore and we are having to move to aarch64 builds almost exclusively. So, not having a way to get the compiler going was becoming a headache. > I'll start testing your patch as well. > > This is not meant to replace - more of a - here is a way that worked for us, if there is a better way that fits into buildroot then please just merge that. Right now we don't have anything in the proper mainline so everything is a deviation at this time. > Thanks, > Christian > > > On Fri, Jul 7, 2023 at 11:51 AM Charles Hardin <ckhardin@gmail.com> > wrote: > > > > > > The go-native is mainly for the aarch64 support since that was > > > added after the go-bootstrap written in C was dropped. So, to > > > get a cross compiler for go a host/native compiler needs to be > > > available to generate the setup for the needed target. This is > > > using the same go version as the expected cross-compiler to > > > keep the packages in sync. A specific version could be used. >
Hi Charles, On Fri, Jul 7, 2023, 1:05 PM Charles Hardin <ckhardin@gmail.com> wrote: > Christian > > On Fri, Jul 7, 2023 at 12:08 PM Christian Stewart <christian@aperture.us> > wrote: > >> Hi Charles, >> >> >> On Fri, Jul 7, 2023 at 12:04 PM Christian Stewart <christian@aperture.us> >> wrote: >> > This is not necessary with this patch series: >> > >> > https://patchwork.ozlabs.org/project/buildroot/list/?series=358451 >> > >> > We can correctly bootstrap on arm64. >> >> I should clarify: actually go bootstrap stage1 cannot work on arm64 >> (you're correct on this). >> >> I had submitted this patch as well to use the host "go" in this case: >> >> >> https://patchwork.ozlabs.org/project/buildroot/patch/20230215073256.186476-3-christian@paral.in/ >> >> Similar to how we use host gcc. >> >> > Weird - i was looking for a fix and never saw - perhaps i didn't have the > magic google search keywords. So, this was "our fix" to just get something > working specifically on a aarch64 builds. This is our annual timeline of > let's take our patches we have locally and send them upstream to see what > happens and see if we can drop our local patches in favor of upstream fixes. > > >> However, I understand that your series intends to add downloading the >> binaries for the Go compiler to Buildroot as well. >> >> This is potentially useful but I still submit that we need to merge >> the above series updating to Go 1.20 and splitting the bootstrap into >> two phases before merging the go-native series. >> >> > I just went for a quick and dirty to just use a native instead of doing > the bootstrap - we have a major issue in that we don't have many x86_64 > build systems anymore and we are having to move to aarch64 builds almost > exclusively. So, not having a way to get the compiler going was becoming a > headache. > Ah - okay this makes sense. In this case I recommend using my patch to use the host Go compiler to bootstrap the Go compiler. This still compiles whatever version of Go buildroot has in the tree from source using the host Go compiler to bootstrap. Thomas, all; I do request that we merge the go 1.20 series and the "use host compiler to bootstrap on unsupported architectures" patch asap as these are significant blockers for some users (as seen here) Thanks! Christian
diff --git a/package/Config.in.host b/package/Config.in.host index dcadbfdfc1..d7932f048b 100644 --- a/package/Config.in.host +++ b/package/Config.in.host @@ -41,6 +41,7 @@ menu "Host utilities" source "package/gnupg/Config.in.host" source "package/go/Config.in.host" source "package/go-bootstrap/Config.in.host" + source "package/go-native/Config.in.host" source "package/google-breakpad/Config.in.host" source "package/gptfdisk/Config.in.host" source "package/imagemagick/Config.in.host" diff --git a/package/go-native/Config.in.host b/package/go-native/Config.in.host new file mode 100644 index 0000000000..7e978e02b5 --- /dev/null +++ b/package/go-native/Config.in.host @@ -0,0 +1,12 @@ +config BR2_PACKAGE_HOST_GO_NATIVE_ARCH_SUPPORTS + bool + # See https://go.dev/dl/ for supported architecture downloads + default y if BR2_HOSTARCH = "x86" + default y if BR2_HOSTARCH = "x86_64" + default y if BR2_HOSTARCH = "arm" + default y if BR2_HOSTARCH = "aarch64" + +config BR2_PACKAGE_HOST_GO_NATIVE + bool "host go-native" + depends on BR2_PACKAGE_HOST_GO_NATIVE_ARCH_SUPPORTS + default y if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS diff --git a/package/go-native/go-native.hash b/package/go-native/go-native.hash new file mode 100644 index 0000000000..3af2f51abc --- /dev/null +++ b/package/go-native/go-native.hash @@ -0,0 +1,5 @@ +# From https://go.dev/dl/ +sha256 45a34313664d66087d49b46ca63a3ecf292f56e9e63bd439829b5eaf0270586d go1.19.10.linux-386.tar.gz +sha256 8b045a483d3895c6edba2e90a9189262876190dbbd21756870cdd63821810677 go1.19.10.linux-amd64.tar.gz +sha256 df98698821211c819e8b2420c77a0f802d989e377718578a31b1f91f6be2c5b4 go1.19.10.linux-arm64.tar.gz +sha256 937f885c35f191be4c6a618868ae0dbbb15b0213925a1a104d122eba02622356 go1.19.10.linux-armv6l.tar.gz diff --git a/package/go-native/go-native.mk b/package/go-native/go-native.mk new file mode 100644 index 0000000000..4b541cce73 --- /dev/null +++ b/package/go-native/go-native.mk @@ -0,0 +1,48 @@ +################################################################################ +# +# go-native +# +################################################################################ + +# Fixup the host arch as needed +ifeq ($(HOSTARCH),x86) +GO_NATIVE_HOST_ARCH = 386 +else ifeq ($(HOSTARCH),x86_64) +GO_NATIVE_HOST_ARCH = amd64 +else ifeq ($(HOSTARCH),arm) +GO_NATIVE_HOST_ARCH = armv6l +else ifeq ($(HOSTARCH),aarch64) +GO_NATIVE_HOST_ARCH = arm64 +else +GO_NATIVE_HOST_ARCH = $(HOSTARCH) +endif + +# Download the native tools from as the same go version from golang +GO_NATIVE_VERSION = $(GO_VERSION) +GO_NATIVE_SITE = https://go.dev/dl +GO_NATIVE_LICENSE = BSD-3-Clause +GO_NATIVE_LICENSE_FILES = LICENSE + +HOST_GO_NATIVE_SOURCE = go$(GO_NATIVE_VERSION).linux-$(GO_NATIVE_HOST_ARCH).tar.gz + +# To build programs that need cgo support the toolchain needs to be +# available, so the toolchain is not needed to build host-go-native +# itself, but needed by other packages that depend on +# host-go-native. +HOST_GO_NATIVE_DEPENDENCIES = toolchain + +HOST_GO_NATIVE_ROOT = $(HOST_DIR)/lib/go-$(GO_NATIVE_VERSION) + +define HOST_GO_NATIVE_INSTALL_CMDS + $(INSTALL) -D -m 0755 $(@D)/bin/go $(HOST_GO_NATIVE_ROOT)/bin/go + $(INSTALL) -D -m 0755 $(@D)/bin/gofmt $(HOST_GO_NATIVE_ROOT)/bin/gofmt + + cp -a $(@D)/lib $(HOST_GO_NATIVE_ROOT)/ + cp -a $(@D)/pkg $(HOST_GO_NATIVE_ROOT)/ + + # There is a known issue which requires the go sources to be installed + # https://golang.org/issue/2775 + cp -a $(@D)/src $(HOST_GO_NATIVE_ROOT)/ +endef + +$(eval $(host-generic-package)) diff --git a/package/go/Config.in.host b/package/go/Config.in.host index ded02d3b3a..4dbbe9a221 100644 --- a/package/go/Config.in.host +++ b/package/go/Config.in.host @@ -2,7 +2,8 @@ config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS bool default y - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS + depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS \ + || BR2_PACKAGE_HOST_GO_NATIVE_ARCH_SUPPORTS depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \ || BR2_i386 || BR2_x86_64 || BR2_powerpc64le \ || BR2_mips64 || BR2_mips64el || BR2_riscv || BR2_s390x @@ -28,4 +29,5 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS bool default y - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS + depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS \ + || BR2_PACKAGE_HOST_GO_NATIVE_ARCH_SUPPORTS diff --git a/package/go/go.mk b/package/go/go.mk index 545d2117b7..af2e3b2f74 100644 --- a/package/go/go.mk +++ b/package/go/go.mk @@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause GO_LICENSE_FILES = LICENSE GO_CPE_ID_VENDOR = golang +ifeq ($(BR2_PACKAGE_HOST_GO_NATIVE),y) +HOST_GO_DEPENDENCIES = host-go-native +else HOST_GO_DEPENDENCIES = host-go-bootstrap +endif HOST_GO_GOPATH = $(HOST_DIR)/share/go-path HOST_GO_HOST_CACHE = $(HOST_DIR)/share/host-go-cache HOST_GO_ROOT = $(HOST_DIR)/lib/go @@ -123,10 +127,16 @@ HOST_GO_HOST_ENV = \ # The go build system is not compatible with ccache, so use # HOSTCC_NOCCACHE. See https://github.com/golang/go/issues/11685. +ifeq ($(BR2_PACKAGE_HOST_GO_NATIVE),y) HOST_GO_MAKE_ENV = \ + GOROOT_BOOTSTRAP=$(HOST_GO_NATIVE_ROOT) +else +HOST_GO_MAKE_ENV = \ + GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_ROOT) +endif +HOST_GO_MAKE_ENV += \ GO111MODULE=off \ GOCACHE=$(HOST_GO_HOST_CACHE) \ - GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_ROOT) \ GOROOT_FINAL=$(HOST_GO_ROOT) \ GOROOT="$(@D)" \ GOBIN="$(@D)/bin" \
The go-native is mainly for the aarch64 support since that was added after the go-bootstrap written in C was dropped. So, to get a cross compiler for go a host/native compiler needs to be available to generate the setup for the needed target. This is using the same go version as the expected cross-compiler to keep the packages in sync. A specific version could be used. Signed-off-by: Charles Hardin <ckhardin@gmail.com> --- package/Config.in.host | 1 + package/go-native/Config.in.host | 12 ++++++++ package/go-native/go-native.hash | 5 ++++ package/go-native/go-native.mk | 48 ++++++++++++++++++++++++++++++++ package/go/Config.in.host | 6 ++-- package/go/go.mk | 12 +++++++- 6 files changed, 81 insertions(+), 3 deletions(-) create mode 100644 package/go-native/Config.in.host create mode 100644 package/go-native/go-native.hash create mode 100644 package/go-native/go-native.mk