diff mbox series

[6/9] kernel/x86: enable x32 support for amd64

Message ID be244a81b5835217795cec0a4886e0958657c1b9.1682539911.git.ehem+openwrt@m5p.com
State New
Headers show
Series (mostly) x86 kernel configuration adjustments | expand

Commit Message

Elliott Mitchell March 30, 2023, 11:30 p.m. UTC
Full amd64 support isn't really appropriate for most situations
OpenWRT is deployed.  Whereas x86-x32 seems extremely appropriate for
these situations.  As such enable x86-x32 support.

CONFIG_ARCH_MMAP_RND_COMPAT_BITS is required to follow along,
otherwise the kernel build breaks.

Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>
---
I suggest OpenWRT should be placing quite a bit of effort towards
x86-x32.  x86-x32 seems a rather superior generic target for OpenWRT.
Only issue is it could be valuable to have at least minimal amd64
userland support alongside the x86-x32 version.
---
 target/linux/x86/64/config-5.10 | 1 -
 target/linux/x86/64/config-5.15 | 1 -
 target/linux/x86/config-5.10    | 2 ++
 target/linux/x86/config-5.15    | 2 ++
 4 files changed, 4 insertions(+), 2 deletions(-)

Comments

Stefan Lippers-Hollmann April 26, 2023, 10:46 p.m. UTC | #1
Hi

On 2023-03-30, Elliott Mitchell wrote:
> Full amd64 support isn't really appropriate for most situations
> OpenWRT is deployed.  Whereas x86-x32 seems extremely appropriate for
> these situations.  As such enable x86-x32 support.
> 
> CONFIG_ARCH_MMAP_RND_COMPAT_BITS is required to follow along,
> otherwise the kernel build breaks.
> 
> Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>
> ---
> I suggest OpenWRT should be placing quite a bit of effort towards
> x86-x32.  x86-x32 seems a rather superior generic target for OpenWRT.
> Only issue is it could be valuable to have at least minimal amd64
> userland support alongside the x86-x32 version.

x86_32 is pretty much dead in the water, with almost zero deployment
by general purpose distributions - apart from VM data centre 
environments doing their own thing (least amount of RAM usage 
possible, everything else being secondary at best). At least Debian
did raise security concerns about the x86_32 ISA in the past.

While I might understand (understand, not support) a desire for this 
as a dedicated subtarget (to appease the virtualization crowd), 
although I still don't see a reason or sufficient uptake in more 
conventional Linux environments. I would not be happy (at all) to 
lose 'normal' x86_64 support (on real hardware) for this exotic 
fringe hybrid. I can imagine that actually building for this 
environment (with a 32 bit userland) might lead to 'funny' results 
as well (as in major toolchain changes necessary to get it working 
as expected).

Regards
	Stefan Lippers-Hollmann
Elliott Mitchell April 27, 2023, midnight UTC | #2
On Thu, Apr 27, 2023 at 12:46:49AM +0200, Stefan Lippers-Hollmann wrote:
> 
> On 2023-03-30, Elliott Mitchell wrote:
> > Full amd64 support isn't really appropriate for most situations
> > OpenWRT is deployed.  Whereas x86-x32 seems extremely appropriate for
> > these situations.  As such enable x86-x32 support.
> > 
> > CONFIG_ARCH_MMAP_RND_COMPAT_BITS is required to follow along,
> > otherwise the kernel build breaks.
> > 
> > Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>
> > ---
> > I suggest OpenWRT should be placing quite a bit of effort towards
> > x86-x32.  x86-x32 seems a rather superior generic target for OpenWRT.
> > Only issue is it could be valuable to have at least minimal amd64
> > userland support alongside the x86-x32 version.
> 
> x86_32 is pretty much dead in the water, with almost zero deployment
> by general purpose distributions - apart from VM data centre 
> environments doing their own thing (least amount of RAM usage 
> possible, everything else being secondary at best). At least Debian
> did raise security concerns about the x86_32 ISA in the past.

Error: undefined symbol "x86_32"

Without the extra context I would resolve that to "i386"/"ia32".  I think
"x32" or "x86_x32" are better strings for this case.

According to what I had read that was in the past when x32 was sharing
less of the i386 ABI.  Notably x32 had been trying to pass time values
in a distinct fashion.  I will conceed I'm unsure whether x32 is ever
truly going to get a seat at the table.

On a different news front, Linux 5.10 has an option
CONFIG_X86_X32_DISABLED which leaves x32 disabled by default (Debian's
kernels were configured this way).  Whereas 5.15 has removed the
CONFIG_X86_X32_DISABLED option which seems to suggest the concerns may
have been assuaged.

> While I might understand (understand, not support) a desire for this 
> as a dedicated subtarget (to appease the virtualization crowd), 
> although I still don't see a reason or sufficient uptake in more 
> conventional Linux environments. I would not be happy (at all) to 
> lose 'normal' x86_64 support (on real hardware) for this exotic 
> fringe hybrid. I can imagine that actually building for this 
> environment (with a 32 bit userland) might lead to 'funny' results 
> as well (as in major toolchain changes necessary to get it working 
> as expected).

I'm not proposing removing amd64 support, I'm proposing x32 is likely a
more valuable target.  Yet what you're describing reads like your desire
is for OpenWRT/x86 to simply be yet another desktop Linux distribution.

Unless you feel a networking device really needs 256GB of memory, virtual
machines are precisely what OpenWRT/x86 *should* target.  I think it is
reasonable to also have a jumbo/desktop build, but using an entire x86
machine doesn't seem to match OpenWRT's main theme.
Thibaut VARÈNE April 27, 2023, 9:27 a.m. UTC | #3
> Le 27 avr. 2023 à 02:00, Elliott Mitchell <ehem+openwrt@m5p.com> a écrit :
> 
> On Thu, Apr 27, 2023 at 12:46:49AM +0200, Stefan Lippers-Hollmann wrote:
> 
>> While I might understand (understand, not support) a desire for this 
>> as a dedicated subtarget (to appease the virtualization crowd), 
>> although I still don't see a reason or sufficient uptake in more 
>> conventional Linux environments. I would not be happy (at all) to 
>> lose 'normal' x86_64 support (on real hardware) for this exotic 
>> fringe hybrid. I can imagine that actually building for this 
>> environment (with a 32 bit userland) might lead to 'funny' results 
>> as well (as in major toolchain changes necessary to get it working 
>> as expected).
> 
> I'm not proposing removing amd64 support, I'm proposing x32 is likely a
> more valuable target.

Do you mean to actually introduce an x86_x32 userspace target in OpenWrt?
If so, I suggest you take a look at [1] to get an idea of the can of worms you might be opening there.

I do not think OpenWrt has the resources to handle this level of breakage for such an exotic, barely upstream supported target.

>  Yet what you're describing reads like your desire
> is for OpenWRT/x86 to simply be yet another desktop Linux distribution.
> 
> Unless you feel a networking device really needs 256GB of memory, virtual
> machines are precisely what OpenWRT/x86 *should* target.  I think it is
> reasonable to also have a jumbo/desktop build, but using an entire x86
> machine doesn't seem to match OpenWRT's main theme.

You seem to ignore perfectly capable so-called « mini pc » routers which are in use out there. They don’t need a « jumbo/desktop » build and they don’t have 256GB RAM, yet they work perfectly well with the current OpenWrt image.

Cheers,
T

[1] http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-x32@lists.debian.org&tags=port-x32
Philip Prindeville April 28, 2023, 5 p.m. UTC | #4
My own experience disagrees.

I spent 17 months bringing up a Xeon-D based traffic shaper for 40Gb/s of traffic in a radio base station.

And yes, when collected crunched traffic statistics, it did use more than 4GB of address space to do so.



> On Mar 30, 2023, at 5:30 PM, Elliott Mitchell <ehem+openwrt@m5p.com> wrote:
> 
> Full amd64 support isn't really appropriate for most situations
> OpenWRT is deployed.  Whereas x86-x32 seems extremely appropriate for
> these situations.  As such enable x86-x32 support.
> 
> CONFIG_ARCH_MMAP_RND_COMPAT_BITS is required to follow along,
> otherwise the kernel build breaks.
> 
> Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>
> ---
> I suggest OpenWRT should be placing quite a bit of effort towards
> x86-x32.  x86-x32 seems a rather superior generic target for OpenWRT.
> Only issue is it could be valuable to have at least minimal amd64
> userland support alongside the x86-x32 version.
> ---
> target/linux/x86/64/config-5.10 | 1 -
> target/linux/x86/64/config-5.15 | 1 -
> target/linux/x86/config-5.10    | 2 ++
> target/linux/x86/config-5.15    | 2 ++
> 4 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/target/linux/x86/64/config-5.10 b/target/linux/x86/64/config-5.10
> index 1515f90932..de93495fb1 100644
> --- a/target/linux/x86/64/config-5.10
> +++ b/target/linux/x86/64/config-5.10
> @@ -470,7 +470,6 @@ CONFIG_X86_PM_TIMER=y
> # CONFIG_X86_POWERNOW_K8 is not set
> # CONFIG_X86_VSYSCALL_EMULATION is not set
> CONFIG_X86_X2APIC=y
> -# CONFIG_X86_X32 is not set
> CONFIG_XEN=y
> CONFIG_XENFS=y
> CONFIG_XEN_512GB=y
> diff --git a/target/linux/x86/64/config-5.15 b/target/linux/x86/64/config-5.15
> index a20891ea55..39e5064e53 100644
> --- a/target/linux/x86/64/config-5.15
> +++ b/target/linux/x86/64/config-5.15
> @@ -493,7 +493,6 @@ CONFIG_X86_PM_TIMER=y
> # CONFIG_X86_POWERNOW_K8 is not set
> # CONFIG_X86_VSYSCALL_EMULATION is not set
> CONFIG_X86_X2APIC=y
> -# CONFIG_X86_X32 is not set
> CONFIG_XEN=y
> CONFIG_XENFS=y
> CONFIG_XEN_512GB=y
> diff --git a/target/linux/x86/config-5.10 b/target/linux/x86/config-5.10
> index afb7adc63a..8be829d549 100644
> --- a/target/linux/x86/config-5.10
> +++ b/target/linux/x86/config-5.10
> @@ -12,6 +12,7 @@ CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
> CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
> +CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
> CONFIG_ARCH_RANDOM=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> @@ -423,6 +424,7 @@ CONFIG_X86_UP_IOAPIC=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_X86_VMX_FEATURE_NAMES=y
> +CONFIG_X86_X32=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_ZLIB_INFLATE=y
> diff --git a/target/linux/x86/config-5.15 b/target/linux/x86/config-5.15
> index b59e809127..afe66b27b1 100644
> --- a/target/linux/x86/config-5.15
> +++ b/target/linux/x86/config-5.15
> @@ -13,6 +13,7 @@ CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y
> CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
> CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
> +CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
> CONFIG_ARCH_NR_GPIO=512
> CONFIG_ARCH_RANDOM=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> @@ -428,6 +429,7 @@ CONFIG_X86_UP_IOAPIC=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_X86_VMX_FEATURE_NAMES=y
> +CONFIG_X86_X32=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_ZLIB_INFLATE=y
> -- 
> (\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
> \BS (    |       ehem+openwrt@m5p.com     PGP 87145445       |    )   /
>  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
> 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
> 
> 
> 
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Philip Prindeville April 28, 2023, 5:10 p.m. UTC | #5
> On Apr 26, 2023, at 6:00 PM, Elliott Mitchell <ehem+openwrt@m5p.com> wrote:
> 
> On Thu, Apr 27, 2023 at 12:46:49AM +0200, Stefan Lippers-Hollmann wrote:
>> 
>> On 2023-03-30, Elliott Mitchell wrote:
>>> Full amd64 support isn't really appropriate for most situations
>>> OpenWRT is deployed.  Whereas x86-x32 seems extremely appropriate for
>>> these situations.  As such enable x86-x32 support.
>>> 
>>> CONFIG_ARCH_MMAP_RND_COMPAT_BITS is required to follow along,
>>> otherwise the kernel build breaks.
>>> 
>>> Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>
>>> ---
>>> I suggest OpenWRT should be placing quite a bit of effort towards
>>> x86-x32.  x86-x32 seems a rather superior generic target for OpenWRT.
>>> Only issue is it could be valuable to have at least minimal amd64
>>> userland support alongside the x86-x32 version.
>> 
>> x86_32 is pretty much dead in the water, with almost zero deployment
>> by general purpose distributions - apart from VM data centre 
>> environments doing their own thing (least amount of RAM usage 
>> possible, everything else being secondary at best). At least Debian
>> did raise security concerns about the x86_32 ISA in the past.
> 
> Error: undefined symbol "x86_32"
> 
> Without the extra context I would resolve that to "i386"/"ia32".  I think
> "x32" or "x86_x32" are better strings for this case.
> 
> According to what I had read that was in the past when x32 was sharing
> less of the i386 ABI.  Notably x32 had been trying to pass time values
> in a distinct fashion.  I will conceed I'm unsure whether x32 is ever
> truly going to get a seat at the table.
> 
> On a different news front, Linux 5.10 has an option
> CONFIG_X86_X32_DISABLED which leaves x32 disabled by default (Debian's
> kernels were configured this way).  Whereas 5.15 has removed the
> CONFIG_X86_X32_DISABLED option which seems to suggest the concerns may
> have been assuaged.
> 
>> While I might understand (understand, not support) a desire for this 
>> as a dedicated subtarget (to appease the virtualization crowd), 
>> although I still don't see a reason or sufficient uptake in more 
>> conventional Linux environments. I would not be happy (at all) to 
>> lose 'normal' x86_64 support (on real hardware) for this exotic 
>> fringe hybrid. I can imagine that actually building for this 
>> environment (with a 32 bit userland) might lead to 'funny' results 
>> as well (as in major toolchain changes necessary to get it working 
>> as expected).
> 
> I'm not proposing removing amd64 support, I'm proposing x32 is likely a
> more valuable target.  Yet what you're describing reads like your desire
> is for OpenWRT/x86 to simply be yet another desktop Linux distribution.
> 
> Unless you feel a networking device really needs 256GB of memory, virtual
> machines are precisely what OpenWRT/x86 *should* target.  I think it is
> reasonable to also have a jumbo/desktop build, but using an entire x86
> machine doesn't seem to match OpenWRT's main theme.


I personally know of companies running virtualized instances of AMD64 on m4.large or m5.large Amazon EC2 instances.

I have three AMD64 firewalls...  Two PC Engines APU6's (AMD GX-412TC "Jaguar" SoC based) running as a hot tandem pair for fail-over at the home office, and a dogfooding instance running on a KVM server that sandboxes my teenage kids and keeps them off the main household networks.
Elliott Mitchell April 29, 2023, 2:58 a.m. UTC | #6
On Thu, Apr 27, 2023 at 11:27:30AM +0200, Thibaut wrote:
> 
> > Le 27 avr. 2023 à 02:00, Elliott Mitchell <ehem+openwrt@m5p.com> a écrit :
> > 
> > On Thu, Apr 27, 2023 at 12:46:49AM +0200, Stefan Lippers-Hollmann wrote:
> > 
> >> While I might understand (understand, not support) a desire for this 
> >> as a dedicated subtarget (to appease the virtualization crowd), 
> >> although I still don't see a reason or sufficient uptake in more 
> >> conventional Linux environments. I would not be happy (at all) to 
> >> lose 'normal' x86_64 support (on real hardware) for this exotic 
> >> fringe hybrid. I can imagine that actually building for this 
> >> environment (with a 32 bit userland) might lead to 'funny' results 
> >> as well (as in major toolchain changes necessary to get it working 
> >> as expected).
> > 
> > I'm not proposing removing amd64 support, I'm proposing x32 is likely a
> > more valuable target.
> 
> Do you mean to actually introduce an x86_x32 userspace target in OpenWrt?
> If so, I suggest you take a look at [1] to get an idea of the can of worms you might be opening there.
> 
> I do not think OpenWrt has the resources to handle this level of breakage for such an exotic, barely upstream supported target.

It seems worthy experimention at least.  Enabling the kernel option adds
somewhere between 1-8191 bytes (kernel build grew 1 page).  If this
reduces memory usage by 10% and increases performance by 10% that seems
notable.

My hope is that others have flushed most of the bugs out by now.  If
there are still too many bugs at this point, it can be left unreleased.

> >  Yet what you're describing reads like your desire
> > is for OpenWRT/x86 to simply be yet another desktop Linux distribution.
> > 
> > Unless you feel a networking device really needs 256GB of memory, virtual
> > machines are precisely what OpenWRT/x86 *should* target.  I think it is
> > reasonable to also have a jumbo/desktop build, but using an entire x86
> > machine doesn't seem to match OpenWRT's main theme.
> 
> You seem to ignore perfectly capable so-called « mini pc » routers which are in use out there. They don’t need a « jumbo/desktop » build and they don’t have 256GB RAM, yet they work perfectly well with the current OpenWrt image.

That is indeed a better choice for consideration.  What type of
installation do you think OpenWRT should target for such devices?  I'm
unsure how much memory such devices typically have.  I was able to find
listings for SO-DIMMs from 4GB to 32GB.

Most recent devices will include an IOMMU and 4GB is more than enough to
usefully use a hypervisor.  In such a case it might well make sense to
use 128MB for handling the network and devote the rest to other tasks.
With a system that size, there is rather MORE urgency to keep VMs small.

Alternatively do you feel a router needs 4GB of memory?  Thing is this
turns OpenWRT into simply another desktop Linux distribution.  At which
point why not go with a Linux distribution designed for the desktop and
has all the desktop features?
Elliott Mitchell April 29, 2023, 3:10 a.m. UTC | #7
On Fri, Apr 28, 2023 at 11:00:43AM -0600, Philip Prindeville wrote:
> My own experience disagrees.
> 
> I spent 17 months bringing up a Xeon-D based traffic shaper for 40Gb/s of traffic in a radio base station.
> 
> And yes, when collected crunched traffic statistics, it did use more than 4GB of address space to do so.

I can believe that.  Though I kind of doubt this is the typical usage of
OpenWRT.  A kernel can have both amd64 and x32 enabled.  My thinking was
in typical usage most programs won't be that large and thus x32 is
smaller and faster.

Note, enabling CONFIG_X86_X32 does not force the use of x32 userspace,
it merely allows it.  The observed kernel runtime was 4KB larger which
translates to +1-8191 bytes.
diff mbox series

Patch

diff --git a/target/linux/x86/64/config-5.10 b/target/linux/x86/64/config-5.10
index 1515f90932..de93495fb1 100644
--- a/target/linux/x86/64/config-5.10
+++ b/target/linux/x86/64/config-5.10
@@ -470,7 +470,6 @@  CONFIG_X86_PM_TIMER=y
 # CONFIG_X86_POWERNOW_K8 is not set
 # CONFIG_X86_VSYSCALL_EMULATION is not set
 CONFIG_X86_X2APIC=y
-# CONFIG_X86_X32 is not set
 CONFIG_XEN=y
 CONFIG_XENFS=y
 CONFIG_XEN_512GB=y
diff --git a/target/linux/x86/64/config-5.15 b/target/linux/x86/64/config-5.15
index a20891ea55..39e5064e53 100644
--- a/target/linux/x86/64/config-5.15
+++ b/target/linux/x86/64/config-5.15
@@ -493,7 +493,6 @@  CONFIG_X86_PM_TIMER=y
 # CONFIG_X86_POWERNOW_K8 is not set
 # CONFIG_X86_VSYSCALL_EMULATION is not set
 CONFIG_X86_X2APIC=y
-# CONFIG_X86_X32 is not set
 CONFIG_XEN=y
 CONFIG_XENFS=y
 CONFIG_XEN_512GB=y
diff --git a/target/linux/x86/config-5.10 b/target/linux/x86/config-5.10
index afb7adc63a..8be829d549 100644
--- a/target/linux/x86/config-5.10
+++ b/target/linux/x86/config-5.10
@@ -12,6 +12,7 @@  CONFIG_ARCH_HIBERNATION_POSSIBLE=y
 CONFIG_ARCH_MAY_HAVE_PC_FDC=y
 CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
 CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
+CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
 CONFIG_ARCH_RANDOM=y
 CONFIG_ARCH_SELECT_MEMORY_MODEL=y
 CONFIG_ARCH_SPARSEMEM_ENABLE=y
@@ -423,6 +424,7 @@  CONFIG_X86_UP_IOAPIC=y
 CONFIG_X86_USE_PPRO_CHECKSUM=y
 CONFIG_X86_VERBOSE_BOOTUP=y
 CONFIG_X86_VMX_FEATURE_NAMES=y
+CONFIG_X86_X32=y
 CONFIG_XZ_DEC_BCJ=y
 CONFIG_XZ_DEC_X86=y
 CONFIG_ZLIB_INFLATE=y
diff --git a/target/linux/x86/config-5.15 b/target/linux/x86/config-5.15
index b59e809127..afe66b27b1 100644
--- a/target/linux/x86/config-5.15
+++ b/target/linux/x86/config-5.15
@@ -13,6 +13,7 @@  CONFIG_ARCH_MAY_HAVE_PC_FDC=y
 CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y
 CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
 CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
+CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
 CONFIG_ARCH_NR_GPIO=512
 CONFIG_ARCH_RANDOM=y
 CONFIG_ARCH_SELECT_MEMORY_MODEL=y
@@ -428,6 +429,7 @@  CONFIG_X86_UP_IOAPIC=y
 CONFIG_X86_USE_PPRO_CHECKSUM=y
 CONFIG_X86_VERBOSE_BOOTUP=y
 CONFIG_X86_VMX_FEATURE_NAMES=y
+CONFIG_X86_X32=y
 CONFIG_XZ_DEC_BCJ=y
 CONFIG_XZ_DEC_X86=y
 CONFIG_ZLIB_INFLATE=y