Message ID | 1481802034-77729-1-git-send-email-chenweilong@huawei.com |
---|---|
State | Changes Requested, archived |
Delegated to: | David Miller |
Headers | show |
On Thu, Dec 15, 2016 at 3:40 AM, Weilong Chen <chenweilong@huawei.com> wrote: > Nessus report the vf appears to leak memory in network packets. > Fix this by padding all small packets manually. > > And the CVE-2003-0001. > https://ofirarkin.files.wordpress.com/2008/11/atstake_etherleak_report.pdf > > Signed-off-by: Weilong Chen <chenweilong@huawei.com> > --- > drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c > index 6d4bef5..137a154 100644 > --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c > +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c > @@ -3654,6 +3654,13 @@ static int ixgbevf_xmit_frame(struct sk_buff *skb, struct net_device *netdev) > return NETDEV_TX_OK; > } > > + /* On PCI/PCI-X HW, if packet size is less than ETH_ZLEN, > + * packets may get corrupted during padding by HW. > + * To WA this issue, pad all small packets manually. > + */ > + if (eth_skb_pad(skb)) > + return NETDEV_TX_OK; > + So the patch description for this probably isn't correct. It looks like the problem isn't leaking data it is the fact that the frames aren't being padded to prevent malicious events. The only issue is the patch is padding by a bit too much. I would recommend replacing this with the following from ixgbe: /* * The minimum packet size for olinfo paylen is 17 so pad the skb * in order to meet this minimum size requirement. */ if (skb_put_padto(skb, 17)) return NETDEV_TX_OK; > tx_ring = adapter->tx_ring[skb->queue_mapping]; > > /* need: 1 descriptor per page * PAGE_SIZE/IXGBE_MAX_DATA_PER_TXD, > -- > 1.7.12 >
Hi, Thanks for you reply. We test you patch, but the problem is still there, it seems do not work. I'm not sure why ixgbe use the limit 17. The kenel use ETH_ZLEN (60) with out FCS. A lot of drivers such as e1000 use it. Any explaination? Thanks. On 2016/12/16 0:13, Alexander Duyck wrote: > On Thu, Dec 15, 2016 at 3:40 AM, Weilong Chen <chenweilong@huawei.com> wrote: >> Nessus report the vf appears to leak memory in network packets. >> Fix this by padding all small packets manually. >> >> And the CVE-2003-0001. >> https://ofirarkin.files.wordpress.com/2008/11/atstake_etherleak_report.pdf >> >> Signed-off-by: Weilong Chen <chenweilong@huawei.com> >> --- >> drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >> index 6d4bef5..137a154 100644 >> --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >> +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >> @@ -3654,6 +3654,13 @@ static int ixgbevf_xmit_frame(struct sk_buff *skb, struct net_device *netdev) >> return NETDEV_TX_OK; >> } >> >> + /* On PCI/PCI-X HW, if packet size is less than ETH_ZLEN, >> + * packets may get corrupted during padding by HW. >> + * To WA this issue, pad all small packets manually. >> + */ >> + if (eth_skb_pad(skb)) >> + return NETDEV_TX_OK; >> + > > So the patch description for this probably isn't correct. It looks > like the problem isn't leaking data it is the fact that the frames > aren't being padded to prevent malicious events. The only issue is > the patch is padding by a bit too much. I would recommend replacing > this with the following from ixgbe: > > /* > * The minimum packet size for olinfo paylen is 17 so pad the skb > * in order to meet this minimum size requirement. > */ > if (skb_put_padto(skb, 17)) > return NETDEV_TX_OK; > > >> tx_ring = adapter->tx_ring[skb->queue_mapping]; >> >> /* need: 1 descriptor per page * PAGE_SIZE/IXGBE_MAX_DATA_PER_TXD, >> -- >> 1.7.12 >> > > . >
The limit of 17 is just based on the hardware. Specifically the olinfo field in the Tx descriptor has a minimum length of 17 has a requirement. The hardware itself is supposed to be capable of padding short frames that are supposed to be transmitted. The drivers are supposed to pad short frames on receive to get them up to 60 bytes. When you are seeing this issue are you sending frames from the VF to one of the local interfaces on the same port or to an external interface? Also are you receiving on another linux ixgbevf driver or are you receiving the packet using a different driver interface such as DPDK? I'm just wanting to verify this as it is possible that the memory leak you are seeing is on the receiver and not on the source if you are transmitting to a local VF or the PF as the receiver will have to pad the frame in such a case to get it up to 60 bytes. - Alex On Tue, Dec 20, 2016 at 3:50 AM, Weilong Chen <chenweilong@huawei.com> wrote: > Hi, > > Thanks for you reply. > We test you patch, but the problem is still there, it seems do not work. > > I'm not sure why ixgbe use the limit 17. The kenel use ETH_ZLEN (60) with > out FCS. A lot of drivers such as e1000 use it. Any explaination? > > Thanks. > > > On 2016/12/16 0:13, Alexander Duyck wrote: >> >> On Thu, Dec 15, 2016 at 3:40 AM, Weilong Chen <chenweilong@huawei.com> >> wrote: >>> >>> Nessus report the vf appears to leak memory in network packets. >>> Fix this by padding all small packets manually. >>> >>> And the CVE-2003-0001. >>> >>> https://ofirarkin.files.wordpress.com/2008/11/atstake_etherleak_report.pdf >>> >>> Signed-off-by: Weilong Chen <chenweilong@huawei.com> >>> --- >>> drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 7 +++++++ >>> 1 file changed, 7 insertions(+) >>> >>> diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>> b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>> index 6d4bef5..137a154 100644 >>> --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>> +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>> @@ -3654,6 +3654,13 @@ static int ixgbevf_xmit_frame(struct sk_buff *skb, >>> struct net_device *netdev) >>> return NETDEV_TX_OK; >>> } >>> >>> + /* On PCI/PCI-X HW, if packet size is less than ETH_ZLEN, >>> + * packets may get corrupted during padding by HW. >>> + * To WA this issue, pad all small packets manually. >>> + */ >>> + if (eth_skb_pad(skb)) >>> + return NETDEV_TX_OK; >>> + >> >> >> So the patch description for this probably isn't correct. It looks >> like the problem isn't leaking data it is the fact that the frames >> aren't being padded to prevent malicious events. The only issue is >> the patch is padding by a bit too much. I would recommend replacing >> this with the following from ixgbe: >> >> /* >> * The minimum packet size for olinfo paylen is 17 so pad the skb >> * in order to meet this minimum size requirement. >> */ >> if (skb_put_padto(skb, 17)) >> return NETDEV_TX_OK; >> >> >>> tx_ring = adapter->tx_ring[skb->queue_mapping]; >>> >>> /* need: 1 descriptor per page * >>> PAGE_SIZE/IXGBE_MAX_DATA_PER_TXD, >>> -- >>> 1.7.12 >>> >> >> . >> >
Thanks for you explanation, it's very professional. My test is like this: The Nessus is deployed on a windows server, the peer is a X86_64 linux host which run several VMs on it. The nic is Intel 82599 and SRIOV is enabled. VFs are passthroughed to the VMs. No DPDK. The Nessus server send small ICMP echo request packets to the VM, and then check the reply, and report the error: "11197 - Multiple Ethernet Driver Frame Padding Information Disclosure (Etherleak)" "Padding observed in one frame : 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 57 37 28 .............W7( 0x10: 76 v Padding observed in another frame : 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 D3 4D 75 ..............Mu 0x10: 28 (" I only have Nessus's windows version, so can't test on linux. Maybe the windows server does not pad small packets to 60 bytes on the receive path. On 2016/12/21 0:36, Alexander Duyck wrote: > The limit of 17 is just based on the hardware. Specifically the > olinfo field in the Tx descriptor has a minimum length of 17 has a > requirement. The hardware itself is supposed to be capable of padding > short frames that are supposed to be transmitted. The drivers are > supposed to pad short frames on receive to get them up to 60 bytes. > > When you are seeing this issue are you sending frames from the VF to > one of the local interfaces on the same port or to an external > interface? Also are you receiving on another linux ixgbevf driver or > are you receiving the packet using a different driver interface such > as DPDK? I'm just wanting to verify this as it is possible that the > memory leak you are seeing is on the receiver and not on the source if > you are transmitting to a local VF or the PF as the receiver will have > to pad the frame in such a case to get it up to 60 bytes. > > - Alex > > On Tue, Dec 20, 2016 at 3:50 AM, Weilong Chen <chenweilong@huawei.com> wrote: >> Hi, >> >> Thanks for you reply. >> We test you patch, but the problem is still there, it seems do not work. >> >> I'm not sure why ixgbe use the limit 17. The kenel use ETH_ZLEN (60) with >> out FCS. A lot of drivers such as e1000 use it. Any explaination? >> >> Thanks. >> >> >> On 2016/12/16 0:13, Alexander Duyck wrote: >>> >>> On Thu, Dec 15, 2016 at 3:40 AM, Weilong Chen <chenweilong@huawei.com> >>> wrote: >>>> >>>> Nessus report the vf appears to leak memory in network packets. >>>> Fix this by padding all small packets manually. >>>> >>>> And the CVE-2003-0001. >>>> >>>> https://ofirarkin.files.wordpress.com/2008/11/atstake_etherleak_report.pdf >>>> >>>> Signed-off-by: Weilong Chen <chenweilong@huawei.com> >>>> --- >>>> drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 7 +++++++ >>>> 1 file changed, 7 insertions(+) >>>> >>>> diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>>> b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>>> index 6d4bef5..137a154 100644 >>>> --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>>> +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>>> @@ -3654,6 +3654,13 @@ static int ixgbevf_xmit_frame(struct sk_buff *skb, >>>> struct net_device *netdev) >>>> return NETDEV_TX_OK; >>>> } >>>> >>>> + /* On PCI/PCI-X HW, if packet size is less than ETH_ZLEN, >>>> + * packets may get corrupted during padding by HW. >>>> + * To WA this issue, pad all small packets manually. >>>> + */ >>>> + if (eth_skb_pad(skb)) >>>> + return NETDEV_TX_OK; >>>> + >>> >>> >>> So the patch description for this probably isn't correct. It looks >>> like the problem isn't leaking data it is the fact that the frames >>> aren't being padded to prevent malicious events. The only issue is >>> the patch is padding by a bit too much. I would recommend replacing >>> this with the following from ixgbe: >>> >>> /* >>> * The minimum packet size for olinfo paylen is 17 so pad the skb >>> * in order to meet this minimum size requirement. >>> */ >>> if (skb_put_padto(skb, 17)) >>> return NETDEV_TX_OK; >>> >>> >>>> tx_ring = adapter->tx_ring[skb->queue_mapping]; >>>> >>>> /* need: 1 descriptor per page * >>>> PAGE_SIZE/IXGBE_MAX_DATA_PER_TXD, >>>> -- >>>> 1.7.12 >>>> >>> >>> . >>> >> > > . >
I find it curious that only the last 4 bytes have data in them. I'm wondering if the NIC/driver in the Windows/Nessus system is interpreting the 4 byte CRC on the end of the frame as padding instead of stripping it. Is there any chance you could capture the entire frame instead of just the padding? Maybe you could run something like wireshark without enabling promiscuous mode on the VF and capture the frames it is trying to send and receive. What I want to verify is what the actual amount of padding is that is needed to get to 60 bytes and where the CRC should start. - Alex On Tue, Dec 20, 2016 at 5:40 PM, Weilong Chen <chenweilong@huawei.com> wrote: > Thanks for you explanation, it's very professional. > > My test is like this: > The Nessus is deployed on a windows server, the peer is a X86_64 linux host > which run several VMs on it. The nic is Intel 82599 and SRIOV is enabled. > VFs are passthroughed to the VMs. No DPDK. > > The Nessus server send small ICMP echo request packets to the VM, and > then check the reply, and report the error: > > "11197 - Multiple Ethernet Driver Frame Padding Information Disclosure > (Etherleak)" > > "Padding observed in one frame : > > 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 57 37 28 .............W7( > 0x10: 76 v > > Padding observed in another frame : > > 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 D3 4D 75 ..............Mu > 0x10: 28 (" > > I only have Nessus's windows version, so can't test on linux. Maybe the > windows server does not pad small packets to 60 bytes on the receive path. > > > On 2016/12/21 0:36, Alexander Duyck wrote: >> >> The limit of 17 is just based on the hardware. Specifically the >> olinfo field in the Tx descriptor has a minimum length of 17 has a >> requirement. The hardware itself is supposed to be capable of padding >> short frames that are supposed to be transmitted. The drivers are >> supposed to pad short frames on receive to get them up to 60 bytes. >> >> When you are seeing this issue are you sending frames from the VF to >> one of the local interfaces on the same port or to an external >> interface? Also are you receiving on another linux ixgbevf driver or >> are you receiving the packet using a different driver interface such >> as DPDK? I'm just wanting to verify this as it is possible that the >> memory leak you are seeing is on the receiver and not on the source if >> you are transmitting to a local VF or the PF as the receiver will have >> to pad the frame in such a case to get it up to 60 bytes. >> >> - Alex >> >> On Tue, Dec 20, 2016 at 3:50 AM, Weilong Chen <chenweilong@huawei.com> >> wrote: >>> >>> Hi, >>> >>> Thanks for you reply. >>> We test you patch, but the problem is still there, it seems do not work. >>> >>> I'm not sure why ixgbe use the limit 17. The kenel use ETH_ZLEN (60) with >>> out FCS. A lot of drivers such as e1000 use it. Any explaination? >>> >>> Thanks. >>> >>> >>> On 2016/12/16 0:13, Alexander Duyck wrote: >>>> >>>> >>>> On Thu, Dec 15, 2016 at 3:40 AM, Weilong Chen <chenweilong@huawei.com> >>>> wrote: >>>>> >>>>> >>>>> Nessus report the vf appears to leak memory in network packets. >>>>> Fix this by padding all small packets manually. >>>>> >>>>> And the CVE-2003-0001. >>>>> >>>>> >>>>> https://ofirarkin.files.wordpress.com/2008/11/atstake_etherleak_report.pdf >>>>> >>>>> Signed-off-by: Weilong Chen <chenweilong@huawei.com> >>>>> --- >>>>> drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 7 +++++++ >>>>> 1 file changed, 7 insertions(+) >>>>> >>>>> diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>>>> b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>>>> index 6d4bef5..137a154 100644 >>>>> --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>>>> +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c >>>>> @@ -3654,6 +3654,13 @@ static int ixgbevf_xmit_frame(struct sk_buff >>>>> *skb, >>>>> struct net_device *netdev) >>>>> return NETDEV_TX_OK; >>>>> } >>>>> >>>>> + /* On PCI/PCI-X HW, if packet size is less than ETH_ZLEN, >>>>> + * packets may get corrupted during padding by HW. >>>>> + * To WA this issue, pad all small packets manually. >>>>> + */ >>>>> + if (eth_skb_pad(skb)) >>>>> + return NETDEV_TX_OK; >>>>> + >>>> >>>> >>>> >>>> So the patch description for this probably isn't correct. It looks >>>> like the problem isn't leaking data it is the fact that the frames >>>> aren't being padded to prevent malicious events. The only issue is >>>> the patch is padding by a bit too much. I would recommend replacing >>>> this with the following from ixgbe: >>>> >>>> /* >>>> * The minimum packet size for olinfo paylen is 17 so pad the >>>> skb >>>> * in order to meet this minimum size requirement. >>>> */ >>>> if (skb_put_padto(skb, 17)) >>>> return NETDEV_TX_OK; >>>> >>>> >>>>> tx_ring = adapter->tx_ring[skb->queue_mapping]; >>>>> >>>>> /* need: 1 descriptor per page * >>>>> PAGE_SIZE/IXGBE_MAX_DATA_PER_TXD, >>>>> -- >>>>> 1.7.12 >>>>> >>>> >>>> . >>>> >>> >> >> . >> >
On 2016/12/21 10:20, Alexander Duyck wrote: > I find it curious that only the last 4 bytes have data in them. I'm > wondering if the NIC/driver in the Windows/Nessus system is > interpreting the 4 byte CRC on the end of the frame as padding instead > of stripping it. > > Is there any chance you could capture the entire frame instead of just > the padding? Maybe you could run something like wireshark without > enabling promiscuous mode on the VF and capture the frames it is > trying to send and receive. What I want to verify is what the actual > amount of padding is that is needed to get to 60 bytes and where the > CRC should start. > > - Alex Here is the verbose output, is this useful? Or we will try according to your advice, thanks, D:\Program Files\Tenable\Nessus>nasl.exe -aX -t 192.169.0.151 etherleak.nasl -------------------------- ---[ ICMP ]--- 0x00: 45 00 00 1D 20 81 00 00 40 01 D7 F3 C0 A9 00 97 E... ...@....... 0x10: C0 A9 00 82 00 00 87 FD 00 01 00 01 78 00 00 00 ............x... 0x20: 00 00 00 00 00 00 00 00 00 00 98 E4 75 DF ............u. -------------------------- ---[ ICMP ]--- 0x00: 45 00 00 1D 20 85 00 00 40 01 D7 EF C0 A9 00 97 E... ...@....... 0x10: C0 A9 00 82 00 00 87 FD 00 01 00 01 78 00 00 00 ............x... 0x20: 00 00 00 00 00 00 00 00 00 00 FB DA F8 13 .............. ---[ ether1 ]--- 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 98 E4 75 ...............u 0x10: DF . ---[ ether2 ]--- 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 FB DA F8 ................ 0x10: 13 . Padding observed in one frame : 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 98 E4 75 ...............u 0x10: DF . Padding observed in another frame : 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 FB DA F8 ................ 0x10: 13
Yes that is much more helpful. So looking at it things are being padded but the last 4 bytes always have this extra data in them. I've been trying to recreate the issue on an 82599 with an SR-IOV VF and I haven't been having much luck reproducing the problem. In your test environment is the 82599 connected directly to the Windows machine or are there any switches/routers/gateways/tunnels/vlans in between? I've tried several iterations but with the 82599 connected directly to another NIC I have I am not able to get it to produce the garbage padding you are seeing. It makes me wonder if there might be something that is manipulating the packets in between the two systems. For example if there was a VLAN being associated with the VF that is later stripped and then the packet handed raw to the test system it might explain what is introducing the extra padding and reason for pulling in the CRC, and your patch would mask the issue since it would push the minimum frame size with a VLAN to 68 instead of 64. - Alex On Wed, Dec 21, 2016 at 6:00 PM, Kefeng Wang <wangkefeng.wang@huawei.com> wrote: > > > On 2016/12/21 10:20, Alexander Duyck wrote: >> I find it curious that only the last 4 bytes have data in them. I'm >> wondering if the NIC/driver in the Windows/Nessus system is >> interpreting the 4 byte CRC on the end of the frame as padding instead >> of stripping it. >> >> Is there any chance you could capture the entire frame instead of just >> the padding? Maybe you could run something like wireshark without >> enabling promiscuous mode on the VF and capture the frames it is >> trying to send and receive. What I want to verify is what the actual >> amount of padding is that is needed to get to 60 bytes and where the >> CRC should start. >> >> - Alex > > Here is the verbose output, is this useful? > Or we will try according to your advice, thanks, > > D:\Program Files\Tenable\Nessus>nasl.exe -aX -t 192.169.0.151 etherleak.nasl > -------------------------- > ---[ ICMP ]--- > 0x00: 45 00 00 1D 20 81 00 00 40 01 D7 F3 C0 A9 00 97 E... ...@....... > 0x10: C0 A9 00 82 00 00 87 FD 00 01 00 01 78 00 00 00 ............x... > 0x20: 00 00 00 00 00 00 00 00 00 00 98 E4 75 DF ............u. > -------------------------- > ---[ ICMP ]--- > 0x00: 45 00 00 1D 20 85 00 00 40 01 D7 EF C0 A9 00 97 E... ...@....... > 0x10: C0 A9 00 82 00 00 87 FD 00 01 00 01 78 00 00 00 ............x... > 0x20: 00 00 00 00 00 00 00 00 00 00 FB DA F8 13 .............. > ---[ ether1 ]--- > 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 98 E4 75 ...............u > 0x10: DF . > ---[ ether2 ]--- > 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 FB DA F8 ................ > 0x10: 13 . > > Padding observed in one frame : > > 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 98 E4 75 ...............u > 0x10: DF . > > Padding observed in another frame : > > 0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 FB DA F8 ................ > 0x10: 13 > >
diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c index 6d4bef5..137a154 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c @@ -3654,6 +3654,13 @@ static int ixgbevf_xmit_frame(struct sk_buff *skb, struct net_device *netdev) return NETDEV_TX_OK; } + /* On PCI/PCI-X HW, if packet size is less than ETH_ZLEN, + * packets may get corrupted during padding by HW. + * To WA this issue, pad all small packets manually. + */ + if (eth_skb_pad(skb)) + return NETDEV_TX_OK; + tx_ring = adapter->tx_ring[skb->queue_mapping]; /* need: 1 descriptor per page * PAGE_SIZE/IXGBE_MAX_DATA_PER_TXD,
Nessus report the vf appears to leak memory in network packets. Fix this by padding all small packets manually. And the CVE-2003-0001. https://ofirarkin.files.wordpress.com/2008/11/atstake_etherleak_report.pdf Signed-off-by: Weilong Chen <chenweilong@huawei.com> --- drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 7 +++++++ 1 file changed, 7 insertions(+)