From patchwork Wed Jan 6 19:15:31 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Khlebnikov X-Patchwork-Id: 564062 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3EDB61402B4 for ; Thu, 7 Jan 2016 06:15:43 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b=n3j5W357; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752227AbcAFTPg (ORCPT ); Wed, 6 Jan 2016 14:15:36 -0500 Received: from mail-lf0-f54.google.com ([209.85.215.54]:34318 "EHLO mail-lf0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751653AbcAFTPd (ORCPT ); Wed, 6 Jan 2016 14:15:33 -0500 Received: by mail-lf0-f54.google.com with SMTP id y184so318795014lfc.1; Wed, 06 Jan 2016 11:15:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=smoHW3UUmjZkwffctmxOukXlaGNhQI4JRNigsiHcOHI=; b=n3j5W357AUD26j6SscNVdoNBwgdqJBnM12fpBro8NTEw8szPuW13Og6dK8lzhyhdc9 Q2CxMBuPuHkGeGIysXXVRa3D4z3CpcBWJbivb2vyu8bQzgBr+uNuQG7vYFTKtIyGDZzX vNoeForOSWw3pQo5qRZMssFyGhMLkGKaAX+/IHEC1UmJ7MevRp6bMlEKF2l2l1HZOUJe n3kC9hmvYudP0R3NUBpwP9TPlIOgBsZUoE9tYt3MOiYMuTLUY7eaDT2M616t9JlLBJFd r1mQkSCDK3U/M8T04FLZ+oGTmrl6WKgrMJlyUcCov/h4ccNvy9fpvPmzFENbbkJKTCzG 7tIw== MIME-Version: 1.0 X-Received: by 10.25.159.9 with SMTP id i9mr29802686lfe.109.1452107731693; Wed, 06 Jan 2016 11:15:31 -0800 (PST) Received: by 10.112.110.102 with HTTP; Wed, 6 Jan 2016 11:15:31 -0800 (PST) Date: Wed, 6 Jan 2016 22:15:31 +0300 Message-ID: Subject: [BUG] skb corruption and kernel panic at forwarding with fragmentation From: Konstantin Khlebnikov To: netdev@vger.kernel.org, David Miller , Eric Dumazet , Linux Kernel Mailing List Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org I've got some of these: [84408.314676] BUG: unable to handle kernel NULL pointer dereference at (null) [84408.317324] IP: [] put_page+0x5/0x50 [84408.319985] PGD 0 [84408.322583] Oops: 0000 [#1] SMP [84408.325156] Modules linked in: ppp_mppe ppp_async ppp_generic slhc 8021q fuse nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace sunrpc bridge stp llc xt_HL xt_TCPMSS xt_state w83627ehf hwmon_vid snd_hda_codec_realtek snd_hda_codec_generic radeon snd_hda_codec_hdmi snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_hda_core edac_core k10temp snd_timer snd drm_kms_helper soundcore ath9k ttm ath9k_common ath9k_hw ath r8169 mii [84408.336804] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.1.15-zurg #1 [84408.339839] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./RS880D, BIOS 080015 04/12/2011 [84408.342964] task: ffff880216d56f50 ti: ffff880216e04000 task.ti: ffff880216e04000 [84408.346136] RIP: 0010:[] [] put_page+0x5/0x50 [84408.349301] RSP: 0018:ffff88021fcc37c0 EFLAGS: 00010216 [84408.352433] RAX: 0000000000000030 RBX: 0000000000000001 RCX: 0000000000000077 [84408.355602] RDX: ffff880213d8818e RSI: 0000000000000200 RDI: 0000000000000000 [84408.358765] RBP: ffff88021fcc37e8 R08: 0000000000000076 R09: ffff880216c01900 [84408.361885] R10: ffffea000859a840 R11: 0000000000000001 R12: ffff8802166a1300 [84408.364988] R13: ffff88021280d8c0 R14: ffff8802166a1300 R15: ffff88021280d410 [84408.368059] FS: 00007f9ada2de700(0000) GS:ffff88021fcc0000(0000) knlGS:0000000000000000 [84408.371211] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [84408.374336] CR2: 0000000000000000 CR3: 0000000216575000 CR4: 00000000000006e0 [84408.377484] Stack: [84408.380623] ffffffff81576ac8 ffff88021fcc37e8 ffff8802166a1300 ffff8802166a1300 [84408.383843] 0000000000000000 ffff88021fcc3808 ffffffff81576b48 ffff88021fcc3808 [84408.387022] 0000000000006e00 ffff88021fcc3828 ffffffff81576cdd 0000000000006e00 [84408.390187] Call Trace: [84408.393293] [84408.393323] [] ? skb_release_data+0x78/0xd0 [84408.399488] [] skb_release_all+0x28/0x30 [84408.402553] [] consume_skb+0x5d/0x80 [84408.405630] [] ip_fragment+0x5c4/0x970 [84408.408676] [] ? ip_copy_metadata+0x160/0x160 [84408.411733] [] ip_finish_output+0x601/0x900 [84408.414788] [] ? nf_hook_slow+0x99/0x100 [84408.417828] [] ip_output+0x66/0xc0 [84408.420847] [] ? ip_fragment+0x970/0x970 [84408.423864] [] ip_forward_finish+0x73/0xa0 [84408.426864] [] ip_forward+0x3af/0x490 [84408.429833] [] ? ip_frag_mem+0x50/0x50 [84408.432782] [] ip_rcv_finish+0x81/0x370 [84408.435778] [] ip_rcv+0x2a2/0x3c0 [84408.438780] [] ? inet_del_offload+0x40/0x40 [84408.441780] [] __netif_receive_skb_core+0x673/0x810 [84408.444785] [] __netif_receive_skb+0x18/0x60 [84408.447766] [] netif_receive_skb_internal+0x23/0x90 [84408.450739] [] netif_receive_skb_sk+0x1c/0x70 [84408.453726] [] br_handle_frame_finish+0x27c/0x520 [bridge] [84408.456774] [] ? ipv4_confirm+0xb8/0xe0 [84408.459787] [] br_handle_frame+0x161/0x290 [bridge] [84408.462803] [] ? ip_local_deliver+0x46/0xa0 [84408.465796] [] __netif_receive_skb_core+0x32e/0x810 [84408.468822] [] __netif_receive_skb+0x18/0x60 [84408.471748] [] netif_receive_skb_internal+0x23/0x90 [84408.474615] [] ? tcp4_gro_complete+0x73/0x80 [84408.477378] [] napi_gro_complete+0x9c/0xe0 [84408.480045] [] dev_gro_receive+0x230/0x360 [84408.482675] [] napi_gro_receive+0x30/0x100 [84408.485240] [] rtl8169_poll+0x2c6/0x6b0 [r8169] [84408.487766] [] net_rx_action+0x1fa/0x320 [84408.490241] [] __do_softirq+0x10b/0x2d0 [84408.492672] [] irq_exit+0xd5/0xe0 [84408.495072] [] do_IRQ+0x58/0xf0 [84408.497463] [] common_interrupt+0x6e/0x6e [84408.499879] [84408.499909] [] ? native_safe_halt+0x6/0x10 [84408.504697] [] ? tick_broadcast_oneshot_control+0xbe/0x200 [84408.507126] [] default_idle+0x1e/0xc0 [84408.509516] [] amd_e400_idle+0x6e/0xf0 [84408.511879] [] arch_cpu_idle+0xf/0x20 [84408.514181] [] cpu_startup_entry+0x327/0x3a0 [84408.516456] [] ? clockevents_register_device+0xec/0x1d0 [84408.518760] [] start_secondary+0x138/0x160 [84408.521066] Code: 48 89 d7 e8 2e f7 ff ff e9 a1 fe ff ff 48 89 d7 e8 51 f7 ff ff e9 94 fe ff ff 66 90 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <48> f7 07 00 c0 00 00 55 48 89 e5 75 1e 8b 47 1c 85 c0 74 27 f0 [84408.526216] RIP [] put_page+0x5/0x50 [84408.528705] RSP [84408.531178] CR2: 0000000000000000 Looks like this happens because ip_options_fragment() relies on correct ip options length in ip control block in skb. But in ip_finish_output_gso() control block in segments is reused by skb_gso_segment(). following ip_fragment() sees some garbage. In my case there was no ip options but length becomes non-zero and ip_options_fragment() picked some bytes from payload and decides to fill huge range with IPOPT_NOOP (1). One of that ones flipped nr_frags in skb_shared_info at the end of data =) Here is quick hack: just make room for ip control block in gso control block. --- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3316,6 +3316,7 @@ static inline struct sec_path *skb_sec_path(struct sk_buff *skb) * Keeps track of level of encapsulation of network headers. */ struct skb_gso_cb { + char pad[32]; /* inet_skb_parm lives here */ int mac_offset; int encap_level; __u16 csum_start; And debug which prevents kernel crash too. --- a/net/ipv4/ip_options.c +++ b/net/ipv4/ip_options.c @@ -215,6 +215,10 @@ void ip_options_fragment(struct sk_buff *skb) int l = opt->optlen; int optlen; + const struct iphdr *iph = ip_hdr(skb); + l = iph->ihl * 4 - sizeof(struct iphdr); + WARN(opt->optlen != l, "%s %d != %d\n", __func__, opt->optlen, l); + while (l > 0) { switch (*optptr) { case IPOPT_END: