mbox series

[0/1,SRU,O,N,M,0/1] Fix UBSAN array-index-out-of-bounds in bcache/bset.c

Message ID 20240701215450.812112-1-ghadi.rahme@canonical.com
Headers show
Series Fix UBSAN array-index-out-of-bounds in bcache/bset.c | expand

Message

Ghadi Elie Rahme July 1, 2024, 9:54 p.m. UTC
BugLink: https://bugs.launchpad.net/bugs/2039368

[Impact]

Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.

[Fix]

There is currently a fix upstream provided by the following patch:

* 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"

[Test Case]

1. Setup bcache on a jammy HWE kernel or mantic or noble machine. This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
2. Restart the machine
3. After restarting the machine, the following UBSAN warnings and call traces can be seen in dmesg:

[ 3.824281] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1098:3
[ 3.826338] index 4 is out of range for type 'btree_iter_set [4]'
[ 3.826812] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 3.827817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 3.828835] Workqueue: events register_cache_worker [bcache]
[ 3.829429] Call Trace:
[ 3.830626] <TASK>
[ 3.831638] dump_stack_lvl+0x48/0x70
[ 3.832227] dump_stack+0x10/0x20
[ 3.832785] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 3.833357] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
[ 3.834052] bch_btree_node_read_done+0xfc/0x450 [bcache]
[ 3.834653] ? mempool_kfree+0xe/0x20
[ 3.835211] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 3.835832] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 3.836474] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 3.837161] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 3.837838] ? __pfx_up_write+0x10/0x10
[ 3.838739] bch_btree_node_get+0x16/0x30 [bcache]
[ 3.844949] </TASK>

[ 4.029242] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1207:3
[ 4.030496] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.030930] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.031841] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.032650] Workqueue: events register_cache_worker [bcache]
[ 4.033149] Call Trace:
[ 4.033549] <TASK>
[ 4.033972] dump_stack_lvl+0x48/0x70
[ 4.034418] dump_stack+0x10/0x20
[ 4.034839] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.035279] btree_mergesort+0x4d4/0x520 [bcache]
[ 4.035730] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.036191] ? __pfx_bch_extent_sort_cmp+0x10/0x10 [bcache]
[ 4.036691] __btree_sort+0x96/0x2d0 [bcache]
[ 4.037182] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.037674] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.038172] ? mempool_kfree+0xe/0x20
[ 4.038617] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.039120] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.039659] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.040220] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.040806] ? __pfx_up_write+0x10/0x10
[ 4.041371] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.048339] </TASK>

[ 4.227653] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:281:4
[ 4.228847] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.229472] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.230680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.231954] Workqueue: events register_cache_worker [bcache]
[ 4.232690] Call Trace:
[ 4.233327] <TASK>
[ 4.233935] dump_stack_lvl+0x48/0x70
[ 4.234568] dump_stack+0x10/0x20
[ 4.235219] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.235833] bch_extent_sort_fixup+0xb95/0xd70 [bcache]
[ 4.236524] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.237159] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.237839] btree_mergesort+0x221/0x520 [bcache]
[ 4.238823] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.239800] __btree_sort+0x96/0x2d0 [bcache]
[ 4.240880] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.243046] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.245223] ? mempool_kfree+0xe/0x20
[ 4.246311] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.247410] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.248471] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.248959] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.249454] ? __pfx_up_write+0x10/0x10
[ 4.249904] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.255145] </TASK>

[ 4.257388] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:36:18
[ 4.258429] index 14 is out of range for type 'btree_iter_set [4]'
[ 4.258964] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.260073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.261188] Workqueue: events register_cache_worker [bcache]
[ 4.261811] Call Trace:
[ 4.262374] <TASK>
[ 4.262912] dump_stack_lvl+0x48/0x70
[ 4.263502] dump_stack+0x10/0x20
[ 4.264042] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.264605] bch_extent_sort_fixup+0xbe5/0xd70 [bcache]
[ 4.265218] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.265821] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.266514] btree_mergesort+0x221/0x520 [bcache]
[ 4.267234] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.267882] __btree_sort+0x96/0x2d0 [bcache]
[ 4.268508] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.269144] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.269825] ? mempool_kfree+0xe/0x20
[ 4.270489] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.271243] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.272293] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.273260] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.274182] ? __pfx_up_write+0x10/0x10
[ 4.274973] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.284807] </TASK>

[ 4.286129] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:291:4
[ 4.286791] index 4 is out of range for type 'btree_iter_set [4]'
[ 4.287231] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
[ 4.288033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 4.288863] Workqueue: events register_cache_worker [bcache]
[ 4.289340] Call Trace:
[ 4.289753] <TASK>
[ 4.290168] dump_stack_lvl+0x48/0x70
[ 4.290581] dump_stack+0x10/0x20
[ 4.290984] __ubsan_handle_out_of_bounds+0xc6/0x110
[ 4.291432] bch_extent_sort_fixup+0xb77/0xd70 [bcache]
[ 4.291882] ? __ubsan_handle_out_of_bounds+0xee/0x110
[ 4.292309] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.292764] btree_mergesort+0x221/0x520 [bcache]
[ 4.293225] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
[ 4.293683] __btree_sort+0x96/0x2d0 [bcache]
[ 4.294153] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
[ 4.294631] bch_btree_node_read_done+0x34d/0x450 [bcache]
[ 4.295175] ? mempool_kfree+0xe/0x20
[ 4.295671] bch_btree_node_read+0xf8/0x1e0 [bcache]
[ 4.296257] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
[ 4.296834] bch_btree_node_get.part.0+0x160/0x340 [bcache]
[ 4.297446] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
[ 4.298087] ? __pfx_up_write+0x10/0x10
[ 4.298678] bch_btree_node_get+0x16/0x30 [bcache]
[ 4.306037] </TASK>

[Where problems could occur]

* The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.

Matthew Mirvish (1):
  bcache: fix variable length array abuse in btree_iter

 drivers/md/bcache/bset.c      | 44 +++++++++++++++++------------------
 drivers/md/bcache/bset.h      | 28 ++++++++++++++--------
 drivers/md/bcache/btree.c     | 40 ++++++++++++++++---------------
 drivers/md/bcache/super.c     |  5 ++--
 drivers/md/bcache/sysfs.c     |  2 +-
 drivers/md/bcache/writeback.c | 10 ++++----
 6 files changed, 70 insertions(+), 59 deletions(-)

Comments

Stefan Bader July 2, 2024, 12:01 p.m. UTC | #1
On 01.07.24 23:54, Ghadi Elie Rahme wrote:
> BugLink: https://bugs.launchpad.net/bugs/2039368
> 
> [Impact]
> 
> Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
> 
> [Fix]
> 
> There is currently a fix upstream provided by the following patch:
> 
> * 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"

The patch itself needs to have this added:

(cherry picked from commit 3a861560ccb35f2a4f0a4b8207fa7c2a35fc7f31)

For Mantic this is a NACK because it is close to EOL and the issue seems 
to have no observed breakage/effect. For Oracular it is a NACK as this 
change is part of v6.10-rc1 and we are close to move to that version there.

Acked-by: Stefan Bader <stefan.bader@canonical.com>
> 
> [Test Case]
> 
> 1. Setup bcache on a jammy HWE kernel or mantic or noble machine. This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
> 2. Restart the machine
> 3. After restarting the machine, the following UBSAN warnings and call traces can be seen in dmesg:
> 
> [ 3.824281] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1098:3
> [ 3.826338] index 4 is out of range for type 'btree_iter_set [4]'
> [ 3.826812] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 3.827817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 3.828835] Workqueue: events register_cache_worker [bcache]
> [ 3.829429] Call Trace:
> [ 3.830626] <TASK>
> [ 3.831638] dump_stack_lvl+0x48/0x70
> [ 3.832227] dump_stack+0x10/0x20
> [ 3.832785] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 3.833357] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
> [ 3.834052] bch_btree_node_read_done+0xfc/0x450 [bcache]
> [ 3.834653] ? mempool_kfree+0xe/0x20
> [ 3.835211] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 3.835832] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 3.836474] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 3.837161] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 3.837838] ? __pfx_up_write+0x10/0x10
> [ 3.838739] bch_btree_node_get+0x16/0x30 [bcache]
> [ 3.844949] </TASK>
> 
> [ 4.029242] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1207:3
> [ 4.030496] index 14 is out of range for type 'btree_iter_set [4]'
> [ 4.030930] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 4.031841] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 4.032650] Workqueue: events register_cache_worker [bcache]
> [ 4.033149] Call Trace:
> [ 4.033549] <TASK>
> [ 4.033972] dump_stack_lvl+0x48/0x70
> [ 4.034418] dump_stack+0x10/0x20
> [ 4.034839] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 4.035279] btree_mergesort+0x4d4/0x520 [bcache]
> [ 4.035730] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.036191] ? __pfx_bch_extent_sort_cmp+0x10/0x10 [bcache]
> [ 4.036691] __btree_sort+0x96/0x2d0 [bcache]
> [ 4.037182] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
> [ 4.037674] bch_btree_node_read_done+0x34d/0x450 [bcache]
> [ 4.038172] ? mempool_kfree+0xe/0x20
> [ 4.038617] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 4.039120] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 4.039659] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 4.040220] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 4.040806] ? __pfx_up_write+0x10/0x10
> [ 4.041371] bch_btree_node_get+0x16/0x30 [bcache]
> [ 4.048339] </TASK>
> 
> [ 4.227653] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:281:4
> [ 4.228847] index 4 is out of range for type 'btree_iter_set [4]'
> [ 4.229472] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 4.230680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 4.231954] Workqueue: events register_cache_worker [bcache]
> [ 4.232690] Call Trace:
> [ 4.233327] <TASK>
> [ 4.233935] dump_stack_lvl+0x48/0x70
> [ 4.234568] dump_stack+0x10/0x20
> [ 4.235219] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 4.235833] bch_extent_sort_fixup+0xb95/0xd70 [bcache]
> [ 4.236524] ? __ubsan_handle_out_of_bounds+0xee/0x110
> [ 4.237159] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.237839] btree_mergesort+0x221/0x520 [bcache]
> [ 4.238823] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.239800] __btree_sort+0x96/0x2d0 [bcache]
> [ 4.240880] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
> [ 4.243046] bch_btree_node_read_done+0x34d/0x450 [bcache]
> [ 4.245223] ? mempool_kfree+0xe/0x20
> [ 4.246311] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 4.247410] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 4.248471] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 4.248959] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 4.249454] ? __pfx_up_write+0x10/0x10
> [ 4.249904] bch_btree_node_get+0x16/0x30 [bcache]
> [ 4.255145] </TASK>
> 
> [ 4.257388] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:36:18
> [ 4.258429] index 14 is out of range for type 'btree_iter_set [4]'
> [ 4.258964] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 4.260073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 4.261188] Workqueue: events register_cache_worker [bcache]
> [ 4.261811] Call Trace:
> [ 4.262374] <TASK>
> [ 4.262912] dump_stack_lvl+0x48/0x70
> [ 4.263502] dump_stack+0x10/0x20
> [ 4.264042] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 4.264605] bch_extent_sort_fixup+0xbe5/0xd70 [bcache]
> [ 4.265218] ? __ubsan_handle_out_of_bounds+0xee/0x110
> [ 4.265821] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.266514] btree_mergesort+0x221/0x520 [bcache]
> [ 4.267234] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.267882] __btree_sort+0x96/0x2d0 [bcache]
> [ 4.268508] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
> [ 4.269144] bch_btree_node_read_done+0x34d/0x450 [bcache]
> [ 4.269825] ? mempool_kfree+0xe/0x20
> [ 4.270489] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 4.271243] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 4.272293] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 4.273260] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 4.274182] ? __pfx_up_write+0x10/0x10
> [ 4.274973] bch_btree_node_get+0x16/0x30 [bcache]
> [ 4.284807] </TASK>
> 
> [ 4.286129] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:291:4
> [ 4.286791] index 4 is out of range for type 'btree_iter_set [4]'
> [ 4.287231] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 4.288033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 4.288863] Workqueue: events register_cache_worker [bcache]
> [ 4.289340] Call Trace:
> [ 4.289753] <TASK>
> [ 4.290168] dump_stack_lvl+0x48/0x70
> [ 4.290581] dump_stack+0x10/0x20
> [ 4.290984] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 4.291432] bch_extent_sort_fixup+0xb77/0xd70 [bcache]
> [ 4.291882] ? __ubsan_handle_out_of_bounds+0xee/0x110
> [ 4.292309] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.292764] btree_mergesort+0x221/0x520 [bcache]
> [ 4.293225] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.293683] __btree_sort+0x96/0x2d0 [bcache]
> [ 4.294153] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
> [ 4.294631] bch_btree_node_read_done+0x34d/0x450 [bcache]
> [ 4.295175] ? mempool_kfree+0xe/0x20
> [ 4.295671] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 4.296257] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 4.296834] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 4.297446] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 4.298087] ? __pfx_up_write+0x10/0x10
> [ 4.298678] bch_btree_node_get+0x16/0x30 [bcache]
> [ 4.306037] </TASK>
> 
> [Where problems could occur]
> 
> * The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
> 
> Matthew Mirvish (1):
>    bcache: fix variable length array abuse in btree_iter
> 
>   drivers/md/bcache/bset.c      | 44 +++++++++++++++++------------------
>   drivers/md/bcache/bset.h      | 28 ++++++++++++++--------
>   drivers/md/bcache/btree.c     | 40 ++++++++++++++++---------------
>   drivers/md/bcache/super.c     |  5 ++--
>   drivers/md/bcache/sysfs.c     |  2 +-
>   drivers/md/bcache/writeback.c | 10 ++++----
>   6 files changed, 70 insertions(+), 59 deletions(-)
>
Paolo Pisati July 2, 2024, 12:42 p.m. UTC | #2
On Tue, Jul 02, 2024 at 12:54:49AM +0300, Ghadi Elie Rahme wrote:
> BugLink: https://bugs.launchpad.net/bugs/2039368

Acked-by: Paolo Pisati <paolo.pisati@canonical.com>
Stefan Bader July 4, 2024, 6:32 p.m. UTC | #3
On 01.07.24 23:54, Ghadi Elie Rahme wrote:
> BugLink: https://bugs.launchpad.net/bugs/2039368
> 
> [Impact]
> 
> Currently there are UBSAN warnings that show up when running bcache on jammy HWE, Mantic and noble. For now no side effects have been observed but such an issue could potentially cause a crash or corrupt data.
> 
> [Fix]
> 
> There is currently a fix upstream provided by the following patch:
> 
> * 3a861560ccb3 "bcache: fix variable length array abuse in btree_iter"
> 
> [Test Case]
> 
> 1. Setup bcache on a jammy HWE kernel or mantic or noble machine. This can be done following the steps in this wiki: https://wiki.ubuntu.com/ServerTeam/Bcache
> 2. Restart the machine
> 3. After restarting the machine, the following UBSAN warnings and call traces can be seen in dmesg:
> 
> [ 3.824281] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1098:3
> [ 3.826338] index 4 is out of range for type 'btree_iter_set [4]'
> [ 3.826812] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 3.827817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 3.828835] Workqueue: events register_cache_worker [bcache]
> [ 3.829429] Call Trace:
> [ 3.830626] <TASK>
> [ 3.831638] dump_stack_lvl+0x48/0x70
> [ 3.832227] dump_stack+0x10/0x20
> [ 3.832785] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 3.833357] bch_btree_iter_push+0x4e6/0x4f0 [bcache]
> [ 3.834052] bch_btree_node_read_done+0xfc/0x450 [bcache]
> [ 3.834653] ? mempool_kfree+0xe/0x20
> [ 3.835211] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 3.835832] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 3.836474] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 3.837161] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 3.837838] ? __pfx_up_write+0x10/0x10
> [ 3.838739] bch_btree_node_get+0x16/0x30 [bcache]
> [ 3.844949] </TASK>
> 
> [ 4.029242] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/bset.c:1207:3
> [ 4.030496] index 14 is out of range for type 'btree_iter_set [4]'
> [ 4.030930] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 4.031841] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 4.032650] Workqueue: events register_cache_worker [bcache]
> [ 4.033149] Call Trace:
> [ 4.033549] <TASK>
> [ 4.033972] dump_stack_lvl+0x48/0x70
> [ 4.034418] dump_stack+0x10/0x20
> [ 4.034839] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 4.035279] btree_mergesort+0x4d4/0x520 [bcache]
> [ 4.035730] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.036191] ? __pfx_bch_extent_sort_cmp+0x10/0x10 [bcache]
> [ 4.036691] __btree_sort+0x96/0x2d0 [bcache]
> [ 4.037182] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
> [ 4.037674] bch_btree_node_read_done+0x34d/0x450 [bcache]
> [ 4.038172] ? mempool_kfree+0xe/0x20
> [ 4.038617] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 4.039120] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 4.039659] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 4.040220] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 4.040806] ? __pfx_up_write+0x10/0x10
> [ 4.041371] bch_btree_node_get+0x16/0x30 [bcache]
> [ 4.048339] </TASK>
> 
> [ 4.227653] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:281:4
> [ 4.228847] index 4 is out of range for type 'btree_iter_set [4]'
> [ 4.229472] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 4.230680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 4.231954] Workqueue: events register_cache_worker [bcache]
> [ 4.232690] Call Trace:
> [ 4.233327] <TASK>
> [ 4.233935] dump_stack_lvl+0x48/0x70
> [ 4.234568] dump_stack+0x10/0x20
> [ 4.235219] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 4.235833] bch_extent_sort_fixup+0xb95/0xd70 [bcache]
> [ 4.236524] ? __ubsan_handle_out_of_bounds+0xee/0x110
> [ 4.237159] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.237839] btree_mergesort+0x221/0x520 [bcache]
> [ 4.238823] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.239800] __btree_sort+0x96/0x2d0 [bcache]
> [ 4.240880] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
> [ 4.243046] bch_btree_node_read_done+0x34d/0x450 [bcache]
> [ 4.245223] ? mempool_kfree+0xe/0x20
> [ 4.246311] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 4.247410] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 4.248471] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 4.248959] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 4.249454] ? __pfx_up_write+0x10/0x10
> [ 4.249904] bch_btree_node_get+0x16/0x30 [bcache]
> [ 4.255145] </TASK>
> 
> [ 4.257388] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:36:18
> [ 4.258429] index 14 is out of range for type 'btree_iter_set [4]'
> [ 4.258964] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 4.260073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 4.261188] Workqueue: events register_cache_worker [bcache]
> [ 4.261811] Call Trace:
> [ 4.262374] <TASK>
> [ 4.262912] dump_stack_lvl+0x48/0x70
> [ 4.263502] dump_stack+0x10/0x20
> [ 4.264042] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 4.264605] bch_extent_sort_fixup+0xbe5/0xd70 [bcache]
> [ 4.265218] ? __ubsan_handle_out_of_bounds+0xee/0x110
> [ 4.265821] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.266514] btree_mergesort+0x221/0x520 [bcache]
> [ 4.267234] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.267882] __btree_sort+0x96/0x2d0 [bcache]
> [ 4.268508] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
> [ 4.269144] bch_btree_node_read_done+0x34d/0x450 [bcache]
> [ 4.269825] ? mempool_kfree+0xe/0x20
> [ 4.270489] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 4.271243] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 4.272293] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 4.273260] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 4.274182] ? __pfx_up_write+0x10/0x10
> [ 4.274973] bch_btree_node_get+0x16/0x30 [bcache]
> [ 4.284807] </TASK>
> 
> [ 4.286129] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-QmAt2N/linux-hwe-6.5-6.5.0/drivers/md/bcache/extents.c:291:4
> [ 4.286791] index 4 is out of range for type 'btree_iter_set [4]'
> [ 4.287231] CPU: 13 PID: 184 Comm: kworker/13:1 Not tainted 6.5.0-41-generic #41~22.04.2-Ubuntu
> [ 4.288033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 4.288863] Workqueue: events register_cache_worker [bcache]
> [ 4.289340] Call Trace:
> [ 4.289753] <TASK>
> [ 4.290168] dump_stack_lvl+0x48/0x70
> [ 4.290581] dump_stack+0x10/0x20
> [ 4.290984] __ubsan_handle_out_of_bounds+0xc6/0x110
> [ 4.291432] bch_extent_sort_fixup+0xb77/0xd70 [bcache]
> [ 4.291882] ? __ubsan_handle_out_of_bounds+0xee/0x110
> [ 4.292309] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.292764] btree_mergesort+0x221/0x520 [bcache]
> [ 4.293225] ? __pfx_bch_ptr_invalid+0x10/0x10 [bcache]
> [ 4.293683] __btree_sort+0x96/0x2d0 [bcache]
> [ 4.294153] bch_btree_sort_and_fix_extents+0x1d/0x40 [bcache]
> [ 4.294631] bch_btree_node_read_done+0x34d/0x450 [bcache]
> [ 4.295175] ? mempool_kfree+0xe/0x20
> [ 4.295671] bch_btree_node_read+0xf8/0x1e0 [bcache]
> [ 4.296257] ? __pfx_closure_sync_fn+0x10/0x10 [bcache]
> [ 4.296834] bch_btree_node_get.part.0+0x160/0x340 [bcache]
> [ 4.297446] ? __bch_btree_ptr_invalid+0x60/0xd0 [bcache]
> [ 4.298087] ? __pfx_up_write+0x10/0x10
> [ 4.298678] bch_btree_node_get+0x16/0x30 [bcache]
> [ 4.306037] </TASK>
> 
> [Where problems could occur]
> 
> * The patch modifies the way bcache allocates space to the btree iterator. The main problems that could occur are different UBSAN warnings showing up that could possibly trigger a crash much easier than the current array index-out-of-bounds being observed.
> 
> Matthew Mirvish (1):
>    bcache: fix variable length array abuse in btree_iter
> 
>   drivers/md/bcache/bset.c      | 44 +++++++++++++++++------------------
>   drivers/md/bcache/bset.h      | 28 ++++++++++++++--------
>   drivers/md/bcache/btree.c     | 40 ++++++++++++++++---------------
>   drivers/md/bcache/super.c     |  5 ++--
>   drivers/md/bcache/sysfs.c     |  2 +-
>   drivers/md/bcache/writeback.c | 10 ++++----
>   6 files changed, 70 insertions(+), 59 deletions(-)
> 

Applied to noble:linux/master-next fixing up s-o-b. Thanks.

-Stefan