Message ID | 20230622165527.2417-1-quintela@redhat.com |
---|---|
State | New |
Headers | show |
On 6/22/23 18:54, Juan Quintela wrote: > The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb: > > Merge tag 'q800-for-8.1-pull-request' ofhttps://github.com/vivier/qemu-m68k into staging (2023-06-22 10:18:32 +0200) > > are available in the Git repository at: > > https://gitlab.com/juan.quintela/qemu.git tags/next-pull-request > > for you to fetch changes up to 23e4307eadc1497bd0a11ca91041768f15963b68: > > migration/rdma: Split qemu_fopen_rdma() into input/output functions (2023-06-22 18:11:58 +0200) > > ---------------------------------------------------------------- > Migration Pull request (20230621) take 2 > > In this pull request the only change is fixing 32 bits complitaion issue. > > Please apply. > > [take 1] > - fix for multifd thread creation (fabiano) > - dirtylimity (hyman) > * migration-test will go on next PULL request, as it has failures. > - Improve error description (tejus) > - improve -incoming and set parameters before calling incoming (wei) > - migration atomic counters reviewed patches (quintela) > - migration-test refacttoring reviewed (quintela) New failure with check-cfi-x86_64: https://gitlab.com/qemu-project/qemu/-/jobs/4527202764#L188 /builds/qemu-project/qemu/build/pyvenv/bin/meson test --no-rebuild -t 0 --num-processes 1 --print-errorlogs 1/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/qom-test OK 6.55s 8 subtests passed ▶ 2/350 ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: assertion failed: (bad == 0) ERROR 2/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test ERROR 151.99s killed by signal 6 SIGABRT >>> G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh MALLOC_PERTURB_=3 QTEST_QEMU_IMG=./qemu-img QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_BINARY=./qemu-system-x86_64 /builds/qemu-project/qemu/build/tests/qtest/migration-test --tap -k ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― stderr: qemu-system-x86_64: Unable to read from socket: Connection reset by peer Memory content inconsistency at 4f65000 first_byte = 30 last_byte = 2f current = 88 hit_edge = 1 ** ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: assertion failed: (bad == 0) (test program exited with status code -6) ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― r~
Richard Henderson <richard.henderson@linaro.org> wrote: > On 6/22/23 18:54, Juan Quintela wrote: >> The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb: >> Merge tag 'q800-for-8.1-pull-request' >> ofhttps://github.com/vivier/qemu-m68k into staging (2023-06-22 >> 10:18:32 +0200) >> are available in the Git repository at: >> https://gitlab.com/juan.quintela/qemu.git tags/next-pull-request >> for you to fetch changes up to >> 23e4307eadc1497bd0a11ca91041768f15963b68: >> migration/rdma: Split qemu_fopen_rdma() into input/output >> functions (2023-06-22 18:11:58 +0200) >> ---------------------------------------------------------------- >> Migration Pull request (20230621) take 2 >> In this pull request the only change is fixing 32 bits complitaion >> issue. >> Please apply. >> [take 1] >> - fix for multifd thread creation (fabiano) >> - dirtylimity (hyman) >> * migration-test will go on next PULL request, as it has failures. >> - Improve error description (tejus) >> - improve -incoming and set parameters before calling incoming (wei) >> - migration atomic counters reviewed patches (quintela) >> - migration-test refacttoring reviewed (quintela) I had the feeling when I wake up that today was going to be a great day. Confirmed. > New failure with check-cfi-x86_64: Aha. CFI. Something that I don't know what it is failing on me. /me googles. /me enables cfi+lto and compiles with clang. 50/491] Compiling C object subprojects/berkeley-testfloat-3/libtestfloat.a.p/source_test_az_f128_rx.c.o [51/491] Compiling C object subprojects/berkeley-testfloat-3/libtestfloat.a.p/source_test_az_f128.c.o [52/491] Compiling C object subprojects/berkeley-testfloat-3/libtestfloat.a.p/source_test_abz_f128.c.o [53/491] Compiling C object subprojects/berkeley-testfloat-3/libtestfloat.a.p/source_test_abcz_f128.c.o [54/491] Compiling C object subprojects/berkeley-testfloat-3/libtestfloat.a.p/source_test_ab_f128_z_bool.c.o [55/491] Linking target qemu-system-x86_64 FAILED: qemu-system-x86_64 clang++ -m64 -mcx16 @qemu-system-x86_64.rsp /usr/bin/ld: cannot find libchardev.fa: Too many open files /usr/bin/ld: cannot find libqmp.fa: Too many open files /usr/bin/ld: cannot find libpage-vary-common.a: Too many open files /usr/bin/ld: cannot find libqemuutil.a: Too many open files /usr/bin/ld: cannot find subprojects/libvhost-user/libvhost-user-glib.a: Too many open files /usr/bin/ld: cannot find subprojects/libvhost-user/libvhost-user.a: Too many open files /usr/bin/ld: cannot find tcg/libtcg_softmmu.fa: Too many open files /usr/bin/ld: cannot find libmigration.fa: Too many open files /usr/bin/ld: cannot find libhwcore.fa: Too many open files /usr/bin/ld: cannot find libqom.fa: Too many open files /usr/bin/ld: cannot find gdbstub/libgdb_softmmu.fa: Too many open files /usr/bin/ld: cannot find libio.fa: Too many open files /usr/bin/ld: cannot find libcrypto.fa: Too many open files /usr/bin/ld: cannot find libauthz.fa: Too many open files /usr/bin/ld: cannot find libblockdev.fa: Too many open files /usr/bin/ld: cannot find libblock.fa: Too many open files /usr/bin/ld: cannot find libchardev.fa: Too many open files /usr/bin/ld: cannot find libqmp.fa: Too many open files /usr/bin/ld: cannot find /usr/lib64/libpixman-1.so: Too many open files /usr/bin/ld: cannot find /usr/lib64/libepoxy.so: Too many open files /usr/bin/ld: cannot find /usr/lib64/libxenctrl.so: Too many open files /usr/bin/ld: cannot find /usr/lib64/libxenstore.so: Too many open files /usr/bin/ld: cannot find /usr/lib64/libxenforeignmemory.so: Too many open files /usr/bin/ld: cannot find /usr/lib64/libxengnttab.so: Too many open files /usr/bin/ld: cannot find /usr/lib64/libxenevtchn.so: Too many open files Confirmed, today is going to be a great day. No check-cfi<anything> target for me. /me investigates what is going on. Found this and retries. AR=llvm-ar CC=clang CXX=clang++ /mnt/code/qemu/full/configure --enable-cfi --target-list=x86_64-softmmu Gives the same error. After a while of desesperation trying to disable features, etc, etc Just doing a plain ulimit -n 4096 fixed the problem. Here we go. Later, Juan.
Richard Henderson <richard.henderson@linaro.org> wrote: > On 6/22/23 18:54, Juan Quintela wrote: >> The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb: >> Merge tag 'q800-for-8.1-pull-request' >> ofhttps://github.com/vivier/qemu-m68k into staging (2023-06-22 >> 10:18:32 +0200) >> are available in the Git repository at: >> https://gitlab.com/juan.quintela/qemu.git tags/next-pull-request >> for you to fetch changes up to >> 23e4307eadc1497bd0a11ca91041768f15963b68: >> migration/rdma: Split qemu_fopen_rdma() into input/output >> functions (2023-06-22 18:11:58 +0200) >> ---------------------------------------------------------------- >> Migration Pull request (20230621) take 2 >> In this pull request the only change is fixing 32 bits complitaion >> issue. >> Please apply. >> [take 1] >> - fix for multifd thread creation (fabiano) >> - dirtylimity (hyman) >> * migration-test will go on next PULL request, as it has failures. >> - Improve error description (tejus) >> - improve -incoming and set parameters before calling incoming (wei) >> - migration atomic counters reviewed patches (quintela) >> - migration-test refacttoring reviewed (quintela) > > New failure with check-cfi-x86_64: > > https://gitlab.com/qemu-project/qemu/-/jobs/4527202764#L188 First of all, is there a way to get to the test log? In particular, I am interested in knowing at least what test has failed (yes, migration-test don't tell you much more). After a bit more wrestling, I have been able to get things compiling with this command: $ /mnt/code/qemu/full/configure --enable-cfi --target-list=x86_64-softmmu --enable-cfi-debug --cc=clang --cxx=clang++ --disable-docs --enable-safe-stack --disable-slirp It should basically be the one that check-cfi-x86_64 is using if I understand the build recipes correctly (that is a BIG IF). And it passes for me with flying colors. Here I have Fedora38, builder has F37. > /builds/qemu-project/qemu/build/pyvenv/bin/meson test --no-rebuild -t > 0 --num-processes 1 --print-errorlogs > 1/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/qom-test > OK 6.55s 8 subtests passed > ▶ 2/350 ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: > assertion failed: (bad == 0) ERROR > 2/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test > ERROR 151.99s killed by signal 6 SIGABRT >>>> > G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh > MALLOC_PERTURB_=3 QTEST_QEMU_IMG=./qemu-img > QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon > QTEST_QEMU_BINARY=./qemu-system-x86_64 > /builds/qemu-project/qemu/build/tests/qtest/migration-test --tap > -k > ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― > stderr: > qemu-system-x86_64: Unable to read from socket: Connection reset by peer This is the interesting bit, why is the conection closed. > Memory content inconsistency at 4f65000 first_byte = 30 last_byte = 2f > current = 88 hit_edge = 1 > ** > ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: assertion failed: (bad == 0) > > (test program exited with status code -6) This makes zero sense, except if we haven't migrated all the guest state, that it is what it has happened. Is there a place on the web interface to see the full logs? Or that is the only thing that the CI system stores? Later, Juan.
On 6/26/23 00:01, Juan Quintela wrote: > Richard Henderson <richard.henderson@linaro.org> wrote: >> On 6/22/23 18:54, Juan Quintela wrote: >>> The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb: >>> Merge tag 'q800-for-8.1-pull-request' >>> ofhttps://github.com/vivier/qemu-m68k into staging (2023-06-22 >>> 10:18:32 +0200) >>> are available in the Git repository at: >>> https://gitlab.com/juan.quintela/qemu.git tags/next-pull-request >>> for you to fetch changes up to >>> 23e4307eadc1497bd0a11ca91041768f15963b68: >>> migration/rdma: Split qemu_fopen_rdma() into input/output >>> functions (2023-06-22 18:11:58 +0200) >>> ---------------------------------------------------------------- >>> Migration Pull request (20230621) take 2 >>> In this pull request the only change is fixing 32 bits complitaion >>> issue. >>> Please apply. >>> [take 1] >>> - fix for multifd thread creation (fabiano) >>> - dirtylimity (hyman) >>> * migration-test will go on next PULL request, as it has failures. >>> - Improve error description (tejus) >>> - improve -incoming and set parameters before calling incoming (wei) >>> - migration atomic counters reviewed patches (quintela) >>> - migration-test refacttoring reviewed (quintela) >> >> New failure with check-cfi-x86_64: >> >> https://gitlab.com/qemu-project/qemu/-/jobs/4527202764#L188 > > First of all, is there a way to get to the test log? In particular, I > am interested in knowing at least what test has failed (yes, > migration-test don't tell you much more). > > After a bit more wrestling, I have been able to get things compiling > with this command: > > $ /mnt/code/qemu/full/configure --enable-cfi > --target-list=x86_64-softmmu --enable-cfi-debug --cc=clang --cxx=clang++ > --disable-docs --enable-safe-stack --disable-slirp > > It should basically be the one that check-cfi-x86_64 is using if I > understand the build recipes correctly (that is a BIG IF). > > And it passes for me with flying colors. > Here I have Fedora38, builder has F37. > >> /builds/qemu-project/qemu/build/pyvenv/bin/meson test --no-rebuild -t >> 0 --num-processes 1 --print-errorlogs >> 1/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/qom-test >> OK 6.55s 8 subtests passed >> ▶ 2/350 ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: >> assertion failed: (bad == 0) ERROR >> 2/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test >> ERROR 151.99s killed by signal 6 SIGABRT >>>>> >> G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh >> MALLOC_PERTURB_=3 QTEST_QEMU_IMG=./qemu-img >> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon >> QTEST_QEMU_BINARY=./qemu-system-x86_64 >> /builds/qemu-project/qemu/build/tests/qtest/migration-test --tap >> -k >> ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― >> stderr: >> qemu-system-x86_64: Unable to read from socket: Connection reset by peer > > This is the interesting bit, why is the conection closed. > >> Memory content inconsistency at 4f65000 first_byte = 30 last_byte = 2f >> current = 88 hit_edge = 1 >> ** >> ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: assertion failed: (bad == 0) >> >> (test program exited with status code -6) > > This makes zero sense, except if we haven't migrated all the guest > state, that it is what it has happened. > > Is there a place on the web interface to see the full logs? Or that is > the only thing that the CI system stores? The "full logs" are https://gitlab.com/qemu-project/qemu/-/jobs/4527202764/artifacts/download?file_type=trace r~
Richard Henderson <richard.henderson@linaro.org> wrote: > On 6/26/23 00:01, Juan Quintela wrote: >> Richard Henderson <richard.henderson@linaro.org> wrote: >>> On 6/22/23 18:54, Juan Quintela wrote: >>>> The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb: >>>> Merge tag 'q800-for-8.1-pull-request' >>>> ofhttps://github.com/vivier/qemu-m68k into staging (2023-06-22 >>>> 10:18:32 +0200) >>>> are available in the Git repository at: >>>> https://gitlab.com/juan.quintela/qemu.git tags/next-pull-request >>>> for you to fetch changes up to >>>> 23e4307eadc1497bd0a11ca91041768f15963b68: >>>> migration/rdma: Split qemu_fopen_rdma() into input/output >>>> functions (2023-06-22 18:11:58 +0200) >>>> ---------------------------------------------------------------- >>>> Migration Pull request (20230621) take 2 >>>> In this pull request the only change is fixing 32 bits complitaion >>>> issue. >>>> Please apply. >>>> [take 1] >>>> - fix for multifd thread creation (fabiano) >>>> - dirtylimity (hyman) >>>> * migration-test will go on next PULL request, as it has failures. >>>> - Improve error description (tejus) >>>> - improve -incoming and set parameters before calling incoming (wei) >>>> - migration atomic counters reviewed patches (quintela) >>>> - migration-test refacttoring reviewed (quintela) >>> >>> New failure with check-cfi-x86_64: >>> >>> https://gitlab.com/qemu-project/qemu/-/jobs/4527202764#L188 >> First of all, is there a way to get to the test log? In particular, >> I >> am interested in knowing at least what test has failed (yes, >> migration-test don't tell you much more). >> After a bit more wrestling, I have been able to get things compiling >> with this command: >> $ /mnt/code/qemu/full/configure --enable-cfi >> --target-list=x86_64-softmmu --enable-cfi-debug --cc=clang --cxx=clang++ >> --disable-docs --enable-safe-stack --disable-slirp >> It should basically be the one that check-cfi-x86_64 is using if I >> understand the build recipes correctly (that is a BIG IF). >> And it passes for me with flying colors. >> Here I have Fedora38, builder has F37. >> >>> /builds/qemu-project/qemu/build/pyvenv/bin/meson test --no-rebuild -t >>> 0 --num-processes 1 --print-errorlogs >>> 1/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/qom-test >>> OK 6.55s 8 subtests passed >>> ▶ 2/350 ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: >>> assertion failed: (bad == 0) ERROR >>> 2/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test >>> ERROR 151.99s killed by signal 6 SIGABRT >>>>>> >>> G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh >>> MALLOC_PERTURB_=3 QTEST_QEMU_IMG=./qemu-img >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon >>> QTEST_QEMU_BINARY=./qemu-system-x86_64 >>> /builds/qemu-project/qemu/build/tests/qtest/migration-test --tap >>> -k >>> ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― >>> stderr: >>> qemu-system-x86_64: Unable to read from socket: Connection reset by peer >> This is the interesting bit, why is the conection closed. >> >>> Memory content inconsistency at 4f65000 first_byte = 30 last_byte = 2f >>> current = 88 hit_edge = 1 >>> ** >>> ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: assertion failed: (bad == 0) >>> >>> (test program exited with status code -6) >> This makes zero sense, except if we haven't migrated all the guest >> state, that it is what it has happened. >> Is there a place on the web interface to see the full logs? Or that >> is >> the only thing that the CI system stores? > > The "full logs" are > > https://gitlab.com/qemu-project/qemu/-/jobs/4527202764/artifacts/download?file_type=trace Not useful. I was hoping that there is something like when one runs ./tests/qtest/migration-test Anyways, to make things faster: - created /mnt/code/qemu/full/configure --enable-cfi --target-list=x86_64-softmmu --enable-cfi-debug --cc=clang --cxx=clang++ --disable-docs --enable-safe-stack --disable-slirp worked as a charm. - Your test run: qemu-system-x86_64: Unable to read from socket: Connection reset by peer one of the sides die, so anything else after that don't matter. And I don't understand what CFI is (and I don't rule out that posibility) or I can't understand how checking indirect functions call can make migration-test die without a single CFI error message? - I tried myself CI pipeline, some exact source: https://gitlab.com/juan.quintela/qemu/-/commit/23e4307eadc1497bd0a11ca91041768f15963b68/pipelines?ref=sent%2Fmigration-20230621b This is what fails: https://gitlab.com/juan.quintela/qemu/-/jobs/4527782025 16/395 ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/reconnect/subprocess [4569]) failed unexpectedly ERROR 16/395 qemu:qtest+qtest-x86_64 / qtest-x86_64/qos-test ERROR 27.46s killed by signal 6 SIGABRT >>> MALLOC_PERTURB_=92 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_BINARY=./qemu-system-x86_64 QTEST_QEMU_IMG=./qemu-img G_TEST_DBUS_DAEMON=/builds/juan.quintela/qemu/tests/dbus-vmstate-daemon.sh /builds/juan.quintela/qemu/build/tests/qtest/qos-test --tap -k ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― stderr: Vhost user backend fails to broadcast fake RARP qemu-system-x86_64: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-8XUX61/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-8XUX61/reconnect.sock,server=on qemu-system-x86_64: Failed to set msg fds. qemu-system-x86_64: vhost VQ 0 ring restore failed: -22: Invalid argument (22) qemu-system-x86_64: Failed to set msg fds. qemu-system-x86_64: vhost VQ 1 ring restore failed: -22: Invalid argument (22) ** ERROR:../tests/qtest/vhost-user-test.c:890:wait_for_rings_started: assertion failed (ctpop64(s->rings) == count): (1 == 2) ** ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/reconnect/subprocess [4569]) failed unexpectedly vhost? virtio-queue? in a non migration test? I don't know what is going on, but this is weird. Do we have a way to run on that image: ./tests/qtest/migration-test in a loop until it fails, and at least see what test is failing? Later, Juan.
Richard Henderson <richard.henderson@linaro.org> wrote: > On 6/22/23 18:54, Juan Quintela wrote: >> The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb: >> Merge tag 'q800-for-8.1-pull-request' >> ofhttps://github.com/vivier/qemu-m68k into staging (2023-06-22 >> 10:18:32 +0200) >> are available in the Git repository at: >> https://gitlab.com/juan.quintela/qemu.git tags/next-pull-request >> for you to fetch changes up to >> 23e4307eadc1497bd0a11ca91041768f15963b68: >> migration/rdma: Split qemu_fopen_rdma() into input/output >> functions (2023-06-22 18:11:58 +0200) >> ---------------------------------------------------------------- >> Migration Pull request (20230621) take 2 >> In this pull request the only change is fixing 32 bits complitaion >> issue. >> Please apply. >> [take 1] >> - fix for multifd thread creation (fabiano) >> - dirtylimity (hyman) >> * migration-test will go on next PULL request, as it has failures. >> - Improve error description (tejus) >> - improve -incoming and set parameters before calling incoming (wei) >> - migration atomic counters reviewed patches (quintela) >> - migration-test refacttoring reviewed (quintela) > > New failure with check-cfi-x86_64: I am looking at the whole series. I can't see a single function that is new/change prototypes/etc that is changed on this series. So is this problem related to CFI? Or it is a migration problem that somehow only happens when one use CFI? Inquiring minds what to know. Any clue? Later, Juan. > > https://gitlab.com/qemu-project/qemu/-/jobs/4527202764#L188 > > /builds/qemu-project/qemu/build/pyvenv/bin/meson test --no-rebuild -t > 0 --num-processes 1 --print-errorlogs > 1/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/qom-test > OK 6.55s 8 subtests passed > ▶ 2/350 ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: > assertion failed: (bad == 0) ERROR > 2/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test > ERROR 151.99s killed by signal 6 SIGABRT >>>> > G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh > MALLOC_PERTURB_=3 QTEST_QEMU_IMG=./qemu-img > QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon > QTEST_QEMU_BINARY=./qemu-system-x86_64 > /builds/qemu-project/qemu/build/tests/qtest/migration-test --tap > -k > ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― > stderr: > qemu-system-x86_64: Unable to read from socket: Connection reset by peer > Memory content inconsistency at 4f65000 first_byte = 30 last_byte = 2f > current = 88 hit_edge = 1 > ** > ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: assertion failed: (bad == 0) > > (test program exited with status code -6) > ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― > > > r~
On 6/26/23 15:05, Juan Quintela wrote: >> The "full logs" are >> >> https://gitlab.com/qemu-project/qemu/-/jobs/4527202764/artifacts/download?file_type=trace > > Not useful. I was hoping that there is something like when one runs > ./tests/qtest/migration-test I thought I saw some patch today that to save more artifacts. But the bottom line is that we don't emit enough stuff from any of our tests to debug them from logs -- we're too used to using other methods. > And I don't understand what CFI is (and I don't rule out that > posibility) or I can't understand how checking indirect functions call > can make migration-test die without a single CFI error message? CFI (control flow inspection/validation/somesuch) adds checking along call paths, which may affect timing. This is almost certainly some sort of race condition. > Do we have a way to run on that image: > > ./tests/qtest/migration-test > > in a loop until it fails, and at least see what test is failing? Not as is, no. You'd have to create a new CI job, and for that you'll need advice beyond myself. r~
Richard Henderson <richard.henderson@linaro.org> wrote: > On 6/22/23 18:54, Juan Quintela wrote: >> The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb: >> Merge tag 'q800-for-8.1-pull-request' >> ofhttps://github.com/vivier/qemu-m68k into staging (2023-06-22 >> 10:18:32 +0200) >> are available in the Git repository at: >> https://gitlab.com/juan.quintela/qemu.git tags/next-pull-request >> for you to fetch changes up to >> 23e4307eadc1497bd0a11ca91041768f15963b68: >> migration/rdma: Split qemu_fopen_rdma() into input/output >> functions (2023-06-22 18:11:58 +0200) >> ---------------------------------------------------------------- >> Migration Pull request (20230621) take 2 >> In this pull request the only change is fixing 32 bits complitaion >> issue. >> Please apply. >> [take 1] >> - fix for multifd thread creation (fabiano) >> - dirtylimity (hyman) >> * migration-test will go on next PULL request, as it has failures. >> - Improve error description (tejus) >> - improve -incoming and set parameters before calling incoming (wei) >> - migration atomic counters reviewed patches (quintela) >> - migration-test refacttoring reviewed (quintela) > > New failure with check-cfi-x86_64: > > https://gitlab.com/qemu-project/qemu/-/jobs/4527202764#L188 > > /builds/qemu-project/qemu/build/pyvenv/bin/meson test --no-rebuild -t > 0 --num-processes 1 --print-errorlogs > 1/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/qom-test > OK 6.55s 8 subtests passed > ▶ 2/350 ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: > assertion failed: (bad == 0) ERROR > 2/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test > ERROR 151.99s killed by signal 6 SIGABRT >>>> > G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh > MALLOC_PERTURB_=3 QTEST_QEMU_IMG=./qemu-img > QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon > QTEST_QEMU_BINARY=./qemu-system-x86_64 > /builds/qemu-project/qemu/build/tests/qtest/migration-test --tap > -k > ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― > stderr: > qemu-system-x86_64: Unable to read from socket: Connection reset by peer > Memory content inconsistency at 4f65000 first_byte = 30 last_byte = 2f > current = 88 hit_edge = 1 > ** > ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: assertion failed: (bad == 0) > > (test program exited with status code -6) > ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― Still running in bisect mode (this takes forever). [cc'ing stefan and kevin] And now I get this problem with gcov: https://gitlab.com/juan.quintela/qemu/-/jobs/4546094720 357/423 qemu:block / io-qcow2-copy-before-write ERROR 6.23s exit status 1 >>> PYTHON=/builds/juan.quintela/qemu/build/pyvenv/bin/python3 MALLOC_PERTURB_=154 /builds/juan.quintela/qemu/build/pyvenv/bin/python3 /builds/juan.quintela/qemu/build/../tests/qemu-iotests/check -tap -qcow2 copy-before-write --source-dir /builds/juan.quintela/qemu/tests/qemu-iotests --build-dir /builds/juan.quintela/qemu/build/tests/qemu-iotests ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― stderr: --- /builds/juan.quintela/qemu/tests/qemu-iotests/tests/copy-before-write.out +++ /builds/juan.quintela/qemu/build/scratch/qcow2-file-copy-before-write/copy-before-write.out.bad @@ -1,5 +1,21 @@ -.... +...F +====================================================================== +FAIL: test_timeout_break_snapshot (__main__.TestCbwError) +---------------------------------------------------------------------- +Traceback (most recent call last): + File "/builds/juan.quintela/qemu/tests/qemu-iotests/tests/copy-before-write", line 210, in test_timeout_break_snapshot + self.assertEqual(log, """\ +AssertionError: 'wrot[195 chars]read 1048576/1048576 bytes at offset 0\n1 MiB,[46 chars]c)\n' != 'wrot[195 chars]read failed: Permission denied\n' + wrote 524288/524288 bytes at offset 0 + 512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + wrote 524288/524288 bytes at offset 524288 + 512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ++ read failed: Permission denied +- read 1048576/1048576 bytes at offset 0 +- 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + + ---------------------------------------------------------------------- Ran 4 tests -OK +FAILED (failures=1) (test program exited with status code 1) ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― I have no clue how I can make qtests to fail with my changes. Specially with a read permission. Any clue? Later, Juan. PD. Yeap, continuing the bisect.