| Message ID | 20240805201719.2345596-1-tavip@google.com |
|---|---|
| Headers | show |
| Series | NXP i.MX RT595, ARM SVD and device model unit tests | expand |
On Mon, 5 Aug 2024 at 21:17, Octavian Purdila <tavip@google.com> wrote: > It also introduces unit tests for device models. To allow accessing > registers from unit tests a system bus mock is created. The main > advantage of unit tests for device models over QTest is that device > models can be tested in isolation and do not require a full qemu > machine. On the other hand the disadvantage is that you need to add a bunch of extra code to mock the interfaces that the device connects to, and then you compile into the test binary a subset of C files which weren't written with the expectation that they got compiled into tests like that, so it feels a bit brittle. The nice thing about qtest is that it doesn't need you to do any of that -- you just run the QEMU machine model and prod the devices it already has. Do you have examples in this series of tests which you were able to write with this unit test setup that you wouldn't have been able to write equivalents of with the qtest framework ? thanks -- PMM
On Mon, Aug 12, 2024 at 05:10:52PM +0100, Peter Maydell wrote: > On Mon, 5 Aug 2024 at 21:17, Octavian Purdila <tavip@google.com> wrote: > > It also introduces unit tests for device models. To allow accessing > > registers from unit tests a system bus mock is created. The main > > advantage of unit tests for device models over QTest is that device > > models can be tested in isolation and do not require a full qemu > > machine. > > On the other hand the disadvantage is that you need to add a > bunch of extra code to mock the interfaces that the device > connects to, and then you compile into the test binary a > subset of C files which weren't written with the expectation > that they got compiled into tests like that, so it feels a > bit brittle. The nice thing about qtest is that it doesn't > need you to do any of that -- you just run the QEMU machine > model and prod the devices it already has. Yep, mocking often ends up being rather a double-edged sword. You can do some really powerful things with it, and in particular it can enable you test scenarios / code paths that are hard or impractical to exercise with regular functional tests where fault injection is difficult/impossible. The cost of this though is that the mocks can pose a significant ongoing maint burden on anyone who refactors code in future, as the mocks need to evolve in lockstep with the code refactoring. 12 months down the line, it can be hard for another maintainer to understand the full purpose of the mocks & thus even harder to know what todo as they refactor the code. The future burden of mocks rarely falls on the person who creates them in the first place. > Do you have examples in this series of tests which you > were able to write with this unit test setup that you > wouldn't have been able to write equivalents of with the > qtest framework ? With regards, Daniel
On Mon, Aug 12, 2024 at 9:22 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Mon, Aug 12, 2024 at 05:10:52PM +0100, Peter Maydell wrote: > > On Mon, 5 Aug 2024 at 21:17, Octavian Purdila <tavip@google.com> wrote: > > > It also introduces unit tests for device models. To allow accessing > > > registers from unit tests a system bus mock is created. The main > > > advantage of unit tests for device models over QTest is that device > > > models can be tested in isolation and do not require a full qemu > > > machine. > > > > On the other hand the disadvantage is that you need to add a > > bunch of extra code to mock the interfaces that the device > > connects to, and then you compile into the test binary a > > subset of C files which weren't written with the expectation > > that they got compiled into tests like that, so it feels a > > bit brittle. The nice thing about qtest is that it doesn't > > need you to do any of that -- you just run the QEMU machine > > model and prod the devices it already has. > > Yep, mocking often ends up being rather a double-edged sword. > You can do some really powerful things with it, and in particular > it can enable you test scenarios / code paths that are hard or > impractical to exercise with regular functional tests where fault > injection is difficult/impossible. > > The cost of this though is that the mocks can pose a significant > ongoing maint burden on anyone who refactors code in future, as > the mocks need to evolve in lockstep with the code refactoring. > > 12 months down the line, it can be hard for another maintainer > to understand the full purpose of the mocks & thus even harder > to know what todo as they refactor the code. The future burden > of mocks rarely falls on the person who creates them in the > first place. > I agree that there are potentially maintenance issues. For what is worth, AFAIR, we only ran into one breaking change in about 3 years for which we used this approach which was also simple to fix (gpio was removed from hwcore so we had to pull in gpio.c directly). > > Do you have examples in this series of tests which you > > were able to write with this unit test setup that you > > wouldn't have been able to write equivalents of with the > > qtest framework ? > No, I think all of the tests in the patch set could be written in qtest.
This patch set adds support for NXP's RT500 MCU [1] and the RT595 EVK[2]. More RT500 device models will be submitted in future patch sets. The goal of this first patch set is to provide a minimal set that allows running the NXP MCU SDK hello world example[4] and to get feedback on a couple of new / non-standard approaches. First, this patch set introduces a (python) tool that generates C header files from ARM SVD files[3]. This significantly reduces the effort to write a new device model by automatically generating: register definitions and layout (including bit fields), register names for easier debugging and tracing, reset register values, etc. It also introduces unit tests for device models. To allow accessing registers from unit tests a system bus mock is created. The main advantage of unit tests for device models over QTest is that device models can be tested in isolation and do not require a full qemu machine. [1] https://www.nxp.com/docs/en/data-sheet/IMXRT500EC.pdf [2] https://www.nxp.com/webapp/Download?colCode=MIMXRT595EVKHUG [3] https://arm-software.github.io/CMSIS_5/SVD/html/index.html [4] Building and running the NXP MCU SDK hello world example Clone the following git repos: https://github.com/nxp-mcuxpresso/cmsis.git, https://github.com/nxp-mcuxpresso/mcux-sdk.git https://github.com/nxp-mcuxpresso/mcux-sdk-examples.git in the following directories: CMSIS, core, examples. cd examples/evkmimxrt595/demo_apps/hello_world/armgcc ARMGCC_DIR=/usr CFLAGS=-I../../../../../CMSIS/CMSIS/Core/Include \ sh build_flash_debug.sh qemu-system-arm --machine rt595-evk -kernel flash_debug/hello_world.elf \ -global armv7m.init-nsvtor=0x08001000 -global armv7m.init-svtor=0x08001000 \ -chardev stdio,id=flexcomm0 Octavian Purdila (19): fifo32: add peek function tests/unit: add fifo test hw/arm: add SVD file for NXP i.MX RT595 hw: add register access utility functions hw/misc: add basic flexcomm device model test/unit: add register access macros and functions test/unit: add flexcomm unit test hw/char: add support for flexcomm usart test/unit: add flexcomm usart unit test hw/i2c: add support for flexcomm i2c test/unit: add i2c-tester test/unit: add unit tests for flexcomm i2c test/unit: add spi-tester hw/misc: add support for RT500's clock controller test/unit: add unit tests for RT500's clock controller hw/ssi: add support for flexspi hw/misc: add support for RT500 reset controller hw/arm: add basic support for the RT500 SoC hw/arm: add RT595-EVK board Sebastian Ene (2): hw/ssi: add support for flexcomm spi test/unit: add unit tests for flexcomm spi Stefan Stanacar (1): scripts: add script to generate C header files from SVD XML files Valentin Ghita (1): tests/unit: add system bus mock configure | 2 +- hw/arm/Kconfig | 13 + hw/arm/meson.build | 4 + hw/arm/rt500.c | 348 + hw/arm/rt595-evk.c | 64 + hw/arm/svd/MIMXRT595S_cm33.xml | 224052 +++++++++++++++++++++++++++ hw/arm/svd/meson.build | 42 + hw/char/flexcomm_usart.c | 302 + hw/char/meson.build | 1 + hw/char/trace-events | 9 + hw/i2c/flexcomm_i2c.c | 224 + hw/i2c/meson.build | 1 + hw/i2c/trace-events | 10 + hw/misc/Kconfig | 12 + hw/misc/flexcomm.c | 304 + hw/misc/meson.build | 5 + hw/misc/rt500_clkctl0.c | 243 + hw/misc/rt500_clkctl1.c | 224 + hw/misc/rt500_rstctl.c | 219 + hw/misc/trace-events | 18 + hw/ssi/Kconfig | 4 + hw/ssi/flexcomm_spi.c | 443 + hw/ssi/flexspi.c | 216 + hw/ssi/meson.build | 2 + hw/ssi/trace-events | 12 + include/hw/arm/rt500.h | 49 + include/hw/char/flexcomm_usart.h | 20 + include/hw/i2c/flexcomm_i2c.h | 27 + include/hw/misc/flexcomm.h | 92 + include/hw/misc/rt500_clk_freqs.h | 18 + include/hw/misc/rt500_clkctl0.h | 37 + include/hw/misc/rt500_clkctl1.h | 38 + include/hw/misc/rt500_rstctl.h | 38 + include/hw/regs.h | 89 + include/hw/ssi/flexcomm_spi.h | 20 + include/hw/ssi/flexspi.h | 34 + include/qemu/fifo32.h | 29 + meson.build | 4 + python/setup.cfg | 1 + python/tests/minreqs.txt | 3 + pythondeps.toml | 3 + scripts/svd-gen-header.py | 342 + tests/unit/i2c_tester.c | 111 + tests/unit/i2c_tester.h | 34 + tests/unit/meson.build | 56 +- tests/unit/reg-utils.h | 103 + tests/unit/spi_tester.c | 60 + tests/unit/spi_tester.h | 32 + tests/unit/sysbus-mock.c | 314 + tests/unit/sysbus-mock.h | 82 + tests/unit/test-fifo.c | 98 + tests/unit/test-flexcomm-i2c.c | 209 + tests/unit/test-flexcomm-spi.c | 204 + tests/unit/test-flexcomm-usart.c | 321 + tests/unit/test-flexcomm.c | 215 + tests/unit/test-rt500-clkctl.c | 270 + 56 files changed, 229725 insertions(+), 2 deletions(-) create mode 100644 hw/arm/rt500.c create mode 100644 hw/arm/rt595-evk.c create mode 100644 hw/arm/svd/MIMXRT595S_cm33.xml create mode 100644 hw/arm/svd/meson.build create mode 100644 hw/char/flexcomm_usart.c create mode 100644 hw/i2c/flexcomm_i2c.c create mode 100644 hw/misc/flexcomm.c create mode 100644 hw/misc/rt500_clkctl0.c create mode 100644 hw/misc/rt500_clkctl1.c create mode 100644 hw/misc/rt500_rstctl.c create mode 100644 hw/ssi/flexcomm_spi.c create mode 100644 hw/ssi/flexspi.c create mode 100644 include/hw/arm/rt500.h create mode 100644 include/hw/char/flexcomm_usart.h create mode 100644 include/hw/i2c/flexcomm_i2c.h create mode 100644 include/hw/misc/flexcomm.h create mode 100644 include/hw/misc/rt500_clk_freqs.h create mode 100644 include/hw/misc/rt500_clkctl0.h create mode 100644 include/hw/misc/rt500_clkctl1.h create mode 100644 include/hw/misc/rt500_rstctl.h create mode 100644 include/hw/regs.h create mode 100644 include/hw/ssi/flexcomm_spi.h create mode 100644 include/hw/ssi/flexspi.h create mode 100755 scripts/svd-gen-header.py create mode 100644 tests/unit/i2c_tester.c create mode 100644 tests/unit/i2c_tester.h create mode 100644 tests/unit/reg-utils.h create mode 100644 tests/unit/spi_tester.c create mode 100644 tests/unit/spi_tester.h create mode 100644 tests/unit/sysbus-mock.c create mode 100644 tests/unit/sysbus-mock.h create mode 100644 tests/unit/test-fifo.c create mode 100644 tests/unit/test-flexcomm-i2c.c create mode 100644 tests/unit/test-flexcomm-spi.c create mode 100644 tests/unit/test-flexcomm-usart.c create mode 100644 tests/unit/test-flexcomm.c create mode 100644 tests/unit/test-rt500-clkctl.c